Logstash upstart script not working for some users

[alappe]

/var/log/upstart/logstash.log only gives the following: Sending logstash logs to /var/log/logstash/logstash.log. Sadly, there's no information in it. So I tried to run command that is executed as the logstash user:

$ /usr/bin/java -Djava.io.tmpdir=/var/lib/logstash -Xmx500m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -jar /opt/logstash/vendor/jar/jruby-complete-1.7.11.jar  -I/opt/logstash/lib /opt/logstash/lib/logstash/runner.rb agent -f /etc/logstash/conf.d -l /var/log/logstash/logstash.log
LoadError: no such file to load -- i18n
  require at org/jruby/RubyKernel.java:1085
  require at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:55
  require at /opt/logstash/lib/logstash/JRUBY-6970.rb:27
   (root) at /opt/logstash/lib/logstash/runner.rb:50

It works if I set $GEM_HOME:

$ GEM_HOME="/opt/logstash/vendor/bundle/jruby/1.9/" /usr/bin/java -Djava.io.tmpdir=/var/lib/logstash -Xmx500m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -jar /opt/logstash/vendor/jar/jruby-complete-1.7.11.jar  -I/opt/logstash/lib /opt/logstash/lib/logstash/runner.rb agent -f /etc/logstash/conf.d -l /var/log/logstash/logstash.log

I don't know if that is the actual problem, or if this is simply because I am running the command differently compared to the upstart daemon. If you have any more clues on how to dig into that, I will try…

[bugcy013]

I am also getting same Error. running as command line.

/opt/logstash# /app_packages/jdk1.7.0_45/bin/java -Xmx500m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -jar /opt/logstash/vendor/jar/jruby-complete-1.7.11.jar -I/opt/logstash/lib /opt/logstash/lib/logstash/runner.rb agent -f conf/logstash.conf
LoadError: no such file to load -- i18n
require at org/jruby/RubyKernel.java:1085
require at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:55
require at /opt/logstash/lib/logstash/JRUBY-6970.rb:27
(root) at /opt/logstash/lib/logstash/runner.rb:50

[jordansissel]

I tried to reproduce this today (in a vm) but I was unable to proceed due to problems with the vm itself - silly computers!

I'll try again once I've got an ubuntu vm online.

[jpb]

I've run into the same issue.

Installing Logstash 1.4.0 from a .deb, the process is continually restarted by Upstart(new pids can be seen in top). If I run the command manually, it runs fine:

$ sudo -u logstash GEM_HOME=/opt/logstash/vendor/bundle/jruby/1.9 /usr/bin/java -Xmx500m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -jar /opt/logstash/vendor/jar/jruby-complete-1.7.11.jar -I/opt/logstash/
lib /opt/logstash/lib/logstash/runner.rb agent -f /etc/logstash/conf.d -l /var/log/logstash/logstash.log
Sending logstash logs to /var/log/logstash/logstash.log.

There are no errors in /var/log/logstash/logstash.log or /var/log/upstart/logstash.log.

I've also cut my configuration down to the bare minimum:

$ cat /etc/logstash/conf.d/logstash.conf
input { stdin { type => "stdin-type"} }
output { stdout { } }

[electrical]

Hi,

/opt/logstash/bin/logstash does several settings including the GEM_HOME one.
When launching the java process directly, all those nice things are not set and are causing that issue.

Can you verify if you have the same issue when you use the script instead calling java directly?

Cheers.

[alappe]

Hey @electrical,

sorry, this might have been misleading. I am using the script, but it keeps crashing/respawning in upstart. See the Vagrantfile below to test it yourself (tail /var/log/upstart/logstash.log to see the respawning):

# -*- mode: ruby -*-
# vi: set ft=ruby :

# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.vm.box = "helderco/trusty64"
  config.vm.provision :shell, :inline => "cd /tmp && wget 'https://download.elasticsearch.org/logstash/logstash/packages/debian/logstash_1.4.0-1-c82dc09_all.deb'"
  config.vm.provision :shell, :inline => "sudo dpkg -i /tmp/logstash_1.4.0-1-c82dc09_all.deb; sudo apt-get -fy install && sudo dpkg -i /tmp/logstash_1.4.0-1-c82dc09_all.deb"
  config.vm.provision :shell, :inline => "sudo echo 'input { stdin{} }\noutput { stdout{} }\n' > /etc/logstash/conf.d/t.conf"
  config.vm.provision :shell, :inline => "sudo service logstash start && sudo service logstash-web start"
end

If you meant testing without upstart but with the script:

$ sudo -u logstash /opt/logstash/bin/logstash agent -f /etc/logstash/conf.d -l /var/log/logstash/logstash.log

This works…

[electrical]

@alappe ah okay. we haven't done anything to the upstart configs, only the sysv init scripts for 1.4.1

[alappe]

@electrical Too bad. And the release is imminent, so no chance? It's a pity because you cannot use the deb on ubuntu at the moment without adding a separate upstart config (as I have no fix for the issue, I wouldn't know what to override the upstart config with anyway)…

[electrical]

You could remove the upstart config and use the init script, i do the same for the puppet module :-)
I'll make sure we fix the upstart config for 1.4.2 / 1.5.0

[alappe]

Alright… thanks for taking care!

[jpb]

@electrical it looks like the init script is no good either:

$ sudo /etc/init.d/logstash start
 * Starting Logstash Daemon                                                                                                                                                                                 start-stop-daemon: unable to stat  (No such file or directory)

See https://logstash.jira.com/browse/LOGSTASH-2155 for more detail.

[electrical]

@jpb that is most likely the old init script.
https://github.com/elasticsearch/logstash/blob/master/pkg/logstash.sysv is the new one.
Can you try out that one? ( it will be shipped with 1.4.1 )

[alappe]

@electrical On my test box, it starts without error message, but after some seconds the process is gone. It doesn't log into logstash.log, logstash.stdout nor logstash.err

Let's see what @jpb sees…

[jpb]

logstash.sysv from master starts successfully, but Logstash dies after ~10 seconds, and there is nothing in the logs.

[varman87]

Make sure there are no other files in your LS_CONF_DIR or point it directly to your conf file. Had the same issue.

[guss77]

I have the same issue - logstash 1.4.1 installed from elasticsearch.org repositories crashes after a minute and respawns. Adding the GEM_HOME line to /etc/default/logstash solved the problem for me (though its a pain editing this every time I deploy a new machine).

[alappe]

@electrical Is there some progress on the horizon? This makes deploying/using logstash a less fun experience…

[CameronNemo]

@alappe for easier deployment, you could try adding an override file for the upstart job:

env GEM_HOME=yada yada

[kawikao]

Same issue as OP. Fresh Ubuntu 14.04, Logstash 1.4.2 from debian repo. Minimal config file. Do a service logstash start. Process starts but nothing is happening and logs are empty. When I do service logstash stop the startup messages show up in the logstash logs. Tried putting GEM_HOME in front of direct java call like OP and that works. We have over 100 servers and growing and every new server shows this error. Not fun. Thanks.

[jordansissel]

So I tried to run command that is executed as the logstash user:

This way of invoking logstash is fine, but you didn't set any environment variables that the logstash start script sets, so it shouldn't work unless you set all the environment variables - this is why I do not recommend invoking logstash with the java command. Use bin/logstash like the upstart script uses instead?

---

I will agree that many users are reporting problems, but the "solution" doesn't make sense because this variable is set by bin/logstash.

I will try to reproduce.

[jordansissel]

Logstash 1.4.2 on Ubuntu 14.04 is working fine for me. I install it, add a config, start the logstash service via upstart, and everything is fine:
https://gist.github.com/jordansissel/de0a1e1dcf650228f2d4

For everyone encountering this problem, can you please attach as a gist:

• a list of all files in /etc/logstash/conf.d/
• the contents of all files in /etc/logstash/conf.d/
• the contents of /var/log/upstart/logstash.log
• the contents of /var/log/logstash/logstash.log
• what command you are using to start logstash

[alappe]

@jordansissel This wasn't meant as a solution but merely a way to debug what could be wrong with the Upstart script…

[kawikao]

@jordansissel thanks for the gist. Turns out the Ubuntu image I was on was not clean. My bad. I loaded an actual fresh Ubuntu VM and my configs and it works. So something we're installing or upgrading to our base OS is causing this problem. Thanks for the help. I'll comment back if I track it down.

btw, my /opt/logstash/bin/logstash file doesn't set GEM_HOME.

[alappe]

So, just to make it repeatable:

# -*- mode: ruby -*-
# vi: set ft=ruby :

# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  installLogstash = <<EOT
  cd /tmp && \
  wget 'https://download.elasticsearch.org/logstash/logstash/packages/debian/logstash_1.4.2-1-2c0f5a1_all.deb' && \
  echo 'fb2c384f61b6834094b31a34d75b58d00c71d81a  logstash_1.4.2-1-2c0f5a1_all.deb' | shasum --check - && \
  sudo apt-get update && \
  sudo apt-get install -y 'java7-runtime-headless' && \
  sudo dpkg -i logstash_1.4.2-1-2c0f5a1_all.deb && \
  echo "input { stdin { } }\n output { stdout { codec => rubydebug } }\n" > /etc/logstash/conf.d/simple.conf && \
  sudo service logstash start && \
  sudo service logstash-web start
EOT

  config.vm.box = "precise64"
  config.vm.provision "shell", inline: installLogstash
end

• a list of all files in /etc/logstash/conf.d/

vagrant@precise64:~$ ls -la /etc/logstash/conf.d/
total 12
drwxrwxr-x 2 root root 4096 Jul  4 01:19 .
drwxrwxr-x 3 root root 4096 Jul  4 01:19 ..
-rw-r--r-- 1 root root   63 Jul  4 01:19 simple.conf
vagrant@precise64:~$ 

• the contents of all files in /etc/logstash/conf.d/
  As written in the Vagrantfile…
• the contents of /var/log/upstart/logstash.log

vagrant@precise64:~$ sudo tail -f /var/log/upstart/logstash.log 
Sending logstash logs to /var/log/logstash/logstash.log.
Sending logstash logs to /var/log/logstash/logstash.log.
Sending logstash logs to /var/log/logstash/logstash.log.
Sending logstash logs to /var/log/logstash/logstash.log.
Sending logstash logs to /var/log/logstash/logstash.log.
Sending logstash logs to /var/log/logstash/logstash.log.
Sending logstash logs to /var/log/logstash/logstash.log.
Sending logstash logs to /var/log/logstash/logstash.log.
Sending logstash logs to /var/log/logstash/logstash.log.
Sending logstash logs to /var/log/logstash/logstash.log.
…

• the contents of /var/log/logstash/logstash.log

vagrant@precise64:~$ sudo cat /var/log/logstash/logstash.log 
vagrant@precise64:~$ 

• what command you are using to start logstash
  See Vagrantfile…
• the content of /var/log/syslog

vagrant@precise64:~$ sudo tail -f /var/log/syslog 
Jul  4 01:25:13 precise64 kernel: [  424.247709] init: logstash-web main process (7216) terminated with status 1
Jul  4 01:25:13 precise64 kernel: [  424.247723] init: logstash-web main process ended, respawning
Jul  4 01:25:20 precise64 kernel: [  431.201597] init: logstash main process ended, respawning
Jul  4 01:25:37 precise64 kernel: [  447.919032] init: logstash main process ended, respawning
Jul  4 01:25:47 precise64 kernel: [  457.968933] init: logstash main process ended, respawning
Jul  4 01:25:55 precise64 kernel: [  465.992797] init: logstash main process ended, respawning
Jul  4 01:26:03 precise64 kernel: [  473.842204] init: logstash main process ended, respawning
Jul  4 01:26:07 precise64 kernel: [  477.930840] init: logstash-web main process (7532) terminated with status 1
Jul  4 01:26:07 precise64 kernel: [  477.930853] init: logstash-web main process ended, respawning
Jul  4 01:26:19 precise64 kernel: [  489.178478] init: logstash main process ended, respawning
…

[ccastillo-contactpoint]

@jordansissel with the debian packages (Ubuntu 14.04) logstash 1.4.2-1-2c0f5a1 starts fine, still there's a problem with logstash-web, which is fixed by adding GEM_HOME="/opt/logstash/vendor/bundle/jruby/1.9" to /etc/default/logstash-web (I tested both, upstart and sysv init scripts and adding GEM_HOME fixes things and make kibana to start).

Without GEM_HOME you get this error:

LoadError: no such file to load -- i18n
require at org/jruby/RubyKernel.java:1085
require at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:55
(root) at /opt/logstash/lib/logstash/runner.rb:46

Which seems to be a problem loading the i18n gem in this line:

https://github.com/elasticsearch/logstash/blob/1.4/lib/logstash/runner.rb#L46

Thanks

[ryanstillions]

@alappe I think your simple.conf might be a bit too... simple, for testing purposes. You mentioned above that you're using:

echo "input { stdin { } }\n output { stdout { codec => rubydebug } }\n" > /etc/logstash/conf.d/simple.conf && \

If you temporarily set LS_OPTS="-vv" in your /etc/init/logstash.conf you'll soon notice in /var/log/logstash/logstash.log that the stdin input is shutting down, which causes the process to shut down, which causes upstart to attempt respawn (I suspect).

{:timestamp=>"2014-09-06T17:14:06.864000-0400", :message=>"stdin shutting down.", :level=>:debug, :file=>"logstash/inputs/stdin.rb", :line=>"43"}
{:timestamp=>"2014-09-06T17:14:06.874000-0400", :message=>"Plugin is finished", :plugin=><LogStash::Outputs::Stdout >, :level=>:info, :file=>"logstash/plugin.rb", :line=>"59"}
{:timestamp=>"2014-09-06T17:14:06.877000-0400", :message=>"Pipeline shutdown complete.", :level=>:info, :file=>"logstash/pipeline.rb", :line=>"89"}

If you give it a config with an input that actually stays open ( like the logstash-apache.conf example found here http://logstash.net/docs/1.4.2/tutorials/getting-started-with-logstash ) the input plugin will stay open and the process stays up just fine.

[sbitpdc]

logstash errors is in /var/log/logstash/logstash.err

[oernii]

1.4.2 doesn't start correctly with the supplied init script on EL6. With the Git version it works. I think the 1.4.3 version should contain the git init script.

[Vincent--]

I have the same issue on my VM with Ubuntu 14.04

Logstash agent starts
Logstash web starts but eats all CPU ("syslog tcp listener died" and "syslog udp listener died" in log, with this adress "address=>0.0.0.0:514")

After some investigations
==> if I starts logstash web using : "sudo service logstash-web start", that doesn't work and port isn't specified for the process

/usr/bin/java -Djava.io.tmpdir=/var/lib/logstash -Xmx500m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -jar /opt/logstash/vendor/jar/jruby-complete-1.7.11.jar -I/opt/logstash/lib /opt/logstash/lib/logstash/runner.rb web -l /var/log/logstash/logstash.log

==> if I starts logstash web using : "sudo /etc/init.d/logstash-web start", that works and port is specified for the process (see the end of the line)

/usr/bin/java -Djava.io.tmpdir=/var/lib/logstash -Xmx500m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -jar /opt/logstash/vendor/jar/jruby-complete-1.7.11.jar -I/opt/logstash/lib /opt/logstash/lib/logstash/runner.rb web -a 0.0.0.0 -p 9292

Strange, isn't it ?
Why ? Because, when we use "service", the system tries to find a file "/etc/init/logstash-web.conf" and uses it to launch logstash.

In the script /etc/init.d/logstash-web, these options are specified and passed to logstash

LS_WEB_ADDRESS="0.0.0.0"
LS_WEB_PORT=9292

In the file /etc/init/logstash-web.conf, there is no option specified

LS_OPTS=""

so, logstash doesn't use 9292 this time

I tried to modify the file /etc/init/logstash-web.conf like that

LS_OPTS="-a 0.0.0.0 -p 9292"

but that didn't work, because there is a bug in the file.

In the few next lines, we have that

LS_OPTS="${LSOPTS} -l ${LS_LOG_FILE}"

and that should be

LS_OPTS="${LS_OPTS} -l ${LS_LOG_FILE}"

(see the underscore between LS and OPTS)

With this 2 modifications, logstash web works like a charm..
(NB : you can specify LS_OPTS="-a 0.0.0.0 -p 9292" in /etc/default/logstash-web)
Hope that helps ;-)

[jordansissel]

logstash-web is removed in the next release (Logstash 1.5.0)

[Vincent--]

Do you mean remove /etc/init.d/logstash-web or remove /etc/init/logstash-web.conf or remove both ?
How the web interface will be started in Logstash 1.5.0 ?

Thanking you in advance.

[oernii]

I have found out why logstash doesn't start with rhel6 service. Logstash uses some unusual method to start the service, via :
chroot --userspec $LS_USER:$LS_GROUP
This removes ALL users groups besides LS_GROUP. Logstash then cannot read the logs (permission denied says strace), It however, does NOT LOG anything even in --debug mode. So there is a bug.

I have set LS_GROUP="squid" to my /etc/sysconfig/logstash as a workaround. Unfortunately only 1 group can be used.

[eshikafe]

What could be the issue here - unable to access "jruby-complete-1.7.11.jar"?
OS: Windows 7 via Cygwin.

[logstash-1.4.2]$ bin/logstash -e 'input { stdin {} } output { stdout {}}'
Error: Unable to access jarfile /home/eausaig/analytics/logstash-1.4.2/logstash-1.4.2/vendor/jar/jruby-complete-1.7.11.jar
[logstash-1.4.2]$ bin/logstash help
Error: Unable to access jarfile /home/eausaig/analytics/logstash-1.4.2/logstash-1.4.2/vendor/jar/jruby-complete-1.7.11.jar

[logstash-1.4.2]$ ls vendor/jar/ -l
total 21188
drwxr-xr-x+ 1 eausaig Domain Users 0 May 15 2014 elasticsearch-1.1.1/
-rwxr-xr-x 1 eausaig Domain Users 21692990 May 15 2014 jruby-complete-1.7.11.jar*

[eshikafe]

I think I found a solution here: #1524