Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Specifying JMX parameters in LS_JAVA_OPTS causes Logstash not to restart properly #4319

Closed
ppf2 opened this Issue Dec 9, 2015 · 4 comments

Comments

Projects
None yet
4 participants
@ppf2
Copy link
Member

commented Dec 9, 2015

We added LS_JAVA_OPTS here. When using the RPM, there are multiple places you can specify LS_JAVA_OPTS, eg. in the /etc/sysconfig/logstash and /etc/init.d/logstash files. But it looks like the /etc/init.d/logstash is the right place to specify it.

Here is an example when seeing up a JMX connection, if you specify it in /etc/sysconfig/logstash file, this causes issues when using /etc/init.d/logstash restart - it will eventually end up launching multiple RMI servers and fail with a bind exception, eg.

sudo /etc/init.d/logstash restart
Configuration error. Not restarting. Re-run with configtest parameter for details

sudo /etc/init.d/logstash configtest 
WARNING: Default JAVA_OPTS will be overridden by the JAVA_OPTS defined in the environment. Environment JAVA_OPTS are -Djava.rmi.server.hostname=localhost -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9994 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false 
Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 9994; nested exception is: 
java.net.BindException: Address already in use

So I moved the LS_JAVA_OPTS settings for the JMX connection from /etc/sysconfig/logstash to /etc/init.d/logstash. After that, I am no longer able to reproduce the restart issue.

Not sure if this is a bug. If not, it will be nice to document this better in our guide (and as a comment within the sysconfig file, etc..).

@untergeek

This comment has been minimized.

Copy link
Member

commented Dec 9, 2015

They absolutely should go in /etc/sysconfig/logstash for RedHat variants, or /etc/default/logstash for Debian/Ubuntu variants. I'm not sure why it's failing for JMX in sysconfig, but we ought to fix that and verify the same isn't happening with Debian variants in /etc/default/logstash.

That said, using JMX to monitor Logstash leaves a bad taste in my mouth. It's going to give broad and potentially meaningless metrics. Sure, you can see how much heap you're using, but with Logstash that's really only useful if there's a memory leak. I don't even know if GC is instrumented in JMX, and if so, if it's even a valuable thing with Logstash. I hope we can get useful metrics out soon.

@ppf2 ppf2 added bug and removed docs labels Dec 9, 2015

@ppf2

This comment has been minimized.

Copy link
Member Author

commented Feb 24, 2016

Reproducible using LS 2.2.2 with the simplest configuration. Once JMX is added to LS_JAVA_OPTS, it will not restart properly. See video repro here.

#4666 is another example of a community user running into the same issue.

@ppf2 ppf2 changed the title Better documentation on where to specify LS_JAVA_OPTS Specifying JMX parameters in LS_JAVA_OPTS causes Logstash not to restart properly Feb 24, 2016

@acchen97

This comment has been minimized.

Copy link
Contributor

commented Mar 3, 2016

Logstash does not support JMX. For monitoring, it's recommended to use the upcoming metrics API in Logstash 5.0, which will expose pipeline, JVM, OS metrics and hot threads.

@acchen97 acchen97 closed this Mar 3, 2016

@suyograo suyograo reopened this Mar 4, 2016

suyograo added a commit to suyograo/logstash that referenced this issue Mar 4, 2016

Fix configtest option in init scripts
No need to export all the JAVA* related environment variables
for configtest. It needs a minimal set of variables to execute
bin/logstash --config-test. This was indirectly causing issues
when used with JMX options in LS_JAVA_OPTS. JMX needs a remote port defined
and configtest was trying to connect to the port twice which caused
restart (elastic#4319) to fail

Fixes elastic#4319
@suyograo

This comment has been minimized.

Copy link
Member

commented Mar 4, 2016

Reopening this. I found a fix for this and was able to validate it on deb packages. #4759

There is still a weird behavior of duplicating LS_JAVA_OPTS that is really baffling me, but I'll chase it for another day. Duplicating options were are reported in #3315

elasticsearch-bot pushed a commit that referenced this issue Mar 4, 2016

Fix configtest option in init scripts
No need to export all the JAVA* related environment variables
for configtest. It needs a minimal set of variables to execute
bin/logstash --config-test. This was indirectly causing issues
when used with JMX options in LS_JAVA_OPTS. JMX needs a remote port defined
and configtest was trying to connect to the port twice which caused
restart (#4319) to fail

Fixes #4319

Fixes #4759

elasticsearch-bot pushed a commit that referenced this issue Mar 4, 2016

Fix configtest option in init scripts
No need to export all the JAVA* related environment variables
for configtest. It needs a minimal set of variables to execute
bin/logstash --config-test. This was indirectly causing issues
when used with JMX options in LS_JAVA_OPTS. JMX needs a remote port defined
and configtest was trying to connect to the port twice which caused
restart (#4319) to fail

Fixes #4319

Fixes #4759
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.