Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JMX tries to double-bind a port when using bundle exec #1859

Closed
headius opened this Issue Jul 25, 2014 · 10 comments

Comments

Projects
None yet
4 participants
@headius
Copy link
Member

headius commented Jul 25, 2014

JMX fails to bind when running bundle exec.

system ~/projects/jruby $ cat blah.rb
#!/usr/bin/env jruby

system ~/projects/jruby $ JRUBY_OPTS="-J-Dcom.sun.management.jmxremote -J-Dcom.sun.management.jmxremote.authenticate=false -J-Dcom.sun.management.jmxremote.ssl=false -J-Dcom.sun.management.jmxremote.port=1100 --server" jruby -S bundle exec ./blah.rb
file:/Users/headius/projects/jruby/lib/jruby.jar!/jruby/kernel/kernel.rb:25 warning: unsupported exec option: close_others
Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 1100; nested exception is: 
    java.net.BindException: Address already in use

Bundler's "bundle exec" basically loads up your gemfile, prepares the appropriate library versions, and then re-starts JRuby with those libraries, using exec. Exec on JRuby does a real native exec, replacing the parent process with the new one. However, because the JVM does not set socket descriptors CLOEXEC, the JMX socket bound in the initial launch of JRuby remains bound in the replaced process, causing the second JRuby/JVM instance to fail to bind.

In order to fix this, we need to set sockets to be CLOEXEC. That requires native access to fcntl(2), but so does the real native exec.

A workaround for this may be to use the -G flag, which boots up Bundler's gemfile before starting up RubyGems.

Reported by @polysics on IRC.

@headius headius added this to the JRuby 9000 milestone Jul 25, 2014

@headius headius added the core label Jul 25, 2014

@mjc

This comment has been minimized.

Copy link
Contributor

mjc commented Jul 25, 2014

Looks like bundle exec currently does more than -G does (https://github.com/bundler/bundler/blob/d4bde24effb590c779969e9e723fd05682de4a14/lib/bundler/cli/exec.rb#L10)

It calls Bundler.definition.validate_ruby! and Bundler.load.setup_environment before exec'ing.

setup_environment modifies the BUNDLE_BIN_PATH, PATH, RUBYOPT, and RUBYLIB here: https://github.com/bundler/bundler/blob/b625f820406bda39e5422c25428ff371a878e205/lib/bundler/runtime.rb#L214

@lpradovera

This comment has been minimized.

Copy link

lpradovera commented Jul 25, 2014

The original issue was discovered by trying to enable JMX remote monitoring. The process replacement causes a

Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 10436; nested exception is:
    java.net.BindException: Address already in use

exception and the script/application fails to start.

headius added a commit that referenced this issue Jul 25, 2014

Unify ChannelFD creation and set cloexec for all socket types.
This does not fix the issues described in #1859 but at least we
are fixing *our* sockets.
@headius

This comment has been minimized.

Copy link
Member Author

headius commented Jul 25, 2014

Fixing this is actually a bit trickier than I realized. We don't have access to the actual socket the JVM starts up for JMX, so we can't set CLOEXEC on it. So, an alternate approach might be to make any JMX-shutting-down calls before we exec?

@mjc

This comment has been minimized.

Copy link
Contributor

mjc commented Jul 25, 2014

@polysics out of curiosity, have you tried using bundle binstub to make a wrapper in bin and then using jruby -G bin/adh start . ?

@headius

This comment has been minimized.

Copy link
Member Author

headius commented Jul 25, 2014

Shorter reproduction that doesn't use bundler:

export JRUBY_OPTS="-J-Dcom.sun.management.jmxremote -J-Dcom.sun.management.jmxremote.authenticate=false -J-Dcom.sun.management.jmxremote.port=1100 --server"
jruby -e 'exec "jruby ./blah.rb"'

I have a fix I will commit shortly, but if someone can come up with a test for this, that would be great. It should test that the JMX server works both before and after the exec, probably by controlling a child JRuby process.

@headius

This comment has been minimized.

Copy link
Member Author

headius commented Jul 25, 2014

Oh, ./blah.rb just has a jruby/env shebang.

@headius headius closed this in 6b7bee2 Jul 25, 2014

@headius headius modified the milestones: JRuby 1.7.14, JRuby 9000 Jul 25, 2014

@headius

This comment has been minimized.

Copy link
Member Author

headius commented Jul 25, 2014

Fixed but still needs some kind of test. I did the following tests by hand locally:

  • successful exec, confirm post-exec process has JMX agent running
  • failed exec, confirm existing process restarts JMX agent and re-registers JRuby mbeans.
@lpradovera

This comment has been minimized.

Copy link

lpradovera commented Jul 28, 2014

Would you like a spec for that, or just some more manual confirmation? If you need a spec, I think I need some directions as to where to put it, there seem to be tests of all types all around spec/ and test/ :)

dcorrigan pushed a commit to scribenet/file_system_project that referenced this issue Aug 5, 2014

dcorrigan
@lpradovera

This comment has been minimized.

Copy link

lpradovera commented Aug 12, 2014

Hello,
is there anything I can do to help the fix to get into a release?

@enebo

This comment has been minimized.

Copy link
Member

enebo commented Aug 12, 2014

Whether someone makes a test for this it was landed on jruby-1_7 already. So 1.7.14 will have this. That hopefully will be less than a week away.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.