JRuby 1.7.0 unix sockets - 100% CPU - worked in preview2 #398

Closed
haberbyte opened this Issue Nov 16, 2012 · 1 comment

Projects

None yet

1 participant

@haberbyte

I noticed this problem first when upgrading a Rails app on a Ubuntu server using jruby-1.7.0-preview2 to jruby-1.7.0.

I'm using the puma webserver listening on a unix socket and nginx as a frontend.
This is the typical nginx config part:

upstream app {
    server unix:/tmp/puma.sock fail_timeout=0;
}

On jruby-1.7.0, after a few requests CPU load of the java process goes up. Sometimes the server still responds for a few requests as the response times are getting really slow.
Finally, the whole thing doesn't repond at all, the puma webserver crashes with this error:

IOError: closed stream
          close at org/jruby/RubyIO.java:2053
  begin_restart at /u/apps/docmago/shared/bundle/jruby/1.9/gems/puma-1.6.3-java/lib/puma/server.rb:637
            run at /u/apps/docmago/shared/bundle/jruby/1.9/gems/puma-1.6.3-java/lib/puma/cli.rb:438
           call at org/jruby/RubyProc.java:249
2012-10-23 22:07:54 +0200: Listen loop error: #<Errno::EBADF: Bad file descriptor - Bad file descriptor>

Reproducing this exact behavior is hard. However, i think i managed to break it down to the following steps (this time on Mac OSX):

  • Create a new empty rails app
  • Put gem 'puma', '1.6.3' in the gemfile and do a bundle install
  • Run bundle exec puma -b unix:/tmp/puma.sock
  • In another terminal run: socat TCP-LISTEN:1234,reuseaddr,fork UNIX-CLIENT:/tmp/puma.sock (this way we can leave nginx out of the equation)
  • Fire a few requests to http://localhost:1234 using your favorite browser and watch CPU load

On jruby-1.7.0-preview2 CPU load stays well under 1% for me.
JRuby-1.7.0 stable however, goes crazy.

This bug might aswell be a bug in the puma webserver.

I hope somebody will be able to reproduce this - i know it probably is a hassle.
I'm trying to find a more simple case that elaborates the problem without that many moving parts.

@ghost ghost was assigned Nov 17, 2012
@ghost
ghost commented Nov 17, 2012

There is definitely a problem in the unix socket code. I think it is the same underlying issue as #334

Until its fixed, I suggest using TCP sockets for the nginx <-> puma comms.

@ghost Unknown added a commit to jnr/jnr-enxio that referenced this issue Nov 17, 2012
Wayne Meissner When cancelling a SelectionKey, remove it from the selected key set. …
…Fixes jruby/jruby#398, and possibly jruby/jruby#334.
79639b1
@ghost Unknown added a commit that closed this issue Nov 17, 2012
Wayne Meissner Upgrade to jnr-enxio 0.3. Fixes #398 and possibly #334. 148ecc4
@ghost ghost closed this in 148ecc4 Nov 17, 2012
@prathamesh-sonpatki prathamesh-sonpatki added a commit to prathamesh-sonpatki/jruby that referenced this issue Dec 3, 2012
@prathamesh-sonpatki Wayne Meissner + prathamesh-sonpatki Upgrade to jnr-enxio 0.3. Fixes #398 and possibly #334. da5ae2c
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment