Experiencing frequent crashing with 1.1.5

[wengzilla]

@zerowidth - I just tried upgrading to the latest ringleader, but have been experiencing a bunch of crashes on regular page requests... On the plus side, the hanging issue (Issue #25) is not affecting me. See below:

ERROR 14:34:52.137634 | Ringleader::App crashed!
ArgumentError: this IO is already registered with selector
/Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/celluloid-io-0.14.0/lib/celluloid/io/reactor.rb:42:in register' /Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/celluloid-io-0.14.0/lib/celluloid/io/reactor.rb:42:inwait'
/Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/celluloid-io-0.14.0/lib/celluloid/io/reactor.rb:26:in wait_writable' /Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/celluloid-io-0.14.0/lib/celluloid/io.rb:47:inwait_writable'
/Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/celluloid-io-0.14.0/lib/celluloid/io/stream.rb:36:in wait_writable' /Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/celluloid-io-0.14.0/lib/celluloid/io/stream.rb:66:inrescue in block in syswrite'
/Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/celluloid-io-0.14.0/lib/celluloid/io/stream.rb:63:in block in syswrite' /Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/celluloid-io-0.14.0/lib/celluloid/io/stream.rb:399:insynchronize'
/Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/celluloid-io-0.14.0/lib/celluloid/io/stream.rb:61:in syswrite' /Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/celluloid-io-0.14.0/lib/celluloid/io/stream.rb:356:indo_write'
/Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/celluloid-io-0.14.0/lib/celluloid/io/stream.rb:249:in write' /Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/ringleader-1.1.5/lib/ringleader/app.rb:139:incopy_stream'
/Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/ringleader-1.1.5/lib/ringleader/app.rb:139:in proxy' /Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/celluloid-0.14.0/lib/celluloid/calls.rb:25:inpublic_send'
/Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/celluloid-0.14.0/lib/celluloid/calls.rb:25:in dispatch' /Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/celluloid-0.14.0/lib/celluloid/calls.rb:125:indispatch'
/Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/celluloid-0.14.0/lib/celluloid/actor.rb:328:in block in handle_message' /Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/celluloid-0.14.0/lib/celluloid/tasks.rb:42:inblock in initialize'
/Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/celluloid-0.14.0/lib/celluloid/tasks/task_fiber.rb:9:in `block in create'

[zerowidth]

This is known issue, per recent discussion in #celluloid. Sounds like @halorgium's working on a fix but it's not there yet.

[zerowidth]

Found it: celluloid/celluloid-io#23.

Closing until upstream gets a fix.

[halorgium]

@zerowidth @wengzilla I would appreciate confirmation of whether that PR fixes the issue.
I believe it is solid, I need to come up with a simple spec as regressing would suck!
Please let me know if you can try it or not.

[zerowidth]

@wengzilla mind giving that PR on celluloid-io a try? I don't have a decent enough environment running right now in which I could test this.

@halorgium unfortunately I don't know yet of a simple way of triggering this, it just seemed to happen on (some) high volume of requests. I've tried causing it with ab and similar, but the only way I reliably saw it was with a high-volume workload on a real set of apps at my previous job. If you've got any ideas, I'm happy to try 'em out...

[halorgium]

@zerowidth you can trigger it if you fill the kernel buffers, which isn't super efficient to do.
I am mostly interested in another solved case.

@tarcieri should we worry too much the regression test? It is a somewhat hot code path, but the change is not specifically tested.

[zerowidth]

Gah, I suck at commenting.

@halorgium, how hard is it to reproduce the "fill the kernel buffers" case? I'm less worried about efficiency than having a reproducible load test on ringleader.

[halorgium]

@zerowidth doing lots of big async writes.
https://gist.github.com/halorgium/b717914f490f4ba4df5d

[wengzilla]

Definitely not an opensource gem building guru, so I probably missed a step... What do I need to do to test this out?

1. I pulled down the celluloid-io branch
2. I then ran
   gem build celluloid-io.gemspec
3. I pulled down ringleader
4. I modified the ringleader Gemspec and added this line
   gem "celluloid-io", :path => 'path_to/celluloid-io'
5. I then ran `gem build ringleader.gemspec
6. I installed the gem in my global gemset via
   gem install /path_to_ringleader_gem/ringleader-1.1.5.gem
7. I then started ringleader as per usual and on startup, I immediately got the following error:

NoMethodError: undefined method owner=' for #<Celluloid::Condition:0x007fbffcd560b0> /Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/celluloid-io-0.14.0/lib/celluloid/io/stream.rb:385:insynchronize'
/Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/celluloid-io-0.14.0/lib/celluloid/io/stream.rb:42:in sysread' /Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/celluloid-io-0.14.0/lib/celluloid/io/stream.rb:128:inreadpartial'
/Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/reel-0.3.0/lib/reel/request.rb:19:in read' /Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/reel-0.3.0/lib/reel/connection.rb:52:inrequest'
/Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/ringleader-1.1.5/lib/ringleader/server.rb:16:in on_connection' /Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/reel-0.3.0/lib/reel/server.rb:27:in[]'
/Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/reel-0.3.0/lib/reel/server.rb:27:in handle_connection' /Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/celluloid-0.14.1/lib/celluloid/calls.rb:25:inpublic_send'
/Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/celluloid-0.14.1/lib/celluloid/calls.rb:25:in dispatch' /Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/celluloid-0.14.1/lib/celluloid/calls.rb:125:indispatch'
/Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/celluloid-0.14.1/lib/celluloid/actor.rb:326:in block in handle_message' /Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/celluloid-0.14.1/lib/celluloid/tasks.rb:42:inblock in initialize'
/Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/celluloid-0.14.1/lib/celluloid/tasks/task_fiber.rb:11:in `block in create'

[tarcieri]

Sorry, this is a bit of a versioning snafu on our part... celluloid 0.14.1 probably should've been 0.15. You're running an out-of-date version of celluloid-io (0.14.0) which is incompatible with Celluloid 0.14.1.

Please try the gem releases of both celluloid 0.4.1 and celluloid-io 0.14.1.

[wengzilla]

Got it to work with celluloid and celluloid-io both on version 0.14.1, but still seeing the selector issue...
Here is what my modified Ringleader gemspec looks like: http://bit.ly/13lNiNj

Will continue to try to use for today and tomorrow and see if the error is persistent.

ERROR 16:02:15.535417 | Ringleader::App crashed!
ArgumentError: this IO is already registered with selector
/Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/celluloid-io-0.14.1/lib/celluloid/io/reactor.rb:42:in register' /Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/celluloid-io-0.14.1/lib/celluloid/io/reactor.rb:42:inwait'
/Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/celluloid-io-0.14.1/lib/celluloid/io/reactor.rb:26:in wait_writable' /Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/celluloid-io-0.14.1/lib/celluloid/io.rb:47:inwait_writable'
/Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/celluloid-io-0.14.1/lib/celluloid/io/stream.rb:36:in wait_writable' /Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/celluloid-io-0.14.1/lib/celluloid/io/stream.rb:66:inrescue in block in syswrite'
/Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/celluloid-io-0.14.1/lib/celluloid/io/stream.rb:63:in block in syswrite' /Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/celluloid-io-0.14.1/lib/celluloid/io/stream.rb:390:insynchronize'
/Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/celluloid-io-0.14.1/lib/celluloid/io/stream.rb:61:in syswrite' /Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/celluloid-io-0.14.1/lib/celluloid/io/stream.rb:356:indo_write'
/Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/celluloid-io-0.14.1/lib/celluloid/io/stream.rb:249:in write' /Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/ringleader-1.1.5/lib/ringleader/app.rb:139:incopy_stream'
/Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/ringleader-1.1.5/lib/ringleader/app.rb:139:in proxy' /Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/celluloid-0.14.1/lib/celluloid/calls.rb:25:inpublic_send'
/Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/celluloid-0.14.1/lib/celluloid/calls.rb:25:in dispatch' /Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/celluloid-0.14.1/lib/celluloid/calls.rb:125:indispatch'
/Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/celluloid-0.14.1/lib/celluloid/actor.rb:326:in block in handle_message' /Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/celluloid-0.14.1/lib/celluloid/tasks.rb:42:inblock in initialize'
/Users/edwardweng/.rvm/gems/ruby-1.9.3-p392/gems/celluloid-0.14.1/lib/celluloid/tasks/task_fiber.rb:11:in `block in create'

[tarcieri]

@halorgium do you think this should be fixed on master?

[tarcieri]

Trying to fix this upstream:

celluloid/celluloid-io#23