Timeout when publishing message

[perlun]

Hi,

I keep getting these now, during high load:

#<Celluloid::TimeoutError: receive timeout exceeded>
/Users/plundberg/git/uxf/ecraft.uxfactory.server-2014.1/src/server/gems/jruby/1.9/gems/celluloid-0.16.0/lib/celluloid/mailbox.rb:82:in `receive'
/Users/plundberg/git/uxf/ecraft.uxfactory.server-2014.1/src/server/gems/jruby/1.9/gems/celluloid-0.16.0/lib/celluloid/calls.rb:97:in `wait'
org/jruby/RubyKernel.java:1517:in `loop'
/Users/plundberg/git/uxf/ecraft.uxfactory.server-2014.1/src/server/gems/jruby/1.9/gems/celluloid-0.16.0/lib/celluloid/calls.rb:96:in `wait'
/Users/plundberg/git/uxf/ecraft.uxfactory.server-2014.1/src/server/gems/jruby/1.9/gems/celluloid-0.16.0/lib/celluloid.rb:116:in `suspend'
/Users/plundberg/git/uxf/ecraft.uxfactory.server-2014.1/src/server/gems/jruby/1.9/gems/celluloid-0.16.0/lib/celluloid/calls.rb:88:in `response'
/Users/plundberg/git/uxf/ecraft.uxfactory.server-2014.1/src/server/gems/jruby/1.9/gems/celluloid-0.16.0/lib/celluloid/calls.rb:92:in `value'
/Users/plundberg/git/uxf/ecraft.uxfactory.server-2014.1/src/server/gems/jruby/1.9/gems/celluloid-0.16.0/lib/celluloid/proxies/sync_proxy.rb:33:in `method_missing'
/Users/plundberg/git/uxf/ecraft.uxfactory.server-2014.1/src/server/gems/jruby/1.9/gems/celluloid-0.16.0/lib/celluloid/notifications.rb:8:in `publish'

The calling code looks like this:

          publish(/applog/, create_log_entry(request, status, headers, body))

During low load, everything works fine, but under high load, I get this all the time. Any ideas? When I looked at calls.rb, it seems to call Celluloid.mailbox.receive without any timeout, which I interpret as the timeout as impossible? But OTOH, I'm not very familiar with the Celluloid codebase, so... I could have missed something. Is there a default timeout involved here?

(Using the latest Celluloid gem from Rubygems. Haven't tried with bleeding edge from GitHub).

[philipmw]

Looks like you're using JRuby. This may be the same root cause as issue #475.

[digitalextremist]

Adding to 0.17.0 as I am able to reproduce #475 myself now.

[digitalextremist]

@e2 this was re-introduced for me lately. Was there anything you remember touching that could have caused this to resurface?

[e2]

O gosh, this looks EXACTLY like the probe specs problem I'm trying to deal with here: #554

[e2]

@digitalextremist, @perlun - I need a failing example. I couldn't create one (I tried), and I don't have it occurring currently. I have a few ideas what it could be - so if someone has an example that fails on jruby-head, I can debug it...

[digitalextremist]

Mine is failing on 1.7.19 ... working to isolate it.

[e2]

@digitalextremist - can you reproduce it consistently enough to test something?

[digitalextremist]

Not yet. I have a piece of code that relaunches dead pools. I suspect it might be actually trying to remake the pool with bad arguments like a variation of that smoker issue.

[e2]

@digitalextremist - if you can reproduce problems, try these patches: https://github.com/abstractive/celluloid/commit/5de8adbb1967c5ad11b397a1a2619fa7cb308710

If they still occur, let me know - I can't seem to get JRuby failing after these.

(The above patches are preventative, and not a fix - you might even get a speedup in actor messaging).

[digitalextremist]

This, for me, was resolved at #585 by magic.

@perlun: flag me down if this is still an issue with 0.17.0-prerelease

[perlun]

Nice! Thanks for your hard work.