Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DrbConnError when Transaction Events Array too large #56

Closed
serioushaircut opened this issue Aug 5, 2015 · 1 comment
Closed

DrbConnError when Transaction Events Array too large #56

serioushaircut opened this issue Aug 5, 2015 · 1 comment

Comments

@serioushaircut
Copy link

Hi Appsignal, hi @thijsc,

back to this error from #38 and #39, but now with better knowledge.
Recap the setup - Appsignal 0.11.11 and Rails with a single vanilla Resque worker that does 35k times a DB lookup, e.g. User.find('dontexist').
Recap the stacktrace:

Exception DRb::DRbConnError 
Error Broken pipe
/usr/lib/ruby/2.1.0/drb/drb.rb:611:in `write'
/usr/lib/ruby/2.1.0/drb/drb.rb:611:in `send_request'
/usr/lib/ruby/2.1.0/drb/drb.rb:921:in `send_request'
/usr/lib/ruby/2.1.0/drb/drb.rb:1221:in `send_message'
/usr/lib/ruby/2.1.0/drb/drb.rb:1110:in `block (2 levels) in method_missing'
/usr/lib/ruby/2.1.0/drb/drb.rb:1197:in `open'
/usr/lib/ruby/2.1.0/drb/drb.rb:1109:in `block in method_missing'
/usr/lib/ruby/2.1.0/drb/drb.rb:1128:in `with_friend'
/usr/lib/ruby/2.1.0/drb/drb.rb:1108:in `method_missing'
/app/.bundle/ruby/2.1.0/gems/appsignal-0.11.9/lib/appsignal/ipc.rb:59:in `enqueue'
/app/.bundle/ruby/2.1.0/gems/appsignal-0.11.9/lib/appsignal/transaction.rb:160:in `complete!'
/app/.bundle/ruby/2.1.0/gems/appsignal-0.11.9/lib/appsignal/transaction.rb:26:in `complete_current!'
/app/.bundle/ruby/2.1.0/gems/appsignal-0.11.9/lib/appsignal.rb:92:in `monitor_transaction'
/app/.bundle/ruby/2.1.0/gems/appsignal-0.11.9/lib/appsignal/integrations/resque.rb:8:in `around_perform_resque_plugin'

Unfortunately it does not state what caused the error, but happy debugging revealed there is a default load_limit for messages over drbunix connections of 25MB! I stepped through the code to the point in ruby 2.1.0 drb.rb line 611 with varying lengths of the messages to be send. And tadaa its true to the byte.

In Appsignal terms that means the Appsignal::Transaction (= the object to be marshalled) which is too large (in marshaled form > 25MB) triggers this error.
The transaction object gets too large quite easily - as stated before - when a worker job has to do a bit.
Our workaround was and still is that we clear the events queue of the transaction object as last statement in the resque jobs of which we know to trigger the error

Appsignal::Transaction.current.clear_events!

This load_limit of DRb could theoretically be configured, but not transparently via Appsignal config, at least I didn't find a way.

Cheers,
Baschtl

@thijsc
Copy link
Member

thijsc commented Feb 11, 2016

We've completely overhauled this system in the 1.0 version of our gem. This should now work like a charm.

@thijsc thijsc closed this as completed Feb 11, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants