Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

If beanstalkd dies, ::Forking workers never reconnect #93

Open
mexisme opened this issue Apr 28, 2015 · 4 comments
Open

If beanstalkd dies, ::Forking workers never reconnect #93

mexisme opened this issue Apr 28, 2015 · 4 comments

Comments

@mexisme
Copy link

mexisme commented Apr 28, 2015

When using the ::Forking worker, I'm seeing errors like the below when beanstalkd is stopped or dies (e.g. server restart).
The worker fails over and over, and never makes a fresh connection.

Exception Beaneater::NotConnected -> Could not connect!
   .bundle/ruby/2.0.0/bundler/gems/beaneater-6eaa016693bd/lib/beaneater/pool.rb:154:in `rescue in safe_transmit'
   .bundle/ruby/2.0.0/bundler/gems/beaneater-6eaa016693bd/lib/beaneater/pool.rb:141:in `safe_transmit'
   .bundle/ruby/2.0.0/bundler/gems/beaneater-6eaa016693bd/lib/beaneater/pool.rb:109:in `transmit_to_rand'
   .bundle/ruby/2.0.0/bundler/gems/beaneater-6eaa016693bd/lib/beaneater/pool_command.rb:40:in `method_missing'
   .bundle/ruby/2.0.0/bundler/gems/beaneater-6eaa016693bd/lib/beaneater/tube/collection.rb:44:in `reserve'
   .bundle/ruby/2.0.0/gems/backburner-0.4.6/lib/backburner/worker.rb:128:in `work_one_job'
   .bundle/ruby/2.0.0/gems/backburner-0.4.6/lib/backburner/workers/forking.rb:33:in `block in fork_one_job'
   .bundle/ruby/2.0.0/gems/backburner-0.4.6/lib/backburner/workers/forking.rb:31:in `fork'
   .bundle/ruby/2.0.0/gems/backburner-0.4.6/lib/backburner/workers/forking.rb:31:in `fork_one_job'
   .bundle/ruby/2.0.0/gems/backburner-0.4.6/lib/backburner/workers/forking.rb:25:in `block in start'
   .bundle/ruby/2.0.0/gems/backburner-0.4.6/lib/backburner/workers/forking.rb:25:in `loop'
   .bundle/ruby/2.0.0/gems/backburner-0.4.6/lib/backburner/workers/forking.rb:25:in `start'
   .bundle/ruby/2.0.0/gems/backburner-0.4.6/lib/backburner/worker.rb:48:in `start'
   .bundle/ruby/2.0.0/gems/backburner-0.4.6/lib/backburner.rb:39:in `work'

I have a suspicion it has something to do with conflation between https://github.com/nesquena/backburner/blob/master/lib/backburner/worker.rb#L58
where you're using a class var (@connection) but in https://github.com/nesquena/backburner/blob/master/lib/backburner/workers/forking.rb#L32 you appear to be trying to override an instance var (also @connection) and yet https://github.com/nesquena/backburner/blob/master/lib/backburner/worker.rb#L126 uses the above self.connection class method (and class var).

Does this mean that all the forked children are trying to reuse the parent's @connection and it's no-longer valid, because the associated beanstalkd has gone/restarted?

It's possible this may be related to #61?

Thanks.

@contentfree
Copy link
Collaborator

@mexisme I see what you're saying with the conflation bit there. What it should probably do is pass the newly-minted Connection into the work_one_job method (which accepts an optional Connection instance). The Worker doesn't seem to have/use a @connection instance var.

@contentfree
Copy link
Collaborator

@nesquena Memoizing the connection as a static/class variable seems to be causing several problems. What do you think about moving it strictly to an instance variable? Then no more problems like this.

@nesquena
Copy link
Owner

nesquena commented Oct 4, 2015

@contentfree 👍 makes sense, thanks for tracking this down

@mexisme
Copy link
Author

mexisme commented Oct 15, 2015

Thanks, @contentfree, that looks like a smarter way forward.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants