ActiveRecord::ConnectionNotEstablished on Heroku

[jamonholmgren]

Hi there,

I think Que is amazing. Nice work.

But I'm getting a ActiveRecord::ConnectionNotEstablished crash when using it on Heroku. Here's the backtrace:

$ heroku run 'rake db:migrate'
Running `rake db:migrate` attached to terminal... up, run.6525
rake aborted!
ActiveRecord::ConnectionNotEstablished: ActiveRecord::ConnectionNotEstablished
/app/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.3/lib/active_record/connection_handling.rb:109:in `connection_pool'
/app/vendor/bundle/ruby/2.1.0/gems/que-0.8.1/lib/que/adapters/active_record.rb:30:in `checkout_activerecord_adapter'
/app/vendor/bundle/ruby/2.1.0/gems/que-0.8.1/lib/que/adapters/active_record.rb:5:in `checkout'
/app/vendor/bundle/ruby/2.1.0/gems/que-0.8.1/lib/que/job.rb:82:in `work'
/app/vendor/bundle/ruby/2.1.0/gems/que-0.8.1/lib/que/worker.rb:78:in `block in work_loop'
/app/vendor/bundle/ruby/2.1.0/gems/que-0.8.1/lib/que/worker.rb:73:in `loop'
/app/vendor/bundle/ruby/2.1.0/gems/que-0.8.1/lib/que/worker.rb:73:in `work_loop'
/app/vendor/bundle/ruby/2.1.0/gems/que-0.8.1/lib/que/worker.rb:17:in `block in initialize'
(See full trace by running task with --trace)

My configuration:

    config.que.worker_count = 1
    config.que.wake_interval = 10.minutes
    config.que.mode = :async

It works fine if I change it to config.que.mode = :off. This is a single Heroku web dyno. Everything works fine locally.

Any ideas?

[jamonholmgren]

Note that it also works fine if I run:

$ heroku run 'QUE_WORKER_COUNT=1 rake que:work'

[jamonholmgren]

I was able to find a workaround. Since this is possibly related to #47, I decided to try putting the Que config lines in an initializer file rather than directly in application.rb.

# /config/initializers/que.rb
Que.worker_count = 1
Que.wake_interval = 10.minutes
Que.mode = :async

This appears to be working.

As a follow-up question, is there a way to prevent normal rake tasks from starting up Que workers? Otherwise I get Que output every time I run rake db:migrate on Heroku.

[jamonholmgren]

Hm, now I'm getting another intermittent problem.

$ heroku run 'rake db:migrate'
Running `rake db:migrate` attached to terminal... up, run.9547
{"lib":"que","hostname":"51647c12-81b9-41f9-a6c7-32793d95d389","pid":2,"thread":70331113231360,"event":"worker_count_change","value":"1"}
{"lib":"que","hostname":"51647c12-81b9-41f9-a6c7-32793d95d389","pid":2,"thread":70331113231360,"event":"mode_change","value":"async"}
/app/vendor/bundle/ruby/2.1.0/gems/que-0.8.1/lib/que/adapters/base.rb:69: [BUG] Segmentation fault at 0x00000000000114
ruby 2.1.2p95 (2014-05-08 revision 45877) [x86_64-linux]

And lots more: https://gist.github.com/jamonholmgren/e863f6402d5407eedacb

It appears that this happens when I run a rake task at the same time that Que is waking up the workers (I changed the wakeup to 10.seconds).

I am also noticing that perhaps the order of the config settings affects this. When changing the order to the following:

Que.wake_interval = 10.seconds
Que.mode = :async
Que.worker_count = 1

...I haven't been able to replicate the above issue. Not sure if this is a coincidence or not.

Anyway, the idea behind Que is great and I have high hopes for it, but just need to figure out how to make it work well on Heroku.

[chanks]

Segfaults are bugs in Ruby, and there's not a lot we can do other than make a reproducible test case and report it. Que does run into them more often than other gems, because it tends to stress Ruby's threading system, which seems to be more bug-prone. If you can make a reproducible case that reliably results in a segfault, I'd certainly follow it with great interest. I've tried to make reproductions a few times and always given up before sinking too much time into them - they're tricky. I would wager that any apparent effects of one process on another are coincidental, though - their only mechanism of interaction would be via the job locking query, and its behavior is very well defined.

The last time I checked, Rails didn't offer a good way to check whether the currently running process is a server or a rake task or a console or a runner or whatever else. This is one of the reasons why the Railtie is the single most unreliable and prone-to-breakage part of the gem, and why I'm continuously tempted to break it off into its own gem and try to dump it on someone who's more interested in Rails internals than I am.

But none of that is helpful to you ;) I'll try to dig into this when I can, but I expect most of my free time over the next two weeks to be taken up by moving into a new place, so it may not be soon. If you or anyone else wants to take the time to dig into it, feel free, and I'll be happy to offer advice and answer questions when I can. If it helps, the only reason I can think of why setting up Que in an initializer would be functionally different than setting it up in application.rb is that those two configurations would be processed at different points in the application boot process. But again, I don't know enough about Rails internals to know what why you're seeing that specific effect from it.

[jamonholmgren]

Thanks, @chanks , very helpful reply. I'll keep playing with it.

In the meantime, I've written a short blog post about getting up and running quickly on Que: http://jamonholmgren.com/easy-background-tasks-on-heroku-with-que

Let me know if I've missed anything important on there if you do get a chance in between moving. :)

[joevandyk]

You can have the job automatically requeue itself, there's a document in
the docs folder that explains how.

On Friday, August 22, 2014, Jamon Holmgren notifications@github.com wrote:

Thanks, @chanks https://github.com/chanks , very helpful reply. I'll
keep playing with it.

In the meantime, I've written a short blog post about getting up and
running quickly on Que:
http://jamonholmgren.com/easy-background-tasks-on-heroku-with-que

Let me know if I've missed anything important on there if you do get a
chance in between moving. :)

—
Reply to this email directly or view it on GitHub
#54 (comment).

[jamonholmgren]

I did read that. I kind of like the scheduled task, though...feels cleaner and more like how I'm used to things working.