delayed_job dies with database connection issues

[jconley]

Received this in our delayed_job.log... and then the DJ process dies. My best guess so far is there was a momentary network disruption between firewalls of our app and db networks.

<ActiveRecord::StatementInvalid: Mysql::Error: MySQL server has gone away: UPDATE delayed_jobs SET locked_by = null, locked_at = null WHERE (locked_by = 'delayed_job host:myhost pid:12345') >

Would be nice if DJ had more robust connection handling... i.e. catch this issue, and retry the connection until it is successful... otherwise we have to write a script to restart DJ periodically, or detect when jobs in the queue are stale.

[smasry]

Experiencing the same issue even when the db is on the same box.

[cyberfox]

Try setting 'reconnect: true' in your database.yml for the appropriate environment; under Rails 2.3.x that should alleviate the problem. (I'm setting that up now myself, so I'll know for sure in a few days, but it makes sense.)

[kelyar]

"reconnect: true" works, but leads to problems with transactions, so I don't think that is a proper solution

[cyberfox]

True, my use case was specific: the only database actions being done in the application were against the DJ table via DJ, so the issue with transactions was not relevant.

You could still do it if your DJ database was on a separate connection from your normal DB, but that certainly becomes more complex to set up.

[TeuF]

Same issue with rails 3 beta4 .... Can't make it work.

$ script/delayed_job start
delayed_job: process with pid 20648 started.

=> SQL (111.9ms) UPDATE delayed_jobs SET locked_by = null, locked_at = null WHERE (locked_by = 'delayed_job host:desk pid:20648')
Mysql::Error: Lost connection to MySQL server during query: SELECT delayed_jobs.* FROM delayed_jobs WHERE ((run_at <= '2010-08-06 14:26:41' AND (locked_at IS NULL OR locked_at < '2010-08-06 10:26:41') OR locked_by = 'delayed_job host:desk pid:20648') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC LIMIT 5

And when I start DJ with rake, every things working:

$ rake jobs:work
[Worker(host:desk pid:20663)] Starting job worker

=>SQL (24.5ms) SHOW TABLES
DEPRECATION WARNING: RAILS_DEFAULT_LOGGER is deprecated! Use Rails.logger instead. (called from say at /home/www/.bundle/ruby/1.8/bundler/gems/delayed_job-0ca4e5f2bd8a5e17dca7c1abab692a049c1669e1-rails3-mailer/lib/delayed/worker.rb:161)
2010-08-06T16:28:04+0200: [Worker(host:desk pid:20663)] Starting job worker

[threetee]

I'm seeing the same behavior with Rails 2.3.5 and the collectiveidea fork of delayed_job (version 2.0.3). When starting from script/delayed_job, it immediately dies with a "MySQL server has gone away" message, but when starting using the work:jobs rake task, it starts up properly.

Rails application log logs this in conjunction with the failed startup:

Delayed::Backend::ActiveRecord::Job Update (0.0ms) Mysql::Error: MySQL server has gone away: UPDATE delayed_jobs SET locked_by = null, locked_at = null WHERE (locked_by = 'delayed_job host:myhost pid:2293')
Mysql::Error: MySQL server has gone away: UPDATE delayed_jobs SET locked_by = null, locked_at = null WHERE (locked_by = 'delayed_job host:myhost pid:2293')

[kelyar]

Solution that I found somewhere on internet was to downgrade daemons gem (which is internal dependancy of DJ, I believe) from 1.1.0 to 1.0.10. That completely solved problems with dying delayed job.

thank you, Anonymous!

[smasry]

I've also experienced many "Mysql Server gone away errors" with activerecord. The fix was to increase the 'max_allowed_packet'. If the stacktrace for an error was too large mysql would terminate the connection.

[inspire22]

suckage, this is still an issue. I've always hated the daemons gem

[thehappycoder]

Are there any plans about this issue?

[bbozo]

Got this with ruby-oci8 and oracle-enhanced AR driver

[bacchir]

I also have this issue using ruby-oci8 and oracle-enhanced

[gregblass]

Running into this issue as well.

[cs0511]

+1

[xjlin0]

are there ways to retry upon database connection lost?