Killed Jobs are not requeued on Heroku

[dim]

I looked at #51, but the re-queueing doesn't seem to work on Heroku (using HEAD). This is from the logs:

2012-03-06T08:21:42+00:00 app[worker.1]: 2012-03-06T08:21:42Z 1 TID-aeam4 INFO: Shutting down
2012-03-06T08:21:42+00:00 app[worker.1]: 2012-03-06T08:21:42Z 1 TID-ow6lw INFO: Pausing 5 seconds to allow workers to finish...
2012-03-06T08:21:52+00:00 heroku[worker.1]: Error R12 (Exit timeout) -> Process failed to exit within 10 seconds of SIGTERM
2012-03-06T08:21:52+00:00 heroku[worker.1]: Stopping process with SIGKILL
2012-03-06T08:21:54+00:00 heroku[worker.1]: Process exited with status 137

Unfortunately, nothing was re-queued. Every restart/deploy on heroku terminates the workers, meaning we are just loosing all running threads. Any ideas?

[mperham]

Only workers who cleanly shut down have their jobs requeued.

[dim]

Hmm, but how I do that, if I have long-running jobs? Ideally, I would have some sort of terminate call-back or an exception I could rescue in the worker's perform method. I also don't want to re-queue the whole job, but e.g. just the outstanding part. Any suggestions for an implementation? I don't mind trying one myself, just looking for ideas.

[fbjork]

Agree. Jobs that get kill should be re-queued.

[mperham]

Dupe of #51

[be9]

Workaround: put Celluloid.sleep(0) inside your long-running loop in Worker#perform.