Don't reuse existing job when spawning a scheduled job #43
Conversation
Since any job with non-inifinte result_ttl will be eventually cleaned up by RQ, it makes it impossible to use periodic jobs with finite or zero ttl. This change changes saved job into a template to store args, timeout and ttl, but actual executed job will be different so that the template will not be affected.
|
Thanks for the PR :) I'm closing this for now because of reasons explained in #42 (comment) |
|
I am testing different failure scenarios in rq-scheduler and found it fairly easy to get it into weird states because of the re-using of job IDs. Here's an example of it silently & permanently deleting a scheduled_job under what I'd consider normal operations: I was able to do this by scheduling a job once per minute, running two workers, having the job sleep for longer than 60 seconds, then waiting 2-3 minutes so the jobs overlapped. I'm concerned that sharing the same job ID for each run leads to race conditions & edge cases that aren't being handled. These would be solved by following @extesy's idea of using the schedule as a template and enqueueing a separate job for each run. I think that the basic correctness of running & scheduling the jobs outweighs the reasons given in #42 (e.g. that it's nicer for monitoring). |
|
Here's how to subclass rq_scheduler.Scheduler and override enqueue_job with @extesy's change: https://gist.github.com/lost-theory/2e94ae20643ac32a6773 That way you don't need to maintain a fork. Just run that file instead of |
Since any job with non-inifinte result_ttl will be eventually cleaned up by RQ, it makes it impossible to use periodic jobs with finite or zero ttl. This change changes saved job into a template to store args, timeout and ttl, but actual executed job will be different so that the template will not be affected.
Fixes #42