Obtain advisory lock before updating or deleting jobs #28
Conversation
Great catch. Thanks for the contribution @owst |
@statianzo would it be possible to get a ruby gems release of this fix? (if its ready, that is |
Published as 0.6.0 |
Great
…On Thu, Oct 12, 2017 at 5:23 PM Jason Staten ***@***.***> wrote:
Published as 0.6.0 <https://rubygems.org/gems/que-web/versions/0.6.0>
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#28 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAH1KvZuaucuFnvHIdo-KXB0bFyCa89Uks5sroNSgaJpZM4Pvcxu>
.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
If we have a failed Que job and press the re-run button when the job is being retried by a worker, we end up scheduling another instance of the job. Similarly, if we try to delete a previously-failed job and it is already being retried, we won't actually delete the job despite the flash telling us that is was deleted.
Both of these scenarios are confusing, and should be avoided (and a warning shown to the user).
The former occurs as when we reschedule a running job, we change the primary key of the job as it is a composite of:
(queue, priority, run_at, job_id)
and by rescheduling we change therun_at
value. Therefore, when the original (running) job completes and deletes the row(s) matching its primary key, the newly-created row (with the different primary key) won't be deleted.To fix both scenarios, if we try to take the advisory lock for the job before updating/deleting, we will only reschedule/delete if the job isn't already being worked. This PR adds this attempted locking and shows a flash to accurately describe to the user what has happened.
N.B. to demonstrate the issue a job such as:
will error first time, if the reschedule button is pressed when it starts running, (i.e. printing
10...9...
) another job will be enqueued. Similarly if the delete button is pressed when the job runs, it won't actually be deleted and will continue to be run.