Obtain advisory lock before updating or deleting jobs

[owst]

If we have a failed Que job and press the re-run button when the job is being retried by a worker, we end up scheduling another instance of the job. Similarly, if we try to delete a previously-failed job and it is already being retried, we won't actually delete the job despite the flash telling us that is was deleted.

Both of these scenarios are confusing, and should be avoided (and a warning shown to the user).

The former occurs as when we reschedule a running job, we change the primary key of the job as it is a composite of: (queue, priority, run_at, job_id) and by rescheduling we change the run_at value. Therefore, when the original (running) job completes and deletes the row(s) matching its primary key, the newly-created row (with the different primary key) won't be deleted.

To fix both scenarios, if we try to take the advisory lock for the job before updating/deleting, we will only reschedule/delete if the job isn't already being worked. This PR adds this attempted locking and shows a flash to accurately describe to the user what has happened.

N.B. to demonstrate the issue a job such as:

class SleepJob < Que::Job
  def run
    ActiveRecord::Base.transaction do
      raise 'error first time!' if error_count.zero?

      10.downto(1) do |i|
        print "#{i}..."
        sleep 1
      end

      destroy
    end
  end
end

will error first time, if the reschedule button is pressed when it starts running, (i.e. printing 10...9...) another job will be enqueued. Similarly if the delete button is pressed when the job runs, it won't actually be deleted and will continue to be run.

[statianzo]

Great catch. Thanks for the contribution @owst

[arronmabrey]

@statianzo would it be possible to get a ruby gems release of this fix? (if its ready, that is 😄)

[statianzo]

Published as 0.6.0

[arronmabrey]

Great 👍 thx!

…

On Thu, Oct 12, 2017 at 5:23 PM Jason Staten ***@***.***> wrote: Published as 0.6.0 <https://rubygems.org/gems/que-web/versions/0.6.0> — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#28 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAH1KvZuaucuFnvHIdo-KXB0bFyCa89Uks5sroNSgaJpZM4Pvcxu> .