New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Obtain advisory lock before updating or deleting jobs #28

Merged
merged 1 commit into from Oct 5, 2017

Conversation

Projects
None yet
3 participants
@owst
Collaborator

owst commented Oct 5, 2017

If we have a failed Que job and press the re-run button when the job is being retried by a worker, we end up scheduling another instance of the job. Similarly, if we try to delete a previously-failed job and it is already being retried, we won't actually delete the job despite the flash telling us that is was deleted.

Both of these scenarios are confusing, and should be avoided (and a warning shown to the user).

The former occurs as when we reschedule a running job, we change the primary key of the job as it is a composite of: (queue, priority, run_at, job_id) and by rescheduling we change the run_at value. Therefore, when the original (running) job completes and deletes the row(s) matching its primary key, the newly-created row (with the different primary key) won't be deleted.

To fix both scenarios, if we try to take the advisory lock for the job before updating/deleting, we will only reschedule/delete if the job isn't already being worked. This PR adds this attempted locking and shows a flash to accurately describe to the user what has happened.

N.B. to demonstrate the issue a job such as:

class SleepJob < Que::Job
  def run
    ActiveRecord::Base.transaction do
      raise 'error first time!' if error_count.zero?

      10.downto(1) do |i|
        print "#{i}..."
        sleep 1
      end

      destroy
    end
  end
end

will error first time, if the reschedule button is pressed when it starts running, (i.e. printing 10...9...) another job will be enqueued. Similarly if the delete button is pressed when the job runs, it won't actually be deleted and will continue to be run.

@statianzo statianzo merged commit 4ae98a7 into statianzo:master Oct 5, 2017

@statianzo

This comment has been minimized.

Owner

statianzo commented Oct 5, 2017

Great catch. Thanks for the contribution @owst

@arronmabrey

This comment has been minimized.

arronmabrey commented Oct 12, 2017

@statianzo would it be possible to get a ruby gems release of this fix? (if its ready, that is 😄)

@statianzo

This comment has been minimized.

Owner

statianzo commented Oct 12, 2017

Published as 0.6.0

@arronmabrey

This comment has been minimized.

arronmabrey commented Oct 12, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment