Ent Unique Jobs

If your application code creates duplicate jobs, the unique jobs feature in Faktory Enterprise makes it easy to ensure only a single copy of a job enqueued. For instance, perhaps you create a job to sync an address change with a 3rd party system every time a form is submitted. If the form is submitted twice, you don't need to create the second job if the first job is still pending.

Use

Every unique job must have a unique_for custom (Float|Integer) value which is the number of seconds your job should be considered unique:

// golang
job := faktory.NewJob("sometype", 1, 2)
job.SetCustom("unique_for", 600) # seconds
jid, err := faktory.Push(job)

This means that a second job can be pushed to Faktory after 10 minutes or after the first job has successfully processed. If your job retries for a while, 10 minutes can pass, thus allowing another copy of the same job to be pushed to Faktory. Design your jobs so that uniqueness is considered best effort, not a 100% guarantee A time limit is mandatory so that unique locks won't last forever.

Jobs are considered unique based on (class, args, queue), meaning a job with the same args can be pushed to different queues.

# ruby
class MyJob
  include Faktory::Job
  faktory_options custom: { unique_for: 10.minutes.to_i }
  def perform(...)
  end
end

The to_i is necessary as ActiveSupport::Duration in Rails serializes to JSON as a String "600" instead of a number so we need to manually convert it.

Scheduled Jobs

The uniqueness period for a scheduled job includes the delay time. If you use MyJob.perform_in(1.hour, ...), the uniqueness for this job will last 70 minutes (1 hour + the 10 minute unique_for TTL). You won't be able to push the same job until it runs successfully or 70 minutes have passed.

Retries

A job that is pending retry will still hold the unique lock and prevent further jobs from being enqueued until the retry succeeds or the timeout passes. Manually removing the job from the retry queue will not remove the lock.

Unlock Policy

The unique_until option allows you to control when the unique lock is removed. The default value is success: the job will not unlock until it executes successfully, it will remain locked even if it raises an error and goes into the retry queue.

The alternative value is start: the job will unlock right before it starts executing. This fixes the possibility of a race condition for some unique jobs between the finish of the job and unlocking.

// golang
job := faktory.NewJob("sometype", 1, 2)
job.SetCustom("unique_for", 600) # seconds
job.SetCustom("unique_until", "start")
jid, err := faktory.Push(job)

With the start policy, if the job raises an error, it will not try to retake the lock and may be duplicate enqueued. I recommend avoiding the start policy unless you know your job is affected by the race condition.

Bypassing Uniqueness

If you really want to push a job and bypass the uniqueness check, don't pass a unique_for value or pass nil.

# ruby
MyJob.set(custom: { unique_for: nil }).perform_async(1,2,3)

ActiveJob

The job uniqueness feature does not work with ActiveJob. You need to use native Faktory::Job instances.