💎 Resque::UniqueByArity

if ci_badges.map(&:color).detect { it != "green"} ☝️ let me know on Discord or RubyForum, as I may have missed the notification.

if ci_badges.map(&:color).all? { it == "green"} 👇️ send money so I can do more of this. FLOSS maintenance is now my full-time job.

resque-unique_by_arity composes queue-time and runtime uniqueness and lets each job choose which leading perform arguments define its uniqueness signature. It supports queue-local uniqueness, cross-queue uniqueness, and runtime uniqueness without hand-written Redis key methods.

Use it when a job has ancillary arguments, such as options or tracing values, that should not create a separate unit of work.

Tokens to Remember
Works with JRuby
Works with Truffle Ruby
Works with MRI Ruby 4
Works with MRI Ruby 3
Works with MRI Ruby 2
Support & Community
Source
Documentation
Compliance
Style
Maintainer 🎖️
... 💖	🧊 🐙 🛖 🧪

Compatible with MRI Ruby 2.3.0+, and concordant releases of JRuby, and TruffleRuby. CI workflows and Appraisals are generated for MRI Ruby 2.4+. This test floor is configured by ruby.test_minimum in .kettle-jem.yml and may be higher than the gem's runtime compatibility floor when legacy Rubies are not practical for the current toolchain.

The amazing test matrix is powered by the kettle-dev stack.

How kettle-dev manages complexity in tests

Gem	Source	Role	Daily download rank
appraisal2	GitHub	multi-dependency Appraisal matrix generation
appraisal2-rubocop	GitHub	RuboCop Appraisal generator integration
kettle-dev	GitHub	development, release, and CI workflow tooling
kettle-jem	GitHub	Appraisals & CI workflow templates
kettle-soup-cover	GitHub	SimpleCov coverage policy and reporting
kettle-test	GitHub	standard test runner and coverage harness
rubocop-lts	GitHub	Ruby-version-aware linting
turbo_tests2	GitHub	parallel test execution

Available as part of the Tidelift Subscription.

Install the gem and add to the application's Gemfile by executing:

bundle add resque-unique_by_arity

If bundler is not being used to manage dependencies, install the gem by executing:

gem install resque-unique_by_arity

Set application defaults with Resque::UniqueByArity.configure, then include a configured plugin instance in each job:

class RefreshAccount
  @queue = :accounts

  include Resque::Plugins::UniqueByArity.new(
    arity_for_uniqueness: 1,
    unique_in_queue: true,
    unique_at_runtime: true,
    arity_validation: :error
  )
end

arity_for_uniqueness is the number of leading arguments used for the key. Use arity_for_uniqueness_at_runtime, arity_for_uniqueness_in_queue, or arity_for_uniqueness_across_queues when an enforcement axis needs a different arity. Choose either unique_in_queue or unique_across_queues, never both. arity_validation accepts :warning, :error, :skip, or nil/false.

The following is showing the default values. These global configs are copied into each per-class config unless they are overridden by the class config.

Create an initializer (e.g. config/initializers/resque-unique_by_arity.rb for rails) and customize the following:

Resque::UniqueByArity.configure do |config|
  config.logger = nil
    config.log_level = :debug
    config.arity_for_uniqueness = 0
    config.unique_at_runtime = false
    config.unique_in_queue = false
    # No need to do the following if keeping default values
    config.runtime_lock_timeout = 60 * 60 * 24 * 5
    config.runtime_requeue_interval = 1
    config.unique_at_runtime_key_base = "r-uar".freeze
    config.lock_after_execution_period = 0
    config.ttl = -1
    config.unique_in_queue_key_base = "r-uiq".freeze
    # Debug Mode is preferably set via an environment variable:
    #   to one of 'true', 'arity', or 'arity,queue,runtime' for all three tools:
    #     ENV['RESQUE_DEBUG'] = 'true'
    # config.debug_mode = true
end

This gem will take care to set the class instance variables (similar to the familiar @queue class instance variable) that are utilized by resque-unique_in_queue and resque-unique_at_runtime (default values shown):

# For resque-unique_at_runtime
@runtime_lock_timeout = 60 * 60 * 24 * 5
@runtime_requeue_interval = 1
@unique_at_runtime_key_base = 'r-uar'.freeze

# For resque-unique_in_queue
@lock_after_execution_period = 0
@ttl = -1
@unique_in_queue_key_base = 'r-uiq'.freeze

All you need to do is configure this gem accordingly:

  include Resque::Plugins::UniqueByArity.new(
    arity_for_uniqueness: 1,
    # Turn on one or both of the following:
    unique_at_runtime: false,
    unique_in_queue: false,
    # No need to do the following if keeping default values
    runtime_lock_timeout: 60 * 60 * 24 * 5,
    runtime_requeue_interval: 1,
    # would override the global setting, probably a bad idea.
    # unique_at_runtime_key_base: 'r-uar'.freeze,
    lock_after_execution_period: 0,
    ttl: -1,
    # would override the global setting, probably a bad idea.
    # unique_in_queue_key_base: 'r-uiq'.freeze
  )

Some jobs have parameters that you do not want to consider for determination of uniqueness. Resque jobs should use simple parameters, not named parameters, so you can just specify the number of parameters, counting from the left, you want to be considered for uniqueness.

class MyJob
  def self.perform(my, cat, is, the, best, opts = {})
    # Only the first 3: [my, cat, is] will be considered for determination of uniqueness
  end
  include Resque::Plugins::UniqueByArity.new(
    arity_for_uniqueness: 3,
    unique_at_runtime: true
  )
end

Want this gem to tell you when it is misconfigured? It can.

class MyJob
  def self.perform(my, cat, opts = {})
    # Because the third argument is optional the arity valdiation will not approve.
    # Arguments to be considered for uniqueness should be required arguments.
    # The warning log might look like:
    #
    #    MyJob.perform has the following required parameters: [:my, :cat], which is not enough to satisfy the configured arity_for_uniqueness of 3
  end
  include Resque::Plugins::UniqueByArity.new(
    arity_for_uniqueness: 3,
    arity_validation: :warning, # or :skip, :error, or an error class to be raised, e.g. RuntimeError
    unique_at_runtime: true
  )
end

Give the job a break after it finishes running, and don't allow another of the same, with matching args @ configured arity, to start within X seconds.

class MyJob
  def self.perform(arg1)
    # do stuff
  end
  include Resque::Plugins::UniqueByArity.new(
    arity_for_uniqueness: 1,
    lock_after_execution_period: 60,
    unique_at_runtime: true
  )
end

If runtime lock keys get stale, they will expire on their own after some period. You can set the expiration period on a per class basis.

class MyJob
  def self.perform(arg1)
    # do stuff
  end
  include Resque::Plugins::UniqueByArity.new(
    arity_for_uniqueness: 1,
    runtime_lock_timeout: 60 * 60 * 24 * 5, # 5 days
    unique_at_runtime: true
  )
end

Prevent your app from running a job that is already running.

class MyJob
  def self.perform(arg1)
    # do stuff
  end
  include Resque::Plugins::UniqueByArity.new(
    arity_for_uniqueness: 1,
    unique_at_runtime: true
  )
end

Preventing jobs with matching signatures from running, and they never get dequeued because there is no actual corresponding job to dequeue.

How to deal?

MyJob.purge_unique_at_runtime_redis_keys

Prevent your app from queueing a job that is already queued in the same queue.

class MyJob
  def self.perform(arg1)
    # do stuff
  end
  include Resque::Plugins::UniqueByArity.new(
    arity_for_uniqueness: 1,
    unique_in_queue: true
  )
end

Prevent your app from queueing a job that is already queued in any queue.

class MyJob
  def self.perform(arg1)
    # do stuff
  end
  include Resque::Plugins::UniqueByArity.new(
    arity_for_uniqueness: 1,
    unique_across_queues: true
  )
end

Preventing jobs with matching signatures from being queued, and they never get dequeued because there is no actual corresponding job to dequeue.

How to deal?

Option: Rampage

# Delete *all* queued jobs in the queue, and
#   delete *all* uniqueness keys for the queue.
Redis.remove_queue('queue_name')

Option: Butterfly

# Delete *no* queued jobs at all, and
#   delete *all* uniqueness keys for the queue (might then allow duplicates).
Resque::UniqueInQueue::Queue.cleanup('queue_name')

Prevent your app from running a job that is already running, and prevent your app from queueing a job that is already queued in the same queue.

class MyJob
  def self.perform(arg1)
    # do stuff
  end
  include Resque::Plugins::UniqueByArity.new(
    arity_for_uniqueness: 1,
    unique_at_runtime: true,
    runtime_lock_timeout: 60 * 60 * 24 * 5, # 5 days
    unique_in_queue: true
  )
end

Prevent your app from running a job that is already running, and prevent your app from queueing a job that is already queued in any queue.

class MyJob
  def self.perform(arg1)
    # do stuff
  end
  include Resque::Plugins::UniqueByArity.new(
    arity_for_uniqueness: 1,
    unique_at_runtime: true,
    runtime_lock_timeout: 60 * 60 * 24 * 5, # 5 days
    unique_across_queues: true
  )
end

Run your worker with RESQUE_DEBUG=true to see payloads printed before they are used to determine uniqueness, as well as a lot of other debugging output.

Redefine methods to customize all the things. Warning: This might be crazy-making.

class MyJob
  def self.perform(arg1)
    # do stuff
  end
  include Resque::Plugins::UniqueByArity.new(
    #...
  )

  # Core hashing algorithm for a job used for *all 3 types* of uniqueness
  # @return [Array<String, arguments>], where the string is the unique digest, and arguments are the specific args that were used to calculate the digest
  def self.redis_unique_hash(payload, arity_for_uniqueness = 1)
    #       for how the built-in version works
    # uniqueness_args = payload["args"] # over simplified & ignoring arity
    # args = { class: job, args: uniqueness_args }
    # return [Digest::MD5.hexdigest(Resque.encode(args)), uniqueness_args]
  end

  def self.unique_in_queue_redis_key_prefix
    # "unique_job:#{self}" # <= default value
  end

  def self.unique_in_queue_redis_key(queue, payload)
    # arity_for_uniqueness = determine_arity # over simplified & ignoring context-specific arity determination
    # unique_hash, _args_for_uniqueness = redis_unique_hash(payload, arity_for_uniqueness)
    # "#{unique_in_queue_key_namespace(queue)}:#{unique_in_queue_redis_key_prefix}:#{unique_hash}"
  end

  def self.unique_in_queue_key_namespace(queue = nil)
    # definition depends on which type of uniqueness is chosen, be careful if you customize
    # "r-uiq:queue:#{queue}:job" # <= is for unique within queue at queue time
    # "r-uiq:across_queues:job" # <= is for unique across all queues at queue time
  end

  def self.runtime_key_namespace
    # "unique_at_runtime:#{self}"
  end

  def self.unique_at_runtime_redis_key(*args)
    # payload = {"class" => self.to_s, "args" => args}
    # unique_hash, _args_for_uniqueness = redis_unique_hash(payload, configuration.arity_for_uniqueness_at_runtime)
    # key = "#{runtime_key_namespace}:#{unique_hash}" # <= simplified default
  end
end

While resque tools are free software and will always be, the project would benefit immensely from some funding. Raising a monthly budget of... "dollars" would make the project more sustainable.

We welcome both individual and corporate sponsors! We also offer a wide array of funding channels to account for your preferences. Currently, Open Collective is our preferred funding platform.

If you're working in a company that's making significant use of resque tools we'd appreciate it if you suggest to your company to become a resque sponsor.

You can support the development of resque tools via GitHub Sponsors, Liberapay, PayPal, Open Collective and Tidelift.

Support us with a monthly donation and help us continue our activities. [Become a backer]

NOTE: kettle-readme-backers updates this list every day, automatically.

No backers yet. Be the first!

Become a sponsor and get your logo on our README on GitHub with a link to your site. [Become a sponsor]

NOTE: kettle-readme-backers updates this list every day, automatically.

No sponsors yet. Be the first!

I’m driven by a passion to foster a thriving open-source community – a space where people can tackle complex problems, no matter how small. Revitalizing libraries that have fallen into disrepair, and building new libraries focused on solving real-world challenges, are my passions. I was recently affected by layoffs, and the tech jobs market is unwelcoming. I’m reaching out here because your support would significantly aid my efforts to provide for my family, and my farm (11 🐔 chickens, 2 🐶 dogs, 3 🐰 rabbits, 8 🐈‍ cats).

If you work at a company that uses my work, please encourage them to support me as a corporate sponsor. My work on gems you use might show up in bundle fund.

I’m developing a new library, floss_funding, designed to empower open-source developers like myself to get paid for the work we do, in a sustainable way. Please give it a look.

Floss-Funding.dev: 👉️ No network calls. 👉️ No tracking. 👉️ No oversight. 👉️ Minimal crypto hashing. 💡 Easily disabled nags

See SECURITY.md.

If you need some ideas of where to help, you could work on adding more code coverage, or if it is already 💯 (see below) check issues or PRs, or use the gem and think about how it could be better.

We so if you make changes, remember to update it.

See CONTRIBUTING.md for more detailed instructions.

See CONTRIBUTING.md.

Everyone interacting with this project's codebases, issue trackers, chat rooms and mailing lists agrees to follow the .

Made with contributors-img.

Also see GitLab Contributors: https://gitlab.com/resque/resque-unique_by_arity/-/graphs/main

⭐️ Star History

This library follows for its public API where practical. For most applications, prefer the Pessimistic Version Constraint with two digits of precision.

For example:

spec.add_dependency("resque-unique_by_arity", "~> 3.0")

See CHANGELOG.md for a list of releases.

The gem is available as open source under the terms of the MIT .

See LICENSE.md for the official copyright notice.

Maintainers have teeth and need to pay their dentists. After getting laid off in an RIF in March, and encountering difficulty finding a new one, I began spending most of my time building open source tools. I'm hoping to be able to pay for my kids' health insurance this month, so if you value the work I am doing, I need your support. Please consider sponsoring me or the project.

To join the community or get help, use the RubyForum or Discord.

To say "thanks!" ☝️ Join the community or 👇️ send money.

💌 💌 💌

Many parts of this project are actively managed by a kettle-jem smart template utilizing StructuredMerge.org merge contracts.

Thanks for RTFM. ☺️