Validating Queue Names

[stevenharman]

Anyone else here every typo'd a queue name and then been left smashing your face into the keyboard for an hour until you realized the typo? Just me? 😄

Our Sidekiq setup has weighted queues which we enumerate in a single place, like this:

# config/sidekiq_queues
SIDEKIQ_QUEUES = {
  'within_30_seconds' => 5, # 50% of the time we check this queue
  'within_5_minutes' => 2,  # 20% of the time
  'within_1_hour' => 2,     # 20% of the time
  'within_24_hours' => 1    # 10% of the time
}.freeze

Our Sidekiq config.yml then enumerates that Hash to build the list of queues to monitor. This works great b/c we can keep the weights and names in source control, right next to the jobs. And we don't screw up typing them into the YAML config.

But this didn't stop me from typo-ing a queue name for a new job. I'd like to remedy that. I'm thinking of adding something to our base Job class, along these lines:

require_relative '../../config/sidekiq_queues'

class ApplicationJob
  include Sidekiq::Job

  InvalidQueueName = Class.new(StandardError)

  def self.sidekiq_options(opts = {})
    opts = opts.transform_keys { |key| String(key) }
    queue = String(opts.fetch('queue', nil))

    raise InvalidQueueName, "Job #{self.class.name} must specify `sidekiq_option queue:`" unless queue.empty?
    raise InvalidQueueName, "'#{queue}' is not a valid queue name for our Sidekiq configuration." unless SIDEKIQ_QUEUES.keys.include?(queue)

    super(opts)
  end
end

What do you think? Am I building a foot gun I'll come to regret? Is there a way to do this that's not dependent on a base job class (or mixin) - like some way to add this to Sidekiq proper?

Thanks!

[mperham]

Queue names are an application concern. Any validation belongs in your code. I don’t see this as likely to change but if you have a proposal, you’re welcome to pitch the idea.

[stevenharman]

Oh, yeah totally agree that queue names are an app concern. I was more thinking about a mechanism whereby you could both declare the "valid" queue names, and by way of declaring them, Sidekiq would auto-magically validate them. But I have no clue what that looks like. 😄

At this point, I'm interested in thoughts on whether or not the above technique is a Good Idea™, or a footgun.

[mperham]

I suppose you could register a queue name validator with Sidekiq but I'm not sure when it would be called?

[stevenharman]

Good point. With the technique I took above, the validation happens at job definition time. Which hopefully means stuff blows up in tests, and so is caught early!

I'll think on it some more, but maybe the ad-hoc validation as I have here is the best course for now.

Thanks, Mike!