Middleware

Evan Prothro edited this page Aug 29, 2018 · 43 revisions

Sidekiq has a similar notion of middleware to Rack: these are small bits of code that can implement functionality. Sidekiq breaks middleware into client-side and server-side.

  • Client middleware runs before the pushing of the job to Redis and allows you to modify/stop the job before it gets pushed.
  • Server middleware runs 'around' job processing.

Custom middleware

Sidekiq 5+ ships with no middleware out of the box. Various gems and services will install middleware to track jobs or provide additional features. You may also add your own.

Client middleware

You can use client middleware to add job metadata to the job before pushing it to Redis. The same job data will be available to the server middleware before the job is executed, if you need to set up some global state, e.g. current locale, current tenant in a multi-tenant app, etc.

Client middleware may receive the class argument as a Class object or a String containing the name of the class.

Not calling yield or returning false or nil will result in no further middleware being called and the job will not be pushed to the queue.

class Middleware::Sidekiq::Client::CustomerJobAttribute
  # @param [String, Class] worker_class the string or class of the worker class being enqueued
  # @param [Hash] job the full job payload
  #   * @see https://github.com/mperham/sidekiq/wiki/Job-Format
  # @param [String] queue the name of the queue the job was pulled from
  # @param [ConnectionPool] redis_pool the redis pool
  # @return [Hash, FalseClass, nil] if false or nil is returned,
  #   the job is not to be enqueued into redis, otherwise the block's
  #   return value is returned
  # @yield the next middleware in the chain or the enqueuing of the job
  def call(worker_class, job, queue, redis_pool)
    # return false/nil to stop the job from going to redis
    return false if queue != 'default'
    job['customer'] = Customer.current_id
    yield
  end
end

Server middleware

You can use server middleware to do something around the execution of any job. The example below logs any exception from any job.

Not calling yield will result in no further middleware being call and the job's perform method will not be called.

class Middleware::Sidekiq::Server::ErrorLogger
  # @param [Object] worker the worker instance
  # @param [Hash] job the full job payload
  #   * @see https://github.com/mperham/sidekiq/wiki/Job-Format
  # @param [String] queue the name of the queue the job was pulled from
  # @yield the next middleware in the chain or worker `perform` method
  # @return [Void]
  def call(worker, job, queue)
    begin
      yield
    rescue => ex
      puts ex.message
    end
  end
end

You may also pass options to the Middleware initializer when you register the middleware:

class Middleware::Sidekiq::Server::ErrorLogger
  def initialize(options=nil)
    # options == { :foo => 1, :bar => 2 }
  end
  def call(worker, msg, queue)
    begin
      yield
    rescue => ex
      puts ex.message
    end
  end
end

Registering Middleware

Register your middleware as part of the chain:

Sidekiq.configure_client do |config|
  config.client_middleware do |chain|
    chain.add Middleware::Sidekiq::Client::CustomerJobAttribute
  end
end

Sidekiq.configure_server do |config|
  config.server_middleware do |chain|
    chain.add Middleware::Sidekiq::Server::ErrorLogger, :foo => 1, :bar => 2
  end
end

I'd suggest putting this code in config/initializers/sidekiq.rb in your Rails app.

Client middleware registered in both places

The jobs running in the Sidekiq server can themselves push new jobs to Sidekiq, thus acting as clients. You must configure your client middleware within the configure_server block also in that case:

Sidekiq.configure_client do |config|
  config.client_middleware do |chain|
    chain.add Middleware::Sidekiq::Client::CustomerJobAttribute
  end
end

Sidekiq.configure_server do |config|
  config.client_middleware do |chain|
    chain.add Middleware::Sidekiq::Client::CustomerJobAttribute
  end
  config.server_middleware do |chain|
    chain.add Middleware::Sidekiq::Server::ErrorLogger, :foo => 1, :bar => 2
  end
end

Removing middleware

If you need to remove a middleware for some reason, you can do this in your configuration:

Sidekiq.configure_server do |config|
  config.server_middleware do |chain|
    chain.remove Some::Middleware
  end
end

Inspecting middleware ordering on sidekiq boot

Sidekiq will print out the configured client and server middlewares when you start it with -v.

Please note any middleware in the chain after the non-yielded block will not run. And depending on the middleware you may need to pay particular attention the middleware ordering.

Miscellaneous

Obscure Scenario: If you share a single redis instance between multiple services, and use client middleware on your Sidekiq server, be warned that it might have to deal with workers from unexpected queues. Sidekiq uses a singe redis set to handle scheduled workers, and any server might requeue a worker that's due to be processed, even if it's not for a queue it was configured to process. When it does so, your middleware might be passed a worker name that does not exist in the current ruby context and you might see uninitialized constant exceptions or similar errors.

https://stackoverflow.com/questions/39780767/sidekiq-delete-or-kill-job-inside-a-middleware?rq=1

Previous: API Next: Resque Compatibility

You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.