Bulk Queueing

The push_bulk method allows us to push a large number of jobs to Redis. This method cuts out the redis network round trip latency. I wouldn't recommend pushing more than 1000 per call but YMMV based on network quality, size of job args, etc. A large number of jobs can cause a bit of Redis command processing latency.

This method takes the same arguments as #push except that args is expected to be an Array of Arrays. All other keys are duplicated for each job. Each job is run through the client middleware pipeline and each job gets its own Job ID as normal.

It returns an array of the of pushed jobs' jids. The number of jobs pushed can be less than the number given if the middleware stopped processing for one or more jobs.

200.times do |idx|
  # each loader job will push 1000 jobs of some other type
  Loader.perform_async(idx)
end

class Loader
  include Sidekiq::Worker
  SIZE = 1000

  def perform(idx)
    # assume we want to create a job for each of 200,000 database records
    # query for our set of 1000 records
    results = SomeModel.limit(SIZE).offset(idx*SIZE).select(:id)
    # push 1000 jobs in one network call to Redis, saves 999 round trips
    Sidekiq::Client.push_bulk('class' => SomeJob, 'args' => results.map{|x| [x.id]})
  end
end

You can reference the push_bulk code in lib/sidekiq/client.rb:79-111

This functionality is only available if you use the Sidekiq workers API; it does not work if you use the ActiveJob wrappers, for example.