Ability to reset retries count

[andrusha]

If you have, say 1000 records you want to import through remote API, it might take significant amount of time and sometimes API might be unresponsive and because it might be inaccessible for unknown period time, job throws an exception and restarts until it can proceed, so thread is not occupied by stale job.
Here is the catch, if API is unresponsive for one hour there would be ~7 retries and if after that we import some records successfully and API is unresponsive again retry_count would be equal 8 and job will have to wait ~1 hour before next try and so on, which is unoptimal.

What I want to have is ability to reset retry_count inside Worker, currently I monkey-patch RetryJob middleware in rather ugly manner:

Sidekiq::Middleware::Server::RetryJobs.class_eval do
  include Storage

  alias_method :old_call, :call

  def call(worker, msg, queue, &block)
    msg.delete 'retry_count'  if storage.get(worker.jid, :reset_retries) == 'true'

    old_call worker, msg, queue, &block
  end
end

class Importer
  include Storage

  def perform(items)
    completed = 0
    items.each do |item|
      api.import item
      completed += 1
    end
  ensure
    storage.set @jid, reset_retries: (completed > 0)
  end
end

Ideally I would like to have something like this:

class Importer
  include RetryJob::Worker

  def perform(items)
    completed = 0
    items.each do |item|
      api.import itme
      completed += 1
    end
  ensure
    reset_retries  if completed > 0
  end
end

Any suggestions on how to implement this?

[mperham]

Implement your own retry system using scheduling and server middleware.

[andrusha]

All right. May ask another question, is there a way to pass variable from worker to middleware without saving/loading it to/from db?

[mperham]

Not easily but there are ways if you are crafty and don't mind a hack. Middleware is designed to be orthogonal to workers.