[Request] sidekiq-cron-like enqueue guarantee

[arthurchui]

In a recent incident, a mission-critical daily cron was skipped because a leader just stepped down and a new leader was elected 40s later. According to the document of Ent Periodic Jobs, it is the expected behaviour:

This implementation does not support backfill. If Sidekiq is shutdown, it will not create jobs for the times missed on restart. I advise people to make their cron jobs resilient to timing. Instead of having an hourly job which processes only the last hour of data, make it process the last N hours.

However in our case, the task can't wait 24 hours till the next job runs. Alternatively, instead of running a daily job, we can run an hourly job that checks if the task has been done for the day. However it feels like a wrong paradigm for a task that is only executed once per day.

The gem sidekiq-cron records the last enqueue time and the poller guarantees to run a cron if the corresponding job hasn't been enqueued at the expected time. Can sidekiq-ent implement a similar feature for missing critical crons?

Possibly not all crons need the guarantee. An attribute reliable: true can be provided when it is mission critical:

  config.periodic do |mgr|
    mgr.register('0 * * * *', "SomeHourlyWorkerClass")
    mgr.register('* * * * *', "DailyDigestJob", reliable: true)
  end

Thoughts?

[mperham]

Something like this?

sidekiq_options retry: 0

# schedule frequently, runs daily
def perform(...)
  return "already done" if Sidekiq.redis { |c| c.get("my-lock") }
  # do work
  Sidekiq.redis { |c| c.set("my-lock", Time.now.to_s, "ex", 86400, "nx") }
end

Now I could wrap this functionality and expose it but three lines seems a minimal cost and you could wrap that into one helper method or module.