[PRO] Performance of cleanup_the_dead and check_for_orphans

[justin808]

Per this issue: #3196 (comment)

Can these be scheduled to run every minute? Or would there be performance issues, and thus it's worth checking if they need to be run?

    dead_qty = Sidekiq.options[:fetch].cleanup_the_dead
    orphaned_qty = Sidekiq.options[:fetch].check_for_orphans

[mperham]

cleanup_the_dead can run frequently as it iterates over known keys. It will run up to once per minute.

check_for_orphans scans the entire redis keyspace for orphaned queues which is an extremely rare but possible edge case (if a sidekiq process crashes with queue XYZ and you immediately deploy a new code version which never references that queue XYZ again, Sidekiq won't know the queue still exists with orphaned jobs in it. sort of like garbage collection where there's no more references to an object so you have to scan the entire object space to find garbage). This scan can take a fair amount of time depending on the app's scale and Redis usage patterns. It will run up to once per hour.

But both of these things only run upon the start of a Sidekiq process and I don't make any further guarantees. One of the reasons why you don't want to run them too frequently is poison pills. You don't want to recover poison pills too quickly otherwise your app can get into a death spiral where the Sidekiq processes are dying quickly due to frequent retries.

[justin808]

@mperham what happens if check_for_orphans runs twice concurrently? Do you have a check in the code to see if this is running and to then skip the concurrent run? Or should we store a flag in Redis that we're still running it?

[mperham]

You have access to the source. You can see it yourself. bundle open sidekiq-pro