uniquejobs hash doesn't get cleaned up

[xjlu]

The versions we are using in one service is

gem 'sidekiq', '4.1.4'
gem 'sidekiq-unique-jobs', '4.0.18'

We use both until_executed and until_timeout. The uniquejob* keys are generated and cleaned up properly, but it leaves a copy of mapping inside the uniquejobs hash, and it keeps growing. This problem started exactly at 4.1.2 (we tested various version around that time). And it seems still a problem with the latest version 4.1.8.

The problem here is that the hash size grows to such an extent that it eventually fills up the redis memory. Normal metrics such as the number of keys don't help debug this issue.

The internal structure of the hash is

["0", [["e820b3a5db2bfe94ea686c10", "uniquejobs:2782395ef5070394f7d5bbf7a6c8b934"], ["1d51c1ce36fa7efda2356a07", "uniquejobs:1b8d30e357006dbf484fbec15d5ffab9"], ["79dc6c691669d857d561875a", "uniquejobs:629965e9a1a162901f6b22918d5ebcbb"], ["90531aeb90f014cb00f17349", "uniquejobs:1b8d30e357006dbf484fbec15d5ffab9"], ["9e6710a97fdfb3f5438477e4", "uniquejobs:1b8d30e357006dbf484fbec15d5ffab9"],
...

[tyrauber]

We are seeing the same problem:

gem 'sidekiq', '~> 4.2.3'
gem 'sidekiq-unique-jobs', '~> 4.0.13'

Periodically, we get a memory error with redis because of this:

Production Redis::CommandError: OOM command not allowed when used memory > 'maxmemory'.

And have to run a rake task to clear these jobs:

require 'sidekiq'
Sidekiq.redis do |conn|
  puts 'Removing uniquejobs:* keys'
  conn.keys('uniquejobs:*').each do |key|
    conn.del(key)
  end

  puts 'Removing uniquejobs hash'
  conn.del('uniquejobs')
end

[riyad]

We've come across the same problem.

I've tried to put together a more careful/verbose script (i.e. still not foolproof). It tries to do a few checks before actually deleting data. https://gist.github.com/riyad/9086d2b17ff1e8c091cdb1c7ac501b62

[girak]

@mhenrixon idea how to fix this? we have been holding on upgrading to versions > 4.0.11

[agarcher]

Any idea when this fix will be in a versioned release?

[agarcher]

So great that the fix is released! 🎉

We have deployed it and confirmed our memory footprint has stopped growing. Now we are turning our minds to clean-up. Has anyone else invested effort into a script that will look through redis and clean out orphaned uniquejobs? With 5.0.0, the leak is fixed, but the already-leaked-keys are not cleaned up.

[mhenrixon]

I haven't looked into this yet. It would be totally awesome to have the possibility to clean up. Maybe I can find some time to do this over easter.

Should be a matter of matching the hashes jids against the jids that are in redis but... there could be an easter 🥚 hidden somewhere.

[agarcher]

There is #195 (comment) but won't that wipe out uniquejobs that are actually still relevant?

[agarcher]

@mhenrixon Any update on a clean up solution? Our Redis instance is sitting at a steady state of 600MB of bogus keys. I've been letting it sit there hoping I will "get something for free" to clean it out. Should there be a separate issue to track this?