Skip to content

Commit

Permalink
Revert back to our fork of sidekiq-unique-jobs
Browse files Browse the repository at this point in the history
This reverts #927.  Sidekiq-unique-jobs 5.x requires redis 3.x but our
infrastructure uses 2.8.  We also have to use the fork rather than a
released version of 4.x because the last release of 4.x doesn't include
a fix for mhenrixon/sidekiq-unique-jobs#195
which means the `uniquejobs` hash key in redis never gets smaller.

Although there is a fix for this in 5.x (see:
https://github.com/mhenrixon/sidekiq-unique-jobs/pulls/200 - this
commit is what is on our fork of 4.x) it may have been changed to rely
on expiry features of redis 3.x that are not available in redis 2.x.

On staging this key is currently 6.5M entries, and consumes ~500Mb.
On production it's only 2.5M entries and consumes ~200Mb.
We tried running a (much simplified version of) this script:
https://gist.github.com/riyad/9086d2b17ff1e8c091cdb1c7ac501b62
in a screen session to remove any expired keys from this hash, but
unfortunately the rate of adding keys to the uniquejobs hash was
greater than the rate of removal.  Instead we waited until the queue
was drained and deleted the key.
  • Loading branch information
h-lame committed Mar 13, 2018
1 parent eac5dd2 commit cbcb805
Show file tree
Hide file tree
Showing 2 changed files with 14 additions and 6 deletions.
5 changes: 4 additions & 1 deletion Gemfile
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,10 @@ gem "diffy", "~> 3.1", require: false
gem "govspeak", "~> 5.5.0"
gem "hashdiff", "~> 0.3.6"
gem "json-schema", require: false
gem "sidekiq-unique-jobs", "~> 5.0", require: false
# We can't use v5 of this because it requires redis 3 and we use 2.8
# We use our own fork because the latest 4.x release has a bug with
# removing jobs from the uniquejobs hash in redis
gem "sidekiq-unique-jobs", git: "https://github.com/alphagov/sidekiq-unique-jobs", require: false
gem 'whenever', '0.10.0', require: false
gem "with_advisory_lock", "~> 3.1"

Expand Down
15 changes: 10 additions & 5 deletions Gemfile.lock
Original file line number Diff line number Diff line change
@@ -1,3 +1,11 @@
GIT
remote: https://github.com/alphagov/sidekiq-unique-jobs
revision: 6c03e39c3ad6ba1468d567c22bb99908f6fdb317
specs:
sidekiq-unique-jobs (4.0.18)
sidekiq (>= 2.6)
thor

GEM
remote: https://rubygems.org/
specs:
Expand Down Expand Up @@ -879,9 +887,6 @@ GEM
activesupport
sidekiq (>= 2.6)
statsd-ruby (>= 1.1.0)
sidekiq-unique-jobs (5.0.10)
sidekiq (>= 4.0, <= 6.0)
thor (~> 0)
simplecov (0.15.1)
docile (~> 1.1.0)
json (>= 1.8, < 3)
Expand Down Expand Up @@ -976,7 +981,7 @@ DEPENDENCIES
rails (= 5.1)
rspec
rspec-rails (~> 3.5)
sidekiq-unique-jobs (~> 5.0)
sidekiq-unique-jobs!
simplecov (= 0.15.1)
simplecov-rcov (= 0.2.3)
spring
Expand All @@ -989,4 +994,4 @@ DEPENDENCIES
with_advisory_lock (~> 3.1)

BUNDLED WITH
1.16.0
1.16.1

0 comments on commit cbcb805

Please sign in to comment.