Skip to content

Commit

Permalink
Use our fork of sidekiq jobs v4.0.18
Browse files Browse the repository at this point in the history
Sidekiq-unique-jobs 5.x requires redis 3.x but our infrastructure uses
2.8.  We also have to use the fork rather than a released version of 4.x
because the last release of 4.x doesn't include a fix for
mhenrixon/sidekiq-unique-jobs#195 which means the `uniquejobs` hash key
in redis never gets smaller.

Although there is a fix for this in 5.x (see:
https://github.com/mhenrixon/sidekiq-unique-jobs/pulls/200 - this
commit is what is on our fork of 4.x) it may have been changed to rely
on expiry features of redis 3.x that are not available in redis 2.x.

On staging this key is currently 2.5M entries, and consumes ~200Mb.
On production it's only 1.2M entries and consumes ~100Mb.
We tried running a (much simplified version of) this script:
https://gist.github.com/riyad/9086d2b17ff1e8c091cdb1c7ac501b62
in a screen session to remove any expired keys from this hash, but
unfortunately the rate of adding keys to the uniquejobs hash was
greater than the rate of removal.  Instead we waited until the queue
was drained and deleted the key.
  • Loading branch information
h-lame committed Mar 13, 2018
1 parent 06c220f commit 5253837
Show file tree
Hide file tree
Showing 2 changed files with 14 additions and 6 deletions.
5 changes: 4 additions & 1 deletion Gemfile
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,10 @@ gem "faraday-cookie_jar", "~> 0.0.6"

gem "govuk_sidekiq", "~> 3.0"
gem "sidekiq-scheduler", "~> 2.2"
gem "sidekiq-unique-jobs", "~> 5.0"
# We can't use v5 of this because it requires redis 3 and we use 2.8
# We use our own fork because the latest 4.x release has a bug with
# removing jobs from the uniquejobs hash in redis
gem "sidekiq-unique-jobs", git: "https://github.com/alphagov/sidekiq-unique-jobs", require: false

gem "activerecord-import", "~> 0.22"

Expand Down
15 changes: 10 additions & 5 deletions Gemfile.lock
Original file line number Diff line number Diff line change
@@ -1,3 +1,11 @@
GIT
remote: https://github.com/alphagov/sidekiq-unique-jobs
revision: 6c03e39c3ad6ba1468d567c22bb99908f6fdb317
specs:
sidekiq-unique-jobs (4.0.18)
sidekiq (>= 2.6)
thor

GEM
remote: https://rubygems.org/
specs:
Expand Down Expand Up @@ -283,9 +291,6 @@ GEM
activesupport
sidekiq (>= 2.6)
statsd-ruby (>= 1.1.0)
sidekiq-unique-jobs (5.0.10)
sidekiq (>= 4.0, <= 6.0)
thor (~> 0)
simplecov (0.15.1)
docile (~> 1.1.0)
json (>= 1.8, < 3)
Expand Down Expand Up @@ -357,7 +362,7 @@ DEPENDENCIES
rspec-rails (~> 3.4)
rspec-sidekiq (~> 3.0)
sidekiq-scheduler (~> 2.2)
sidekiq-unique-jobs (~> 5.0)
sidekiq-unique-jobs!
simplecov (~> 0.11)
simplecov-rcov (~> 0.2)
sqlite3
Expand All @@ -366,4 +371,4 @@ DEPENDENCIES
webmock

BUNDLED WITH
1.16.0
1.16.1

0 comments on commit 5253837

Please sign in to comment.