Revert to using sidekiq-unique-jobs gem #927
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We were previously using a fork of it for a bug fix.
h-lame
added a commit
that referenced
this pull request
Mar 13, 2018
This reverts #927. Sidekiq-unique-jobs 5.x requires redis 3.x but our infrastructure uses 2.8. We also have to use the fork rather than a released version of 4.x because the last release of 4.x doesn't include a fix for mhenrixon/sidekiq-unique-jobs#195 which means the `uniquejobs` hash key in redis never gets smaller. Although there is a fix for this in 5.x (see: https://github.com/mhenrixon/sidekiq-unique-jobs/pulls/200 - this commit is what is on our fork of 4.x) it may have been changed to rely on expiry features of redis 3.x that are not available in redis 2.x. On staging this key is currently 6.5M entries, and consumes ~500Mb. On production it's only 2.5M entries and consumes ~200Mb. We're running a (much simplified version of) this script: https://gist.github.com/riyad/9086d2b17ff1e8c091cdb1c7ac501b62 in a screen session to remove any expired keys from this hash.
h-lame
added a commit
that referenced
this pull request
Mar 13, 2018
This reverts #927. Sidekiq-unique-jobs 5.x requires redis 3.x but our infrastructure uses 2.8. We also have to use the fork rather than a released version of 4.x because the last release of 4.x doesn't include a fix for mhenrixon/sidekiq-unique-jobs#195 which means the `uniquejobs` hash key in redis never gets smaller. Although there is a fix for this in 5.x (see: https://github.com/mhenrixon/sidekiq-unique-jobs/pulls/200 - this commit is what is on our fork of 4.x) it may have been changed to rely on expiry features of redis 3.x that are not available in redis 2.x. On staging this key is currently 6.5M entries, and consumes ~500Mb. On production it's only 2.5M entries and consumes ~200Mb. We tried running a (much simplified version of) this script: https://gist.github.com/riyad/9086d2b17ff1e8c091cdb1c7ac501b62 in a screen session to remove any expired keys from this hash, but unfortunately the rate of adding keys to the uniquejobs hash was greater than the rate of removal. Instead we waited until the queue was drained and deleted the key.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We were previously using a fork of it for a bug fix.
I've given it a spin on integration and has been running without issues.