New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Runaway checks hit redis at 2600 requests per second within 30 minutes of starting #10
Comments
This has just happened again. As I take down pods and deploy new ones it just runs away:
|
This just happened again - we are sorting out our new production and had 30 minutes of downtime. Upon startup, sidekiq was slammed with 58,000 |
It looks like that problem is when new version of pod is deploying:
I think #7 should help to limit this old pod's job to execute only ~60 times. |
Another case where this problem may be occured is when there is several replicas of one sidekiq and one of them is stuck. Then it's job will be scheduled to another pods and them will be ping-pong it until hostname timed out. |
Hi, Sorry for the delay. |
Hi, I think on what do you think? |
Hi @rosskevin, |
Version 1.0.1 released. Should solve this issue. |
This should be reopened (verified on 1.0.1):
This is a brand new cluster/business with essentially no traffic other than alpha users. The spike occurs on a new deployment (old one deleted, new one added) What information could I provide that can help? |
@rosskevin Sorry for my late reply. Yes, something similar was happening on my cluster too. Even with my PR, I experienced spikes of 150k jobs on an empty cluster. Last week, I had to remove sidekiq-alive, since my cluster keeps on crashing (in fact I had to remove the HTTP health check as well). I will have to investigate whether this problem occurs due to sidekiq-alive or a GKE update or whatever. I will try v1.0.1 from @arturictus (Thanks for #12) and see how it goes. |
Reopen this. |
already open |
It looks like the problem persists in some or another way. I think the problem is because of no sidekiqctl in binary anymore. Any ideas how to handle this problem? |
Hi @arrowcircle did you try with version 2. |
* Add release and changelog workflow * Set custom label for actions deps
* Add release and changelog workflow * Set custom label for actions deps
* Create dependabot.yml * Remove deprecated options call on config * Clean up logging and add ability to set custom logger * Clean up code and add shutdown callback * Regenerate gemfile * Correctly set sidekiq queues * Update workflow * Add github-actions to dep updates * Add lockfile platforms * Update version to 2.2.0 * Bump actions/checkout from 2 to 3 (#7) Bumps [actions/checkout](https://github.com/actions/checkout) from 2 to 3. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v2...v3) --- updated-dependencies: - dependency-name: actions/checkout dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Rename gem to sidekiq-alive (#8) * Add release and changelog workflow (#10) * Add release and changelog workflow * Set custom label for actions deps * Fix setting custom sidekiq queue (#9) * Move all configuration in to startup block * Check respond to on config object * Update specs * Run specs in random order * Set git identity for release workflow * Rename gem to sidekiq-alive-next * Update README.md * Create github release in the same job * Update gemspec metadata (#11) * Update gemspec metadata * Require ruby 2.7 * Add static rubocop checks (#12) * Add static rubocop checks * Allign arguments * Automate release (#13) * Add rake task to update version * Add version update steps * Unset deployment mode for release * Add release ssh key for release workflow * Checkout tag in release * Add additional details to startup info (#14) * Use scan for finding and purging jobs (#15) * Add changelog workflow (#16) * Bump rspec from 3.11.0 to 3.12.0 (#17) Bumps [rspec](https://github.com/rspec/rspec-metagem) from 3.11.0 to 3.12.0. - [Release notes](https://github.com/rspec/rspec-metagem/releases) - [Commits](rspec/rspec-metagem@v3.11.0...v3.12.0) --- updated-dependencies: - dependency-name: rspec dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Sidekiq 7 support (#19) * Update sidekiq requirement from >= 5, < 7 to >= 5, < 8 Updates the requirements on [sidekiq](https://github.com/mperham/sidekiq) to permit the latest version. - [Release notes](https://github.com/mperham/sidekiq/releases) - [Changelog](https://github.com/mperham/sidekiq/blob/main/Changes.md) - [Commits](sidekiq/sidekiq@v6.5.7...v7.0.0) --- updated-dependencies: - dependency-name: sidekiq dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * Update for sidekiq 7 compatibility Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update README.md * Update changelog.yml * Use ruby 3.1.2 for development * Replace pry with debug gem * Update rack requirement from >= 2.2.4, < 3 to >= 3, < 4 (#20) * Update rack requirement from >= 2.2.4, < 3 to >= 2.2.4, < 4 Updates the requirements on [rack](https://github.com/rack/rack) to permit the latest version. - [Release notes](https://github.com/rack/rack/releases) - [Changelog](https://github.com/rack/rack/blob/main/CHANGELOG.md) - [Commits](rack/rack@2.2.4...3.0.0) --- updated-dependencies: - dependency-name: rack dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * Add rackup gem and improve tests * Narrow down rack requirement Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Andrejs Cunskis <andrejs.cunskis@gmail.com> * Update nokogiri and workflow ruby setup * Bump sidekiq from 7.0.0 to 7.0.1 (#21) Bumps [sidekiq](https://github.com/mperham/sidekiq) from 7.0.0 to 7.0.1. - [Release notes](https://github.com/mperham/sidekiq/releases) - [Changelog](https://github.com/mperham/sidekiq/blob/main/Changes.md) - [Commits](sidekiq/sidekiq@v7.0.0...v7.0.1) --- updated-dependencies: - dependency-name: sidekiq dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump rackup from 0.2.2 to 0.2.3 (#22) Bumps [rackup](https://github.com/rack/rackup) from 0.2.2 to 0.2.3. - [Release notes](https://github.com/rack/rackup/releases) - [Commits](rack/rackup@v0.2.2...v0.2.3) --- updated-dependencies: - dependency-name: rackup dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump rack from 3.0.0 to 3.0.1 (#23) Bumps [rack](https://github.com/rack/rack) from 3.0.0 to 3.0.1. - [Release notes](https://github.com/rack/rack/releases) - [Changelog](https://github.com/rack/rack/blob/main/CHANGELOG.md) - [Commits](rack/rack@3.0.0...v3.0.1) --- updated-dependencies: - dependency-name: rack dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump sidekiq from 7.0.1 to 7.0.2 (#24) Bumps [sidekiq](https://github.com/mperham/sidekiq) from 7.0.1 to 7.0.2. - [Release notes](https://github.com/mperham/sidekiq/releases) - [Changelog](https://github.com/mperham/sidekiq/blob/main/Changes.md) - [Commits](sidekiq/sidekiq@v7.0.1...v7.0.2) --- updated-dependencies: - dependency-name: sidekiq dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Loosen rackup requirement (#25) * Bump rack from 3.0.1 to 3.0.2 (#27) Bumps [rack](https://github.com/rack/rack) from 3.0.1 to 3.0.2. - [Release notes](https://github.com/rack/rack/releases) - [Changelog](https://github.com/rack/rack/blob/main/CHANGELOG.md) - [Commits](rack/rack@v3.0.1...v3.0.2) --- updated-dependencies: - dependency-name: rack dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump debug from 1.6.3 to 1.7.0 (#26) Bumps [debug](https://github.com/ruby/debug) from 1.6.3 to 1.7.0. - [Release notes](https://github.com/ruby/debug/releases) - [Commits](ruby/debug@v1.6.3...v1.7.0) --- updated-dependencies: - dependency-name: debug dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * renaming back and apply changes * rubocop fix * rubocop fix (for real) * add comment to why different ways to access to queues * use badges from this repo * Remove breaking version restrictions Add testing with multiple sidekiq versions Remove rack version requirement * Enable new cops by default * Ignore load error for sidekiq/capsule * Move dynamic sidekiq version definition to Gemfile * Improve sidekiq version detection * Add redis client wrappers * Remove redundant module function * Fix sq_config mocking * fix sidekiq 5 * Update release actions * improve Helpers version * Update release.yml for default project labels Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Andrejs <andrejs.cunskis@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
I've just integrated this but found a single instance escalated within 30 minutes to 2600 requests per second. This is a no load/new architecture and this is the only job running.
I did notice an environment variable/config error on my part, but not sure it would cause this - I had set ttl to
config.time_to_live = 8000
.I did happen to lose a kubernetes cluster node in the pool, so the sidekiq pod may have been restarted on another host. Could this be the same as @fruwe is discussing in #7?
Could this TTL 8000 cause it to spin out of control (wouldn't think so)?
The text was updated successfully, but these errors were encountered: