Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Up Sidekiq Concurrency, pin to DB pool amount. #6725

Closed
wants to merge 1 commit into from
Closed

Conversation

dandrabik
Copy link
Member

WHAT

Given we have an increase in background job volume, I want to experiment with upping the number of workers per server. Trying 8 workers up from 4.

WHY

More jobs, more problems.

HOW

  1. Pin the sidekiq concurrency to the Rails Thread Max which is used to set the number of DB connections (so every thread gets a Db connection and we don't get errors for no Connection available).
  2. Pass the thread count in an ENV variable in the work Procfile used by Heroku so the BG workers can have a different thread count than the Web servers.

Mainly getting this from here: sidekiq/sidekiq#2985 (comment)

Screenshots

PR Checklist Your Answer
Have you added and/or updated tests? NO, not really testable
Have you deployed to Staging? Not yet - deploying now!
Self-Review: Have you done an initial self-review of the code below on Github? Yes
Design Review: If applicable, have you compared the coded design to the mockups? N/A

@dandrabik
Copy link
Member Author

dandrabik commented Sep 10, 2020

Ok, Looking at the memory graphs, I don't think we should add more workers per dyno (We are at max memory usage), instead we should just add more dynos.

Screen Shot 2020-09-10 at 4 21 10 PM

@dandrabik dandrabik closed this Sep 10, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant