Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

perf: Merge worker-pool and scheduler #26566

Merged
merged 4 commits into from
May 27, 2024
Merged

perf: Merge worker-pool and scheduler #26566

merged 4 commits into from
May 27, 2024

Conversation

ankush
Copy link
Member

@ankush ankush commented May 26, 2024 •
edited
Loading

Scheduler is a daemon process that runs once every minute to do enqueue job and
then does nothing. This still incurs ~70MB RAM cost (70MB RSS, 55MB USS).

This can be avoided by making it part of workers itself.

Rough Design (very similar to RQ):

  • There's a single global filelock
  • Each worker before starting work tries to aquire this lock, only one succeeds.
  • Workers still check every 10 minutes if scheduler lock is available, if available one of them wins and runs scheduler.

We run ~3000 benches currently, that's at least ~150GB of RAM usage.

@github-actions github-actions bot added the add-test-cases Add test case to validate fix or enhancement label May 26, 2024
ankush added a commit to frappe/press that referenced this pull request May 26, 2024
@ankush
fix: reduce scheduler priority in supervisor
c16101e 
It doesn't need to start before workers, infact scheduler doesn't do
anything for 60 seconds after starting

ref: frappe/frappe#26566
@ankush ankush force-pushed the sched_merge branch 2 times, most recently from 3e236e7 to 7f2ec58 Compare May 27, 2024 06:10
@ankush ankush added defer backport Backports for some PR are deferred for a week or two to test them properly before releasing and removed add-test-cases Add test case to validate fix or enhancement labels May 27, 2024
@ankush ankush marked this pull request as ready for review May 27, 2024 06:22
@ankush ankush added Skip CI Doesn't run Ci for this PR. Run Server Tests Runs Server CI builds even if backend changes not detected. labels May 27, 2024
@ankush
perf: optimize scheduling
59bbbd7 
- Randomize scheduling order
- Cache dormant status for 1 hour (this gets checked every second, too much work)
@ankush ankush force-pushed the sched_merge branch 2 times, most recently from 1b32b80 to 88396a5 Compare May 27, 2024 06:57
@ankush ankush changed the title perf: Merge worker and scheduler perf: Merge worker-pool and scheduler May 27, 2024
@ankush
refactor: keep scheduler+worker behaviour to workerpool
d131787 
- Single worker adds some extra costs to each worker (importing scheduler)
- Workerpool can import it all once and share
- This way we need not have many different
@ankush ankush merged commit 7a37c7f into frappe:develop May 27, 2024
23 checks passed
@ankush ankush deleted the sched_merge branch May 27, 2024 07:12
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jun 11, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers
No reviews
Assignees
No one assigned
Labels
defer backport Backports for some PR are deferred for a week or two to test them properly before releasing Run Server Tests Runs Server CI builds even if backend changes not detected. Skip CI Doesn't run Ci for this PR.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
1 participant
@ankush