ref(server): Move all heavy actors into dedicated arbiters #1378
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jjbayer
approved these changes
Aug 4, 2022
| @@ -135,6 +135,9 @@ impl ServiceState { | |||
| let outcome_producer = Arbiter::start(|_| outcome_producer); | |||
| registry.set(outcome_producer.clone()); | |||
|
|
|||
| let outcome_aggregator = OutcomeAggregator::new(&config, outcome_producer.recipient()); | |||
| registry.set(outcome_aggregator.start()); | |||
There was a problem hiding this comment.
Two questions:
- Out of curiosity, why did you move this up here?
- Why does the outcome aggregator not get its own thread?
There was a problem hiding this comment.
That probably wouldn't have been necessary, and I'm happy to revert that. I merely moved it up so that it's next to the outcome producer.
It doesn't have its own thread since it looks like it does relatively cheap stuff internally. I'm not yet convinced that we should have one thread for each actor/service, so I'm mostly going by what we know causes issues.
flub
approved these changes
Aug 4, 2022
* master: fix(metrics): Light normalize before extracting metrics (#1366)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Each actix arbiter spawns a thread with its own single-threaded event loop. This
allows to create isolation between actors that may cause CPU spikes when
backlogged, and makes fair scheduling in actix easier.
While some of our actors, notably the
OutcomeProducerand (metrics)Aggregatorhad been spawned into a dedicated arbiter, most of the other actorswere still running on the main arbiter. The
EnvelopeManageris known to causean increase in CPU consumption due to how it spawns futures internally. This
slows down the project cache and healthcheck, which in turn blocks incoming HTTP
requests.
Since there's a global slowdown of the main arbiter, it becomes hard to separate
cause from effect when investigating slow response times. It is not clear
whether the project cache causes increased queue sizes or whether increased
queue sizes delay project fetching.
To create clear separation, this PR introduces a dedicated arbiter for every
heavy actor.