scx_lavd: Add configurable migration threshold with avg utilization #2912
+45
−4
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
multics69
reviewed
Oct 18, 2025
multics69
reviewed
Oct 18, 2025
The current utilization based migration scheme look back period is too short and results in overly aggressive load balancing. For example, in certain workloads on a machine that has an average utilization at ~90%, sampling load deltas between compute domains still showed > 50% differences in load. Allow users to define some threshold for task stealing using sum of util_avg of CPUs on a compute domain instead to smooth out load tracking and reduce overall task migrations. These are sampled on a ~90% workload to measure the cost of cross CCX/LLC domain migrations. bpftop sample: +-----------------+--------------+---------------+ | lavd_dispatch | forced_steal | no_task_steal | +-----------------+--------------+---------------+ | avg_runtime(ns) | 3593 | 762 | | cputime(%) | 250.99 | 40.87 | | events/s | 704720 | 746275 | +-----------------+--------------+---------------+ latencies(ns): +-----+---------------+--------------------+----------------------+ | | dispatch(avg) | consume(local_llc) | consume(remote_llc) | +-----+---------------+--------------------+----------------------+ | avg | 3884 | 595.9 | 2216.6 | | p90 | 7721 | 1952.3 | 5709.0 | | p99 | 14715 | 6900.0 | 13023.5 | +-----+---------------+--------------------+----------------------+ In general, cross CCX/LLC domain migrations are expensive at high utilization and results in lower IPC due to cache locality. Disabling forced task stealing allows us to trade off some work conservation for better cache characteristics and scheduler overhead. Co-authored-by: Claude <claude@anthropic.com> Signed-off-by: David Dai <david.dai@linux.dev>
daidavid
force-pushed
the
lavd_config_mig_threshold
branch
from
540ba8e to
c3ef934
Compare
multics69
approved these changes
Oct 22, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The current utilization based migration scheme look back period is too short and results in overly aggressive load balancing. For example, in certain workloads on a machine that has an average utilization at ~90%, sampling load deltas between compute domains still showed > 50% differences in load.
Allow users to define some threshold for task stealing using sum of util_avg of CPUs on a compute domain instead to smooth out load tracking and reduce overall task migrations.
These are sampled on a ~90% workload to measure the cost of cross CCX/LLC domain migrations.
bpftop sample:
+-----------------+--------------+---------------+
| lavd_dispatch | forced_steal | no_task_steal |
+-----------------+--------------+---------------+
| avg_runtime(ns) | 3593 | 762 |
| cputime(%) | 250.99 | 40.87 |
| events/s | 704720 | 746275 |
+-----------------+--------------+---------------+
latencies(ns):
+-----+---------------+--------------------+----------------------+
| | dispatch(avg) | consume(local_llc) | consume(remote_llc) |
+-----+---------------+--------------------+----------------------+
| avg | 3884 | 595.9 | 2216.6 |
| p90 | 7721 | 1952.3 | 5709.0 |
| p99 | 14715 | 6900.0 | 13023.5 |
+-----+---------------+--------------------+----------------------+
In general, cross CCX/LLC domain migrations are expensive at high utilization and results in lower IPC due to cache locality. Disabling forced task stealing allows us to trade off some work conservation for better cache characteristics and scheduler overhead.