You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Add a runtime API that reports the cumulative time since process start, during which the Go scheduler was "stalled" — defined as: at least one runnable goroutine existed but was not running on any P. This mirrors Linux PSI semantics (/proc/pressure/cpu) at the Go-runtime level.
Callers can take two samples and compute a fraction-of-time-stalled metric over any window they choose, identical to how PSI is consumed today.
Motivation
The current scheduler-latency histogram is fragile
Today, overload in our infrastructure is detected by reading the Go scheduler-latency histogram and applying an empirically-chosen threshold — e.g. p90(scheduler latency) > 10ms ⇒ overload. Two problems:
Empirical thresholds are workload-specific. Spiky workloads break the threshold model. During a short burst, the scheduler latency distribution can shift orders of magnitude compared to a steady state. A threshold tuned to catch steady-state overload either fires on every benign spike or misses real sustained overload. There is no single latency value that distinguishes "brief expected spike" from "actual stall".
PSI semantics solve all of this
The Linux kernel's PSI metric works extremely well as an overload signal precisely because:
It measures what fraction of time the system was overloaded, not how badly it was overloaded at any instant.
The output is bounded in [0, 1], so thresholds have an intuitive, workload-independent interpretation ("we were stalled more than 20% of the last 10 seconds").
It is robust to spikes: a 10ms stall in a 1s window is just a 1% stall, regardless of how deep the queue got.
Same threshold works across hardware, workload shapes, and traffic patterns.
Why we cannot use Linux PSI directly for Go services
We tried, and it does not work:
if GOMAXPROCS ≤ host cores: the Go runtime never oversubscribes the kernel scheduler, so kernel-level CPU PSI stays near zero even when goroutines are queued behind each other inside the Go runtime. The kernel cannot see Go's internal runqueues.
if GOMAXPROCS > host cores: kernel PSI starts to register, but only on machines with a small number of cores. On large machines (e.g. 30+ cores) the same setup produces stalls inside the Go scheduler without producing meaningful kernel PSI, because work-stealing imbalance keeps the number of kernel-runnable OS threads close to the core count even while goroutines pile up on a subset of Ps.
The fundamental issue is that PSI is computed at the wrong abstraction layer. The scheduler we care about — the one that decides which goroutine runs next — is the Go scheduler, not the kernel scheduler.
Experiment results: high-frequency sampling
Before proposing a runtime change, we experimented with approximating the PSI signal entirely inside the app by sampling the existing scheduler-latency signal at very high frequency and integrating the result into a fraction-of-time-stalled estimate.
The resulting signal is dramatically less noisy than the threshold-on-histogram approach: spikes shrink to their actual duration, sustained overload stands out clearly, and the same threshold becomes meaningful across very different workload shapes. This is strong empirical evidence that the PSI shape of the signal is the right one — what we are missing is a way to produce it without the cost of high-frequency sampling. Also our current sampling method consistently undercounts because we use max during sampling period as stall time and ignore the rest of the histogram. Still you can see the diff in noise on the same workload with sinusoid shaped overload pattern.
Summary
Add a runtime API that reports the cumulative time since process start, during which the Go scheduler was "stalled" — defined as: at least one runnable goroutine existed but was not running on any P. This mirrors Linux PSI semantics (/proc/pressure/cpu) at the Go-runtime level.
Callers can take two samples and compute a fraction-of-time-stalled metric over any window they choose, identical to how PSI is consumed today.
Motivation
The current scheduler-latency histogram is fragile
Today, overload in our infrastructure is detected by reading the Go scheduler-latency histogram and applying an empirically-chosen threshold — e.g.
p90(scheduler latency) > 10ms ⇒ overload. Two problems:Sensitivity to runtime changes. A change in the runtime can shift the distribution without any change in actual load, invalidating thresholds that were calibrated against a previous release. This proposal is itself motivated by such a change. See runtime/metrics: /sched/latencies:seconds sampling population changed in 1.26, breaking mean and percentile signals #79391 for details.
Empirical thresholds are workload-specific. Spiky workloads break the threshold model. During a short burst, the scheduler latency distribution can shift orders of magnitude compared to a steady state. A threshold tuned to catch steady-state overload either fires on every benign spike or misses real sustained overload. There is no single latency value that distinguishes "brief expected spike" from "actual stall".
PSI semantics solve all of this
The Linux kernel's PSI metric works extremely well as an overload signal precisely because:
Why we cannot use Linux PSI directly for Go services
We tried, and it does not work:
GOMAXPROCS ≤ host cores: the Go runtime never oversubscribes the kernel scheduler, so kernel-level CPU PSI stays near zero even when goroutines are queued behind each other inside the Go runtime. The kernel cannot see Go's internal runqueues.GOMAXPROCS > host cores: kernel PSI starts to register, but only on machines with a small number of cores. On large machines (e.g. 30+ cores) the same setup produces stalls inside the Go scheduler without producing meaningful kernel PSI, because work-stealing imbalance keeps the number of kernel-runnable OS threads close to the core count even while goroutines pile up on a subset of Ps.The fundamental issue is that PSI is computed at the wrong abstraction layer. The scheduler we care about — the one that decides which goroutine runs next — is the Go scheduler, not the kernel scheduler.
Experiment results: high-frequency sampling
Before proposing a runtime change, we experimented with approximating the PSI signal entirely inside the app by sampling the existing scheduler-latency signal at very high frequency and integrating the result into a fraction-of-time-stalled estimate.
The resulting signal is dramatically less noisy than the threshold-on-histogram approach: spikes shrink to their actual duration, sustained overload stands out clearly, and the same threshold becomes meaningful across very different workload shapes. This is strong empirical evidence that the PSI shape of the signal is the right one — what we are missing is a way to produce it without the cost of high-frequency sampling. Also our current sampling method consistently undercounts because we use max during sampling period as stall time and ignore the rest of the histogram. Still you can see the diff in noise on the same workload with sinusoid shaped overload pattern.