proposal: runtime: add runtime.SleepUntilIdle cooperative yield API

[dt]

Proposal Details

Abstract

This proposal introduces runtime.SleepUntilIdle(), a cooperative yield method for the Go runtime. It allows goroutines performing background or latency-tolerant work to voluntarily yield execution when other goroutines are runnable, avoiding interference with latency-sensitive work.

Background and Motivation

Go applications such as CockroachDB often run both latency-sensitive ("foreground") goroutines, such as serving requests, and latency-insensitive ("background") goroutines, such as refreshing caches or rebuilding indexes. When too many background goroutines become runnable, the scheduler's queues saturate, causing foreground work to wait. Having tasks try to throttle themselves is common, but with current primitives it is hard to both avoid bursts of excessive runnable work and also efficiently utilize available hardware capacity.

The core challenge is an information gap: the runtime knows when throttling is needed (it tracks runnable goroutine counts and idle P availability) but not which goroutines can be safely throttled. Individual goroutines know when they are latency-tolerant but lack cheap access to the runtime's instantaneous scheduling state.

A cooperative yield mechanism bridges this gap. Tasks voluntarily offer to step aside when the runtime indicates that other work is waiting, without requiring global prioritization or preemption.

Proposed Change

API

package runtime

// SleepUntilIdle sleeps for up to nanos nanoseconds or until the scheduler has
// idle capacity, whichever comes first, returning the approximate number of
// nanoseconds spent sleeping (which could be more than nanos due to delays in
// being rescheduled).
//
// If the scheduler already has idle capacity, SleepUntilIdle returns immediately
// without sleeping, and aims to do so as cheaply as possible (making it amenable
// to being called frequently or in tight loops).

func SleepUntilIdle(nanos int64) int64

Semantics

• Cooperative: The call is entirely voluntary; no prioritization or preemption.
• Ephemeral: The goroutine calling SleepUntilIdle is only treated differently during the call to SleepUntilIdle(); once it returns it runs normally.
• Control signal: The runtime's own npidle and run queue state control yielding.
• Low overhead: No-ops when yield is not required must be near-free so that callers can invoke it frequently (e.g. every iteration of a tight loop).
• Separate queue: Yielded goroutines are parked separately from the normal run queue, so they do not count as waiting work for the purposes of other yield decisions, preventing thrashing.

Example Usage

func rebuildIndex() {
  for _, item := range items {
    runtime.SleepUntilIdle(int64(100 * time.Millisecond))
    process(item)
  }
}

Implementation Notes

The envisioned implementation uses a small inlineable outer wrapper that checks sched.npidle and returns immediately if there are idle Ps, making the common no-yield case near-free when called in tight loops.

After the fast path, a slower path checks for waiting work in local runq and global runq while still being relatively cheap, but checking all other Ps' runqs or even netpoll is expensive enough that it likely is restricted to a subset or sample of calls. When work is found, the slow path parks the goroutine on a separate yieldq.

findRunnable pulls from yieldq when it has no other runnable work, resuming yielded goroutines as soon as capacity becomes available to minimize underutilization of hardware.

A prototype of this implementation approach was developed in CockroachDB's fork of Go with more details available in comments below.

Alternatives Considered

User-space Instrumentation without changing runtime/scheduler

A user-space implementation could attempt to infer scheduling pressure via runtime/metrics or OS-reported load, but these signals are too coarse or lagging for tight-loop yield decisions, and polling them frequently enough incurs non-trivial synchronization cost. Alternatively, user-space code could try to track every goroutine's blocked<->runnable transitions to replicate the runtime's view, but this would be cumbersome, expensive, and undermine Go's lightweight concurrency model.

Effective cooperative yielding requires two things: knowing when to de-schedule (to prevent latency impact) and knowing when to re-schedule (to avoid leaving utilization on the table). The runtime already maintains the authoritative state for both: npidle and run queue lengths provide a near-free signal for when yielding is needed, and findRunnable is the natural place to resume yielded goroutines immediately when capacity becomes available.

Gosched

runtime.Gosched() is an existing in-runtime yield, but a) it lacks a near-free fast path when yielding is unnecessary and b) it places yielded goroutines back on the normal run queue so multiple goroutines trying to cooperatively yield just thrash, repeatedly yielding to each other. While the implementation of Gosched could likely be adjusted/optimized in-place to address (a), the separate queue for parked goroutines makes the semantics of SleepUntilIdle distinct from Gosched. That said, SleepUntilIdle(0) implies not parking in said queue, and just immediately being eligible for re-scheduling, making it effectively be Gosched() but with optimized, faster noop path in which case perhaps Gosched becomes a wrapper for SleepUntilIdle(0) to reduce code duplication.

Compatibility

• Opt-in; existing code is unaffected, no change to scheduler invariants, GC, or preemption.

Prior Art

• runtime.goschedIfBusy (internal GC helper) has a similar design which uses npidle to decide whether to yield.

[dt]

An implementation of something very similar to this proposed API (though lacking the sleep duration upper-bound parameter) exists as runtime.Yield in CockroachDB's fork of Go and is used by default as of CockroachDB v26.1. Benchmarking it has yielded -- pun intended -- very positive results.

In a benchmark driving sustained >95% CPU utilization with mixed foreground and background workloads, adding runtime.Yield() calls to background tight loops reduced p99 goroutine scheduling latency from 20+ms to ~100us. Since foreground request processing in CRDB involves many goroutine handoffs, those scheduling delays compounded in user-visible p99 latency, where the improvement went from spiking from a 2ms baseline to hundreds of milliseconds p99 during background operations to instead only climbing to about ~3ms during periods of cpu saturation.

Prior attempts in CRDB to implement entirely "userspace" approaches to limiting background work, which lacked this visibility into the scheduler's instantaneous state, consistently failed to drive sustained CPU utilization above ~60% without starting to see >1ms goroutine scheduling delays.

[gabyhelp]

Related Issues

• proposal: runtime: add WaitIdle function #65336 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}