test_cpu_threadpool 'oversubscribed dispatch' ratio guard is unstable: it fired at 101.567 on the 4-core CI runner with a 0.48us denominator

[localai-bot]

What

build-test-cpu went red on an unrelated PR with one failure out of 404:

303/404 Test #303: test_cpu_threadpool ...***Failed    0.36 sec
tests/vt/test_cpu_threadpool.cpp:536: MESSAGE: empty-op dispatch: 2 threads 0.48 us, 5 threads 48.752 us, ratio 101.567
tests/vt/test_cpu_threadpool.cpp:539: ERROR: CHECK( ratio < 100.0 ) is NOT correct!
  values: CHECK( 101.567 <  100 )
[doctest] test cases:     9 |     8 passed | 1 failed | 1 skipped
[doctest] assertions: 19602 | 19601 passed | 1 failed |

Evidence: https://github.com/mudler/vllm.cpp/actions/runs/31698569035/job/94441971697

Why the guard is fragile

TEST CASE: oversubscribed dispatch does not cost a scheduler timeslice
(tests/vt/test_cpu_threadpool.cpp:499, added by 2028c8c74 for #391) asserts a
ratio of two wall-clock medians, over_us / fits_us, against a fixed 100.

The numerator and denominator are both machine-shape dependent, and the
denominator is the problem: it is the dispatch cost at a thread count that
fits the box. On a 2-core runner that was 0.48 us, small enough that
ordinary scheduler noise in the numerator moves the ratio by tens.

The two observations, same commit, same code:

Box	fits	over	ratio
2-core CI runner	2 threads, 0.48 us	5 threads, 48.752 us	101.567 (RED)
20-core dev box	10 threads, 7.213 us	21 threads, 19.467 us	2.699 (GREEN, 37x margin)

A guard that reads 2.7 on one machine and 101.6 on another is not measuring what
it names. It fires at 1.6% over the line on the small box while a genuine
regression on the large box would need a 37x blowup to trip it. That is the
shape this repo has recorded before: a tolerance that bounds nothing at one end
and everything at the other.

Not a flake to be re-run away

It is intermittent -- build-test-cpu is green on the scheduled main lane,
including the newest baseline 7572b0f4e2fb -- but the cause is structural, not
random: it depends on the runner's core count and on how small fits_us lands.
Re-running moves the dice, it does not fix the guard.

Suggested direction (needs its own spec, per AGENTS.md)

The intent is "the waiter yields its core instead of spinning through a
scheduler timeslice". That is a claim about an ABSOLUTE cost -- a timeslice is
milliseconds -- not about a ratio. Bounding over_us against a timeslice-scaled
absolute, or skipping when fits_us is below a floor where the ratio stops
being meaningful, would both keep the finding and lose the machine dependence.

Explicitly not proposed: raising 100 to some larger number. That is widening
a scope to turn a red gate green, which the protocol forbids and which would
make the large-box end even more permissive than it already is.

Found while landing #606, whose diff is argument parsing only and cannot reach
the threadpool. Filed rather than fixed in flow: this changes a gate's semantics,
so it needs a spec and red-before evidence of its own.

[localai-bot]

New data point: the guard is unstable at FIXED core count, not only across core counts.

This issue frames the guard as machine-shape dependent (2-core CI vs 20-core
dev). A third independent measurement, taken on the reviewed head of #624 while
repairing its record, is stronger than that:

Box	fits	over	ratio
2-core CI runner	2 threads, 0.48 us	5 threads, 48.752 us	101.567 RED
20-core box (this issue)	10 threads, 7.213 us	21 threads, 19.467 us	2.699
20-core box (#624 repair)	10 threads, 13.256 us	21 threads, 13.135 us	0.990872

Two same-class boxes, 2.7x apart on the same statistic. So a fix that only
scales the threshold by core count would not be enough — the ratio moves by ~3x
at fixed shape, which means any fixed threshold is picking a number out of a
distribution nobody has characterised.

The test's self-claim, verbatim at tests/vt/test_cpu_threadpool.cpp:496:

be trusted to fail loudly on a quiet machine and never to fail spuriously on a
busy one

The CI run falsifies the second half directly.

Whoever specs the fix: the useful shape is probably a bound derived from the
measured noise floor of this statistic on this box — the same discipline
.agents/benchmarking.md already requires for performance gates — rather than a
constant. A ratio whose denominator can collapse to sub-microsecond has no stable
scale to compare against.

Recorded in .agents/specs/serve-recipe-args.md § Now and on #624, since #624 is
where the red surfaced.

[localai-bot]

Correction: it is a 4-core runner, not 2-core — and the test itself proves it.

This issue's title and body (and my own comment above) say "2-core runner". That
is wrong, and the refutation was inside the failure message all along:

• tests/vt/test_cpu_threadpool.cpp:501 — if (cores < 4) { MESSAGE("skipped: needs >= 4 cores to oversubscribe meaningfully"); return; }. A 2-core box
  cannot produce a ratio at all; it skips.
• :512-513 — const int fits = cores / 2; const int over = cores + 1;. The
  observed run reports 2 threads and 5 threads, so cores/2 == 2 and
  cores+1 == 5 ⇒ hardware_concurrency() == 4.

GitHub's ubuntu-latest is 4-vCPU for public repositories, which agrees.

Caught by a fresh reviewer deriving it from the test source rather than reusing
the label. It had propagated into this title, this body, a comment of mine, and a
downstream spec — four places, from one unchecked inference.

Nothing about the argument changes. The mechanism is the same and the numbers
are untouched: fits_us = 0.48 µs is a sub-microsecond denominator, so ordinary
noise moves the ratio by tens, and the guard fires at 1.6% over. If anything the
correction sharpens it — the guard is not misbehaving on some unusually tiny
2-core box, it is misbehaving on the standard GitHub-hosted runner this project
uses for every PR.

Combined with the fixed-core-count spread already recorded above (2.699 vs
0.990872 on two 20-core boxes), the picture is that neither core count nor a
constant threshold explains the variance.