Throughput regression from capacity-wake suppression in #216

[hardbyte]

Summary

A focused 128-worker bisect shows PR #216 (ec7d9ae, "Reduce empty capacity wake claims") is the load-bearing throughput regression between alpha.3 and current main.

Commit	Label	Run 1	Run 2	Mean @ 128w
d7748d0	alpha.3 baseline	4,973	4,766	4,870
8ebcbc5	after #214, claimer cap	5,316	5,012	5,164
d930e72	after #215, striped claims	4,676	4,997	4,837
ec7d9ae	after #216, empty wake	2,921	2,254	2,588
5a8db21	after #220, current	2,869	2,590	2,729

Reading

• Make queue claimer cap configurable #214 and Stabilize striped queue-storage runtime claims #215 are within run-to-run variance of the alpha.3 baseline.
• Reduce empty dispatcher claims after capacity wakes #216 is the cliff: 4,837 -> 2,588, about a 47% drop from a single PR.
• Recover capacity wake refill after partial claims #220 partially repaired one case, especially visible around 16 workers, but it did not address the bulk of the 64/128-worker regression.
• Shorten queue-storage runtime claim transactions #222 did not validate the lock-window/deserialization hypothesis; it should not be treated as the perf recovery.

Likely cause

The capacity-wake suppression introduced in #216 is globally too aggressive. Under high concurrency, completion-driven capacity wakes appear to be useful work-dispatch signals even after prior empty claims. Suppressing those wakes leaves workers under-refilled and shifts too much responsibility to the slower fallback poll path.

Candidate fix

Revert or rewrite the capacity-release wake suppression. A narrower candidate is to keep the exponential fallback poll backoff from #216, but restore alpha.3-style behavior where every capacity release wake drains the dispatcher.

A probe branch exists at brian/probe-unsuppress-capacity-wakes for that shape.

[hardbyte]

PR #224 restored capacity-release wake drains while keeping #216 poll backoff. Initial benchmark read from @hardbyte: directionally better but only partial recovery (~3,120 jobs/s mean at 128w, about +14% vs current and still ~36% below the alpha.3 bisect baseline). The remaining #216 suspect is PollBackoff, which can double fallback sleeps during permit-saturated and SKIP-LOCKED-empty cycles.

Opened follow-up A/B PR #225 to remove the poll backoff as well while keeping capacity wakes unsuppressed and fixed config.poll_interval fallback polling.