feat(host): detect CPU steal at callback-deadline kills (detection only)#63
Merged
Conversation
CPU bounding is wall-clock only, and a true fuel/instruction budget is impossible on this stack (wazero v1.12.0 / extism v1.7.1 expose no fuel/epoch/gas API). A CPU-steal-throttled VM can therefore blow the per-callback deadline on a guest running pure, well-behaved guest code, and that kill is booked as a fault that can quarantine a healthy game. Measure host-stolen CPU at the kill site and record it ALONGSIDE the existing GameCallbackDeadline metric. Detection only: no change to fault(), quarantine, End(), or any kill/condemn decision; no exoneration yet. - Build-tagged, failure-tolerant steal sampler (steal_linux.go reads the /proc/stat aggregate hypervisor-STEAL field; steal_other.go is a no-op ok=false stub). Sampled at callback BOUNDARIES, never per-instruction. Injectable via a package var so tests can stub it. - Single signal: ONLY /proc/stat steal. cgroup throttle is deliberately not read or OR'd in — steal blames the host (the eventual exonerate case), cgroup throttle blames the guest (a runaway you must not exonerate). - Non-breaking optional StealMetrics extension interface, recorded via a type assertion at the wall-clock-kill site so existing Metrics implementers compile and run unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Gate /proc/stat sampling on the metrics sink implementing StealMetrics. The seam is dormant in prod until the platform implements it, so a host with no consumer now pays ZERO /proc/stat reads (was 2 reads per callback, discarded on the happy path). When enabled: one read before the callback, and the second only at the kill site (not on every callback). - Changeset patch -> minor: this adds the exported StealMetrics interface to the public kit API, which is a minor bump by semver. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
Author
|
Updated to address the two review nits (build verified darwin +
The remaining review note (the Linux real-sampler test only executes in CI, not on a darwin dev box) is inherent to the build tag and is covered by the synthetic- |
BCook98
marked this pull request as ready for review
June 17, 2026 09:42
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The per-callback kill switch is wall-clock only. A CPU-steal-throttled VM (a
noisy neighbor on oversubscribed hardware) can blow that deadline on a guest
running pure, well-behaved guest code — and that kill is currently booked as
a fault that can quarantine a healthy game. (The slow-shared-Postgres analogue
is already carved out via
hostIOKill; CPU steal is not.)Why a real fuel budget is impossible
The principled fix would be an instruction/fuel budget — bound the guest by work
done, not time elapsed. That is not available on this stack: wazero v1.12.0
and extism v1.7.1 expose no fuel/epoch/gas metering API. The only remaining
move is to measure host-stolen CPU at kill time.
Scope: DETECTION ONLY
This PR adds a measurement and nothing else. It does not change
fault(), quarantine,End(), or any kill/condemn decision, and it does notexonerate any kill. The new metric is recorded alongside (never instead of)
the existing
GameCallbackDeadline.The single signal + blame direction + coarseness
steal_linux.goreads only the/proc/stataggregatecpuline'shypervisor-STEAL field (the 8th value after the
cpulabel). It deliberatelydoes not read cgroup
throttled_usecand does not OR signals:exonerate case (a well-behaved guest punished for the platform's scheduling).
That is exactly the runaway you must not exonerate, so it is excluded here.
Coarseness caveats, documented in-code: the signal is host-wide (aggregate
across every vCPU and tenant on the VM, not this room's goroutine) and jiffy
resolution (~10ms, comparable to the 100ms deadline). It is sampled at
callback boundaries (once before the guest runs, once at return/kill), never
on a hot per-instruction path. Any read/parse failure (and all non-Linux hosts)
yields
ok=false→ no metric → current behavior.The non-breaking seam
StealMetricsis a separate optional interface, NOT added to the existingMetricsinterface (which would break the platform's implementer on the nextkit bump):
At the existing
deadlined && !hostIOKillbranch the kill site type-asserts theconfigured
MetricstoStealMetricsand records only if steal advanced acrossthe killed callback's window. Older implementers compile and run unchanged; the
seam stays dormant until something implements it.
REQUIRED platform follow-up
This seam is dormant in prod until the platform opts in. After the next kit
release, the platform's metrics type must:
StealMetrics.GameCallbackStealDeadline, andso the signal is actually collected. Until that lands there is no data — by
design, this PR ships the detection seam only.
Tests / verification
host/gameabi/steal_test.go(cross-platform,-race): table-driven over astubbed injected sampler with a non-exempt fixture (
OnFault != nil) anda recording
Metricsdouble that also implementsStealMetrics— stealunchanged ⇒ deadline only; steal advanced ⇒ both; host-I/O kill ⇒ neither
(unchanged). Not the load-spike game (it is
QuarantineExempt).host/gameabi/steal_linux_test.go(//go:build linux): parses a synthetic/proc/statvia an injectable path and asserts the live reader is monotonic.This runs in CI on ubuntu, not on the darwin dev host.
go build ./...,GOOS=linux go build ./host/gameabi/,GOOS=linux go vet ./host/gameabi/,go vet ./host/gameabi/,go test -race ./host/gameabi/(full suite green).A PATCH changeset describes the detection-only addition.
🤖 Generated with Claude Code