Skip to content

feat(host): detect CPU steal at callback-deadline kills (detection only)#63

Merged
BCook98 merged 2 commits into
mainfrom
uplift/host-steal-detection
Jun 17, 2026
Merged

feat(host): detect CPU steal at callback-deadline kills (detection only)#63
BCook98 merged 2 commits into
mainfrom
uplift/host-steal-detection

Conversation

@BCook98

@BCook98 BCook98 commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Problem

The per-callback kill switch is wall-clock only. A CPU-steal-throttled VM (a
noisy neighbor on oversubscribed hardware) can blow that deadline on a guest
running pure, well-behaved guest code — and that kill is currently booked as
a fault that can quarantine a healthy game. (The slow-shared-Postgres analogue
is already carved out via hostIOKill; CPU steal is not.)

Why a real fuel budget is impossible

The principled fix would be an instruction/fuel budget — bound the guest by work
done, not time elapsed. That is not available on this stack: wazero v1.12.0
and extism v1.7.1 expose no fuel/epoch/gas metering API. The only remaining
move is to measure host-stolen CPU at kill time.

Scope: DETECTION ONLY

This PR adds a measurement and nothing else. It does not change
fault(), quarantine, End(), or any kill/condemn decision, and it does not
exonerate any kill. The new metric is recorded alongside (never instead of)
the existing GameCallbackDeadline.

The single signal + blame direction + coarseness

steal_linux.go reads only the /proc/stat aggregate cpu line's
hypervisor-STEAL field (the 8th value after the cpu label). It deliberately
does not read cgroup throttled_usec and does not OR signals:

  • Steal blames the HOST — the hypervisor took our CPU. That is the eventual
    exonerate case (a well-behaved guest punished for the platform's scheduling).
  • cgroup throttle blames US — the guest exceeded its own cpu.max quota.
    That is exactly the runaway you must not exonerate, so it is excluded here.

Coarseness caveats, documented in-code: the signal is host-wide (aggregate
across every vCPU and tenant on the VM, not this room's goroutine) and jiffy
resolution
(~10ms, comparable to the 100ms deadline). It is sampled at
callback boundaries (once before the guest runs, once at return/kill), never
on a hot per-instruction path. Any read/parse failure (and all non-Linux hosts)
yields ok=false → no metric → current behavior.

The non-breaking seam

StealMetrics is a separate optional interface, NOT added to the existing
Metrics interface (which would break the platform's implementer on the next
kit bump):

type StealMetrics interface { GameCallbackStealDeadline(slug, callback string) }

At the existing deadlined && !hostIOKill branch the kill site type-asserts the
configured Metrics to StealMetrics and records only if steal advanced across
the killed callback's window. Older implementers compile and run unchanged; the
seam stays dormant until something implements it.

REQUIRED platform follow-up

This seam is dormant in prod until the platform opts in. After the next kit
release, the platform's metrics type must:

  1. implement StealMetrics.GameCallbackStealDeadline, and
  2. wire it to a new Prometheus counter,

so the signal is actually collected. Until that lands there is no data — by
design, this PR ships the detection seam only.

Tests / verification

  • host/gameabi/steal_test.go (cross-platform, -race): table-driven over a
    stubbed injected sampler with a non-exempt fixture (OnFault != nil) and
    a recording Metrics double that also implements StealMetrics — steal
    unchanged ⇒ deadline only; steal advanced ⇒ both; host-I/O kill ⇒ neither
    (unchanged). Not the load-spike game (it is QuarantineExempt).
  • host/gameabi/steal_linux_test.go (//go:build linux): parses a synthetic
    /proc/stat via an injectable path and asserts the live reader is monotonic.
    This runs in CI on ubuntu, not on the darwin dev host.
  • Verified locally: go build ./..., GOOS=linux go build ./host/gameabi/,
    GOOS=linux go vet ./host/gameabi/, go vet ./host/gameabi/,
    go test -race ./host/gameabi/ (full suite green).

A PATCH changeset describes the detection-only addition.

🤖 Generated with Claude Code

BCook98 and others added 2 commits June 17, 2026 19:26
CPU bounding is wall-clock only, and a true fuel/instruction budget is
impossible on this stack (wazero v1.12.0 / extism v1.7.1 expose no
fuel/epoch/gas API). A CPU-steal-throttled VM can therefore blow the
per-callback deadline on a guest running pure, well-behaved guest code, and
that kill is booked as a fault that can quarantine a healthy game.

Measure host-stolen CPU at the kill site and record it ALONGSIDE the existing
GameCallbackDeadline metric. Detection only: no change to fault(), quarantine,
End(), or any kill/condemn decision; no exoneration yet.

- Build-tagged, failure-tolerant steal sampler (steal_linux.go reads the
  /proc/stat aggregate hypervisor-STEAL field; steal_other.go is a no-op
  ok=false stub). Sampled at callback BOUNDARIES, never per-instruction.
  Injectable via a package var so tests can stub it.
- Single signal: ONLY /proc/stat steal. cgroup throttle is deliberately not
  read or OR'd in — steal blames the host (the eventual exonerate case),
  cgroup throttle blames the guest (a runaway you must not exonerate).
- Non-breaking optional StealMetrics extension interface, recorded via a type
  assertion at the wall-clock-kill site so existing Metrics implementers
  compile and run unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Gate /proc/stat sampling on the metrics sink implementing StealMetrics.
  The seam is dormant in prod until the platform implements it, so a host
  with no consumer now pays ZERO /proc/stat reads (was 2 reads per callback,
  discarded on the happy path). When enabled: one read before the callback,
  and the second only at the kill site (not on every callback).
- Changeset patch -> minor: this adds the exported StealMetrics interface to
  the public kit API, which is a minor bump by semver.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@BCook98

BCook98 commented Jun 17, 2026

Copy link
Copy Markdown
Contributor Author

Updated to address the two review nits (build verified darwin + GOOS=linux, full gameabi suite green with -race):

  1. Gated /proc/stat sampling on a StealMetrics sink. The seam is dormant until the platform implements StealMetrics, so a host with no consumer now pays zero /proc/stat reads (was 2 reads/callback, discarded on the happy path). When a sink is present: one read before the callback, and the second only at the kill site — not on every callback.
  2. Changeset patchminor — this adds the exported StealMetrics interface to the public kit API.

The remaining review note (the Linux real-sampler test only executes in CI, not on a darwin dev box) is inherent to the build tag and is covered by the synthetic-/proc/stat fixtures CI runs.

@BCook98
BCook98 marked this pull request as ready for review June 17, 2026 09:42
@BCook98
BCook98 merged commit b5f722c into main Jun 17, 2026
6 checks passed
@BCook98
BCook98 deleted the uplift/host-steal-detection branch June 17, 2026 09:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant