Skip to content

Fail closed on measurement-path execution errors - #384

Merged
chrishayuk merged 1 commit into
mainfrom
measurement-guards
Sep 1, 2026
Merged

Fail closed on measurement-path execution errors#384
chrishayuk merged 1 commit into
mainfrom
measurement-guards

Conversation

@chrishayuk

Copy link
Copy Markdown
Owner

Invariant: a benchmark result cannot be emitted from a failed Metal command buffer or an invalid CPU kernel shape.

Four changes, one invariant. Branch measurement-guards, rebased on origin/main 8238b53.

1. Metal: a failed command buffer never becomes a result

cb_status::wait_checked returned Result and all 109 production and test call sites discarded it with let _ =. A faulted buffer, or one Metal dropped after an earlier fault, returned from its wait with stale bytes in every output and the step carried on (#229 class).

  • Typed refusal where a channel exists. GroupedError::CommandBufferFailed { site, detail } and Mxfp4Error::CommandBufferFailed on the Kimi KDA/MLA chain, the standalone KDA and MLA encoders, the grouped-expert and bf16 MoE block paths, and the MXFP4 matmul. In kimi_layer::encode_layer_chain the refusal happens before state.advance_to, and the scratch goes back to the pool untouched, so a fault cannot advance the MLA cache.
  • Abort where none exists. cb_status::wait_or_abort for Vec<f32>-returning decode stages and the Option-returning MatMul/QuantMatVec/bf16_gemv/router methods. A None from those is taken by callers as "fall back to the CPU", which would report the fallback's number as the GPU's.
  • expect in tests. 28 sites.
  • The LARQL_SPIN_WAIT=1 arm in moe_interleave.rs now runs require_completed after the spin; it previously bypassed the only refusal on that path.
  • The three naked wait_until_completed() calls in larql-cli's metal-lowered step (the vindex3 exec --backend metal-lowered instrument) go through cb_status too; the step's wait propagates as VindexError.
  • wait_checked is #[must_use], and cb_status/tests.rs now scans the tree for a discarded result as well as a naked wait.
  • Witnesses, via cb_status::inject_fault_at_for_test(site_fragment) (a real fault cannot be produced deterministically; injected faults are not counted as GPU observations):
    • kimi_layer::tests::a_failed_command_buffer_refuses_the_chain_and_advances_nothing — KDA+MLA chain, fault at the chain's own wait: Err(CommandBufferFailed) naming the site, no readback, mla_state.len() == 0, and the same chain then runs and advances once.
    • cb_status::tests::a_faulted_matmul_aborts_instead_of_returning_a_result — production MatMul::matmul on the GPU path, should_panic.
    • cb_status::tests::wait_or_abort_stops_the_step_on_an_injected_fault, an_injected_fault_is_reported_once_at_its_site_and_then_cleared.

2. CPU: kernels refuse malformed geometry instead of zero-filling

Every Q4_K/Q6_K × Q8_K kernel (scalar, NEON, NEON 2-row, three hand-asm variants, AVX2, the fused gate+up pair, the Q6_K trio) and the Q4_K f32 twins (q4k_matvec_into, q4k_dual_matvec_into) zero-filled out and returned on a shape mismatch or a short weight slab. A zero vector is a plausible logit vector.

  • New larql_compute::KernelShapeError { kernel, out_len, rows, x_len, cols, weight_bytes, needed_bytes }; every kernel and the public q4k_q8k_matvec_parallel return Result<(), KernelShapeError> and leave out untouched on refusal. Zero dimensions remain the empty product (zeros written, Ok).
  • FormatRoute::q8k_matvec carries the new signature, so the registry's kernel pointers refuse the same way.
  • Callers with a channel propagate (CpuBackend::q4k_matvec / q4k_dual_matvecNone, the trait's documented contract for a shape error). Callers without one refuse by name (unwrap_or_else(panic) with the typed message) instead of returning zeros: the MoE expert paths' own short-slab guards, Q4kFfn::forward, the Q4_K lm_head, the direct attention projection, walk-FFN Q8K.
  • Tests, through the public path, in q4k_q8k_dot/tests.rs: one fixture serves q8k_matvec_parallel_computes_exactly_the_scalar_reference_on_the_fixture (bit-identical to the scalar reference; control asserts the fixture is not all zeros) and, beside it, refusals for a wrong-length activation, an output shorter than rows, a short weight slab (previously a panic, now typed), and the same through the Q6_K route. The nine per-kernel zero-fill tests became *_zero_dims_are_empty_and_short_weights_are_refused, asserting out untouched and the error's fields.

3. wasmtime 36.0.13 → 36.0.14

cargo update -p wasmtime: lockfile only, the wasmtime/cranelift/pulley/winch family at 36.0.14 and nothing else. Clears RUSTSEC-2026-0269 (the reason main's quality workflow has been red).

4. The MSRV gate runs the compiler it claims

rust-toolchain.toml pins the development toolchain and rustup honours it over what dtolnay/rust-toolchain installed, so since 2026-08-22 the MSRV job checked at 1.98 while claiming 1.88. The check step now runs under RUSTUP_TOOLCHAIN: ${{ steps.msrv.outputs.version }}, preceded by a step that prints rustc --version and fails if it is not the declared version.

Making the gate real exposed that the declared MSRV was false: the NEON dot-product intrinsics (stdarch_neon_dotprod) used by the VINDEX3 CPU integer path (opplan/exec/cpu/{integer,stationary}.rs) and the cpu7_probe example are unstable through 1.97 (bisected locally: 1.88, 1.89, 1.90, 1.91, 1.97 fail on exactly that error; 1.98 passes). rust-version is now 1.98, with the reason recorded beside it and in AGENTS.md. The gate was previously hollow; making it execute the declared compiler proved the declaration was false; 1.98 is the first verified toolchain that supports the existing VINDEX3 CPU integer path. Rewriting that path as inline asm would turn a measurement-integrity PR into execution-path surgery, so it stays.

Not in this PR

Activation default, V3 EOS, the W10 gate. Real, but a different thesis.

Benchmark evidence is only meaningful when the declared execution actually
completed. Two legacy substrate paths could instead return plausible output
after failure: Metal command-buffer errors were widely discarded, and CPU
quantized kernels zero-filled on invalid geometry.

Make both classes fail closed.

* propagate Metal command-buffer failures where an error channel exists
* abort at non-recoverable Metal sites rather than misreport CPU fallback
* refuse the Kimi chain before advancing MLA cache state
* add fault-injection witnesses for Kimi state preservation and matmul abort
* replace Q4_K/Q6_K shape-mismatch zero-fill with typed KernelShapeError
* leave output untouched on CPU refusal
* pin public Q4K×Q8K parallel execution bit-identical to the scalar reference
* update wasmtime 36.0.13 -> 36.0.14 to clear RUSTSEC-2026-0269
* make the MSRV CI gate execute the declared compiler and verify it
* raise the declared MSRV to 1.98, the first toolchain supporting the existing
  VINDEX3 NEON dot-product path

The MSRV gate was previously hollow; making it execute the declared compiler
proved the declaration was false. 1.98 is the first verified toolchain that
supports the existing VINDEX3 CPU integer path.

The activation default, VINDEX3 EOS handling, and W10 legacy parity gate are
intentionally outside this change.
@chrishayuk
chrishayuk merged commit 0563baf into main Sep 1, 2026
54 of 61 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant