Fail closed on measurement-path execution errors - #384
Merged
Conversation
Benchmark evidence is only meaningful when the declared execution actually completed. Two legacy substrate paths could instead return plausible output after failure: Metal command-buffer errors were widely discarded, and CPU quantized kernels zero-filled on invalid geometry. Make both classes fail closed. * propagate Metal command-buffer failures where an error channel exists * abort at non-recoverable Metal sites rather than misreport CPU fallback * refuse the Kimi chain before advancing MLA cache state * add fault-injection witnesses for Kimi state preservation and matmul abort * replace Q4_K/Q6_K shape-mismatch zero-fill with typed KernelShapeError * leave output untouched on CPU refusal * pin public Q4K×Q8K parallel execution bit-identical to the scalar reference * update wasmtime 36.0.13 -> 36.0.14 to clear RUSTSEC-2026-0269 * make the MSRV CI gate execute the declared compiler and verify it * raise the declared MSRV to 1.98, the first toolchain supporting the existing VINDEX3 NEON dot-product path The MSRV gate was previously hollow; making it execute the declared compiler proved the declaration was false. 1.98 is the first verified toolchain that supports the existing VINDEX3 CPU integer path. The activation default, VINDEX3 EOS handling, and W10 legacy parity gate are intentionally outside this change.
This was referenced Sep 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Invariant: a benchmark result cannot be emitted from a failed Metal command buffer or an invalid CPU kernel shape.
Four changes, one invariant. Branch
measurement-guards, rebased onorigin/main8238b53.1. Metal: a failed command buffer never becomes a result
cb_status::wait_checkedreturnedResultand all 109 production and test call sites discarded it withlet _ =. A faulted buffer, or one Metal dropped after an earlier fault, returned from its wait with stale bytes in every output and the step carried on (#229 class).GroupedError::CommandBufferFailed { site, detail }andMxfp4Error::CommandBufferFailedon the Kimi KDA/MLA chain, the standalone KDA and MLA encoders, the grouped-expert and bf16 MoE block paths, and the MXFP4 matmul. Inkimi_layer::encode_layer_chainthe refusal happens beforestate.advance_to, and the scratch goes back to the pool untouched, so a fault cannot advance the MLA cache.cb_status::wait_or_abortforVec<f32>-returning decode stages and theOption-returningMatMul/QuantMatVec/bf16_gemv/router methods. ANonefrom those is taken by callers as "fall back to the CPU", which would report the fallback's number as the GPU's.expectin tests. 28 sites.LARQL_SPIN_WAIT=1arm inmoe_interleave.rsnow runsrequire_completedafter the spin; it previously bypassed the only refusal on that path.wait_until_completed()calls inlarql-cli's metal-lowered step (thevindex3 exec --backend metal-loweredinstrument) go throughcb_statustoo; the step's wait propagates asVindexError.wait_checkedis#[must_use], andcb_status/tests.rsnow scans the tree for a discarded result as well as a naked wait.cb_status::inject_fault_at_for_test(site_fragment)(a real fault cannot be produced deterministically; injected faults are not counted as GPU observations):kimi_layer::tests::a_failed_command_buffer_refuses_the_chain_and_advances_nothing— KDA+MLA chain, fault at the chain's own wait:Err(CommandBufferFailed)naming the site, no readback,mla_state.len() == 0, and the same chain then runs and advances once.cb_status::tests::a_faulted_matmul_aborts_instead_of_returning_a_result— productionMatMul::matmulon the GPU path,should_panic.cb_status::tests::wait_or_abort_stops_the_step_on_an_injected_fault,an_injected_fault_is_reported_once_at_its_site_and_then_cleared.2. CPU: kernels refuse malformed geometry instead of zero-filling
Every Q4_K/Q6_K × Q8_K kernel (scalar, NEON, NEON 2-row, three hand-asm variants, AVX2, the fused gate+up pair, the Q6_K trio) and the Q4_K f32 twins (
q4k_matvec_into,q4k_dual_matvec_into) zero-filledoutand returned on a shape mismatch or a short weight slab. A zero vector is a plausible logit vector.larql_compute::KernelShapeError { kernel, out_len, rows, x_len, cols, weight_bytes, needed_bytes }; every kernel and the publicq4k_q8k_matvec_parallelreturnResult<(), KernelShapeError>and leaveoutuntouched on refusal. Zero dimensions remain the empty product (zeros written,Ok).FormatRoute::q8k_matveccarries the new signature, so the registry's kernel pointers refuse the same way.CpuBackend::q4k_matvec/q4k_dual_matvec→None, the trait's documented contract for a shape error). Callers without one refuse by name (unwrap_or_else(panic)with the typed message) instead of returning zeros: the MoE expert paths' own short-slab guards,Q4kFfn::forward, the Q4_K lm_head, the direct attention projection, walk-FFN Q8K.q4k_q8k_dot/tests.rs: one fixture servesq8k_matvec_parallel_computes_exactly_the_scalar_reference_on_the_fixture(bit-identical to the scalar reference; control asserts the fixture is not all zeros) and, beside it, refusals for a wrong-length activation, an output shorter thanrows, a short weight slab (previously a panic, now typed), and the same through the Q6_K route. The nine per-kernel zero-fill tests became*_zero_dims_are_empty_and_short_weights_are_refused, assertingoutuntouched and the error's fields.3. wasmtime 36.0.13 → 36.0.14
cargo update -p wasmtime: lockfile only, the wasmtime/cranelift/pulley/winch family at 36.0.14 and nothing else. Clears RUSTSEC-2026-0269 (the reason main'squalityworkflow has been red).4. The MSRV gate runs the compiler it claims
rust-toolchain.tomlpins the development toolchain and rustup honours it over whatdtolnay/rust-toolchaininstalled, so since 2026-08-22 the MSRV job checked at 1.98 while claiming 1.88. The check step now runs underRUSTUP_TOOLCHAIN: ${{ steps.msrv.outputs.version }}, preceded by a step that printsrustc --versionand fails if it is not the declared version.Making the gate real exposed that the declared MSRV was false: the NEON dot-product intrinsics (
stdarch_neon_dotprod) used by the VINDEX3 CPU integer path (opplan/exec/cpu/{integer,stationary}.rs) and thecpu7_probeexample are unstable through 1.97 (bisected locally: 1.88, 1.89, 1.90, 1.91, 1.97 fail on exactly that error; 1.98 passes).rust-versionis now1.98, with the reason recorded beside it and in AGENTS.md. The gate was previously hollow; making it execute the declared compiler proved the declaration was false; 1.98 is the first verified toolchain that supports the existing VINDEX3 CPU integer path. Rewriting that path as inline asm would turn a measurement-integrity PR into execution-path surgery, so it stays.Not in this PR
Activation default, V3 EOS, the W10 gate. Real, but a different thesis.