feat(conformance): add declarative conformance runner (022)#362
Merged
Conversation
Conformance cases become data a shared runner executes, not pointers to hand-written Rust tests. Adding a case/assertion/fixture is a pure data edit (SC-001): declarative case shape in cases.json, four oracle types (spec-expectation / live-differential / snapshot / invariant-metamorphic), observable channels + observers (CLI-process, filesystem, four Docker channels), the single normalizer, isolated Docker workspaces with RAII cleanup, committed provenance-checked snapshots with staleness gating, scoped allowed differences, validation classes V16-V20, and the new parity_conformance_runner live binary. Folds in review fixes: - staleness gates only on evidence-determining inputs; host tool versions (node/docker/compose) are informational, so committed snapshots replay fresh across machines (fixes the parity lane going red on CI Node 20) - no-reference-for-platform is non-blocking in the live binary - metamorphic idempotence observes the full container set (detects a leftover second container instead of masking it) - snapshot check uses the certify/validate sibling snapshot resolution - --force-tty-if-json honors its documented case-insensitive env vocabulary Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019vhr7Tcf8ybvSZSE1owqBT
github-actions
Bot
added
feature
…kout The declarative conformance runner hashes fixture bytes into fixtureHash/ caseHash and the hermetic snapshot_staleness tests recompute them and compare to the recorded provenance. On a Windows checkout git's autocrlf rewrote LF to CRLF, changing the fixture bytes so the recomputed caseHash no longer matched the LF-computed committed value (committed_snapshot_hashes_match_the_committed_case on the windows dev-fast lane). Extend .gitattributes with -text for the byte-exact conformance/fixtures/** and conformance/snapshots/**, mirroring the 020/021 entries. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019vhr7Tcf8ybvSZSE1owqBT
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Declarative conformance runner (022)
Implements the 022-conformance-runner spec: conformance cases become data a shared runner executes, not pointers to hand-written Rust tests. Adding a case/assertion/fixture is a pure data edit — no new Rust function (SC-001).
What's included
conformance/registry/cases.json— orderedoperations[],oracleType, per-channelexpected[], scopedallowedDifferences[],fsAllowlist,cleanup,resourceGroup. A record is exactly one of legacy (binary-backed) or declarative, enforced fail-loud at load.oracle_type.rs, one explicit dispatch):spec-expectation,live-differential(deacon vs the pinned oracle),snapshot(vs a committed provenance-checked snapshot),invariant-metamorphic(relationship across ≥2 ops).observe/): CLI-process (exit/stdout/stderr/structured-output), allowlist-scoped filesystem/file-content, and the four Docker channels (image/process-graph/injected-process/temporal). Evidence captured raw then normalized by the singlenormalize.rs; raw + normalized persisted separately.conformance/snapshots/<os-arch>/<case-id>/with 13-field provenance + staleness gating; a reviewedconformance-snapshot refreshbin is the only writer.(behavior, context, observablePath)+ exactly one resolvablewaiverId/divergenceId; no global ignore lists; self-invalidating when stale.validate.rs), all block a PR viaregistry_valid;certifysurfaces snapshot coverage as non-blocking info.parity_conformance_runner(parity profile only), registered in the parity registry + wired into all nextest profiles.Review fixes folded in (from
/code-review xhigh)node/docker/compose). They are informational provenance only; gating on them made every committed snapshot stale on every machine but the recorder's, which would have turned the parity lane red on CI (recorded Node 22.23.1 vs CI Node 20). Staleness now compares only the evidence-determining inputs (case/fixture hashes, oracle/source pins, imageDigests, normalizer).no-reference-for-platformis non-blocking in the live binary — surfaced (never silently skipped) but not a hard failure, consistent with the runner exit-code contract andcertify.snapshot check's snapshot root with thecertify/validatesibling-resolution.--force-tty-if-jsonnow honors its documented case-insensitive env vocabulary via clap'sBoolishValueParser.Verification
cargo fmt --all -- --checkandcargo clippy --all-targets --all-features -- -D warningsclean.parity_conformance_runnerpasses against real Docker + the pinned oracle.🤖 Generated with Claude Code
https://claude.ai/code/session_019vhr7Tcf8ybvSZSE1owqBT