conformance: tests.lock pins the case census, not the assertions; vendored Wycheproof vectors have no provenance pin

[lann]

conformance/README.md:32-36 says the lockfile means "case changes land intentionally via just update-conformance-lock with a reviewable diff." A reader reasonably concludes the suite's assertions are pinned. They are not.

render_lock (adapters/wasmtime/src/main.rs:233-250) emits exactly { name, features } per case, and the runner validates exactly name-set and tag equality (runner/src/main.rs:186-197). The lock is an inventory, and it is a good one — missing/extra/duplicate case, tag drift and missing-features drift are all caught in both directions. What it does not pin:

1. Vector bytes have no provenance. conformance/vectors/*.json are plain checked-in files. There is no checksum, no upstream Wycheproof commit pin, and no verification anywhere in the repo. Editing a tag field in hmac_sha256_test.json to match a buggy implementation produces zero lockfile diff and a green run. For a suite whose entire authority derives from being Wycheproof, that is the load-bearing gap.

2. Assertion strength is unpinned. Weakening Err(Error::AuthenticationFailed) => Ok(()) to Err(_) => Ok(()) anywhere in vectors.rs/probes.rs changes no case name and no tag; the lock is byte-identical. The suite's real strength today is that it discriminates specific error variants (vectors.rs:67, :140, :149, :169, :222, :248, :281; probes.rs:204, :257, :294, :427, :706, :891) — and nothing protects that property.

3. Hardcoded criteria are unpinned. SPECCHECK_VALID_CASE = 3 (translate.rs:690) is a magic index governing the entire Ed25519 verification-criterion claim.

Related, and worth stating because conformance/matrix.md publishes the counts: the ivSize != 96 ⇒ invalid-nonce rule (vectors/README.md:58) converts 80 of the 371 AES-GCM cases — including all 36 CounterWrap vectors — from cryptographic assertions into a nonce-length check. Correct per the WIT, honestly documented, but the matrix cell reads as more cryptographic coverage than it is.

Also: the deserialization structs (translate.rs:393-430, :715-746) declare no flags field and no deny_unknown_fields, so every Wycheproof flag (SmallIv, CounterWrap, EdgeCasePoly1305, SignatureMalleability, EdgeCasePublicKey, …) is silently discarded. Consequences: a failure message like aes-gcm/wycheproof/tc42/bytes carries no vector semantics, and a vector refresh introducing a new flag class needing different treatment is adopted with no policy review. Contrast is_valid's explicit panic on an unknown result (translate.rs:436-442) — that is the right pattern.

• Pin vector provenance: record the upstream Wycheproof commit and per-file digests, and verify them in the build (or vendor via a lockfile).
• Either extend the lock to cover assertion shape, or correct conformance/README.md:32-36 to say the lock pins the case inventory only.
• Carry flags through into case metadata so failures are self-describing, and reject unknown flags loudly the way unknown results already are.

From the 2026-07 adversarial review.

[lann]

PR #95 takes the first two boxes. Vector bytes are pinned in conformance/vectors/SHA256SUMS and checked by just verify-vectors as the first step of just conformance — verified by tampering with a result field, which now fails the run. And rather than extend the lock to cover assertion shape, I corrected the README to say what each mechanism actually covers: inventory by lockfile, bytes by digest, assertion strength by review alone. Claiming less felt better than implying a guarantee that is not there.

The third box is open. Carrying flags through is easy; the question it raises is whether they belong in case names — which would move both lockfiles — or only in failure messages. That is a call worth making explicitly.

[lann]

Correcting my earlier comment on this issue: it described a SHA256SUMS file checked by just verify-vectors, and that is not what merged in #95. The digest mechanism was removed during review, on the argument that it does not do work:

• A checksum stored beside the file it checksums carries no more authority than the file does — both are protected by the same diff and the same review, so the mechanism recurses without reaching firmer ground.
• Git already content-addresses these files. An edited tag field was never invisible; it was a diff.
• - "result": "valid" is a strictly more legible diff than a changed hex string, so the check displaced better evidence with worse.

The premise I had carried over from #88's toolchain pin does not hold here. There the digest is the only link to bytes fetched over the network that never appear in a diff; that is exactly why it is load-bearing there and vacuous here.

So the first box is ticked for the half with external authority — the upstream revision, which is what makes the files re-derivable and terminates the regress:

• C2SP/wycheproof@b61843a9a5115bb758134b6a1f5d5e502d445342 for the nine Wycheproof files
• novifinancial/ed25519-speccheck@65519336fda78a3d016e947df6d82848aca0c9da for the speccheck set

All ten were re-fetched at those exact revisions and confirmed byte-identical before recording. There is no build-time verification and no just recipe: a live check needs the network, and a gate that fails when GitHub is unreachable is worse than no gate. The NIST .rsp files ship in a zip with no revision to cite, which the README now says instead of implying a pin that does not exist.

The second box is ticked by correcting the README rather than extending the lock, so it states what each mechanism covers: the lockfile pins the inventory, the recorded revisions pin provenance, and assertion strength rests on review alone.

The third box is untouched and is why this stays open.