Skip to content

conformance-ct: generated rows have no per-case inventory pin — the census function rides on CI matrix count diffs #302

Description

@lann

The incumbent tests.lock enumerated every case (16,427 names + tags) and the runner required exact per-target agreement: missing case, extra case, and tag drift were each hard errors (7110a99:conformance/runner/src/main.rs:222-247). The component-test lock stores, for the ~17,960 vector/contract-derived cases, only [[generated]] row prefixes + tags — no case list, no count (guest-ct/tests.lock; format at component-test lockfile.rs:36-43). Verified consequences in component-test source at the pin:

  • check_coverage requires exact [[case]] entries (probes/declines, ~59 of ~18k) but for generated leaves checks only grammar and at-most-once — no lower bound; a row reporting zero leaves passes (lockfile.rs:164-212).
  • The aggregate's per-target coverage is exactly check_coverage (aggregate.rs:228-230); no cross-target leaf-set agreement exists, so a uniform drop (same suite artifact everywhere) is invisible to aggregation.
  • lock --check compares prefixes + tags only (component-test-cli main.rs:98-105).

Residual pins, per row class:

  • Incumbent rows: the native census test expands plan::ROWS and requires exact id+tag equality with the frozen fixture (census_test.rs), so a translate-level regression fails it — but only via the JSON path; a regression confined to the rkyv emission (build.rs/corpus.rs row splitting) is unobserved by it.
  • Post-cutover rows (ecdsa-p256-sha512/wycheproof, ecdsa-p384-sha512/wycheproof from conformance: vendor the SHA-512 ECDSA P1363 files; execute the cross-hash verify paths #299): deliberately outside plan::ROWS and the fixture. Their populations are pinned by exactly one artifact: the committed matrix row label ("816 cases") diffed by matrix-check — which is CI-only (not part of conformance-ct::all locally), count-granular (a same-count case substitution passes every gate; the old lock diff listed each changed case name), and refreshed wholesale by matrix-update (cp).

conformance/README.md's claims are currently stronger than the mechanism: "Growing the suites therefore never silently sheds coverage" and "the inventory is the binding" hold at prefix granularity for ~99.6% of cases.

Fix shapes (either restores a real lower bound):

  • per-row case counts or a leaf-set digest in the lockfile format (upstream component-test change), or
  • a census-test row per new prefix (a second, growing fixture — the frozen incumbent fixture stays frozen), plus adding matrix-check to local all.

Related: #83 records that the old lock pinned inventory but not assertions; this issue records that the new lock pins neither, per-case, for generated rows. The S5 ratification issue (filed separately) depends on this one.

Found by adversarial review of the #274 migration; mechanics independently confirmed by two reviewers against component-test at c90b2f0.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions