[Spec Change] Suite/result signing for integrity verification

[adhabnr-ux]

Flagged in spec/CRITIQUE.md #9 and deferred out of 1.0 on purpose — this is one of the two open items (along with the conformance suite below) that's explicitly scoped for v1.1/v2.0 rather than guessed at under release pressure.

The problem: there's currently no way to verify an EvalPort suite or result set hasn't been tampered with after publication. A benchmark dataset hosted publicly (e.g. anything in benchmarks/) could be silently modified to favor a specific model, and a consumer has no spec-level way to detect that.

What exists today as a stopgap: metadata.source for provenance tracking, and the assumption that Git/version control provides an audit trail. Neither is a real integrity guarantee — Git history can be rewritten on a fork, and metadata.source is just a string nobody verifies.

Open questions for this thread:

1. Detached signatures (a .sig file alongside the suite) vs. an embedded metadata.openeval.signature field — the former keeps the core document untouched by crypto concerns, the latter is easier to lose track of when suites get copied around.
2. What gets signed — the canonical JSON serialization (requires nailing down a canonicalization algorithm, e.g. JCS/RFC 8785) or the raw bytes as published (simpler, but two byte-identical-content files with different whitespace would have different signatures)?
3. Key management and trust — do we point at existing infra (Sigstore/cosign, PGP) rather than inventing a new one? EvalPort has no reason to build its own PKI.
4. Is this a MUST for the spec to reach 2.0, or does it stay an optional extension indefinitely, the way openeval.cost is optional?

If you've worked on this problem for another data format (SBOM signing, npm provenance, in-toto, etc.) and have opinions on what transfers cleanly, that's exactly the kind of prior art this thread needs.

[adhabnr-ux]

One piece of prior art that's directly relevant and already partly adopted in this repo: Discussion #12 mentions the SDK publish pipeline was switched to OIDC Trusted Publishing for PyPI/npm (tracked in #7) specifically to get rid of long-lived tokens. That's the same trust root Sigstore's keyless signing flow (cosign sign-blob, or the sigstore-python/sigstore-js libraries directly) uses — a GitHub Actions OIDC token proves "this artifact was built by workflow X in repo Y at commit Z," and Sigstore's Fulcio CA issues a short-lived cert against that identity instead of anyone managing a long-lived private key. If EvalPort already trusts GitHub Actions OIDC identity enough to publish packages with it, extending that same trust root to sign suites/result sets published from CI is a small conceptual step, not new infrastructure — no new PKI, no key rotation/revocation story to build, and it's the same approach npm provenance attestations and most modern SBOM tooling converged on for exactly this reason.

That mostly answers question 3 in the original post (existing infra vs. inventing one) in favor of Sigstore over PGP — PGP requires the humans/CI publishing suites to actually manage and protect a long-lived private key, which is the exact failure mode Trusted Publishing was adopted to get away from a few weeks ago per #12.

On questions 1 and 2 (detached vs. embedded, what gets signed), I'd lean detached and against inventing a canonicalization scheme, for a mechanical reason more than a philosophical one: signing the canonical JSON serialization (JCS/RFC 8785) means the verifier has to canonicalize before checking, which means the verifier needs a correct, spec-compliant JCS implementation in every language EvalPort tooling exists in (Python, TypeScript, and whatever a third-party runner uses) — that's real surface area for the canonicalization step itself to have bugs or cross-implementation disagreements, which would be a uniquely bad place for a bug (a subtly-wrong canonicalizer either rejects valid signed suites or, worse, could be tricked into accepting a tampered one). Signing the raw published bytes sidesteps that entirely — no canonicalization library needed anywhere, at the cost of "two byte-identical-content files with different whitespace have different signatures," which is a real cost but a much easier one to live with (re-sign after any re-serialization, don't hand-edit a signed file) than "every implementation's canonicalizer must agree forever." Detached also matches what cosign sign-blob --bundle already produces natively — a .sigstore.json (or .intoto.jsonl for a fuller in-toto attestation, if EvalPort wants provenance metadata beyond just "signed," like build parameters) sidecar, not something inserted into the suite document itself. Embedding would also create a chicken-and-egg problem this way: you can't sign the final document if the signature is itself a field inside the thing being signed, without excluding that field from what gets hashed — which is exactly the kind of subtlety that raw-bytes + detached avoids by construction.

On question 4 — agree this should stay optional indefinitely rather than a MUST for 2.0, same treatment as openeval.cost. Most consumers of a locally-generated eval result set have no threat model that requires this; it matters specifically for publicly-hosted benchmarks (the benchmarks/ case the original post calls out), where "did anyone tamper with this after I fetched it" is a real question. Making it mandatory everywhere would be spec weight nobody but that use case needs.

If there's rough agreement on Sigstore + detached + raw-bytes, I'd be glad to sketch what the actual .sig/.sigstore.json sidecar convention and a spec/tools/verify-signature reference script would look like — happy to prototype rather than just argue for it in the abstract.