feat(eval): operator offline answer-key signing CLI - #370
Merged
Conversation
Adds operations_center.eval.sign — the tool the operator runs OFFLINE to perform the one irreducibly-human step in the trust-hardening spec: anchoring the EVAL answer key. Nothing automated calls it; the private key it consumes must never touch a fleet host (a key reachable by a compromised agent could forge answer-key labels — the exact attack the design prevents). - `keygen`: generate an Ed25519 keypair; writes the PEM private key (kept offline), prints the public-key hex to paste into eval/constitution/operator_pubkey.ed25519. - `sign`: convert unsigned candidate cases into signed graded cases with the offline private key (idempotent; --case-id to limit). Re-chains the ledger via the new corpus.write_ledger (adding a signature changes an entry's hash, so the chain after it is recomputed) — verify_chain still validates the result. Verified end-to-end with a throwaway ephemeral key (never committed): report-only → sign seeds → gate graduates to blocking, all pass; an in-place edit of a signed answer is caught (entry_hash mismatch). 41 eval unit tests (keygen roundtrip, candidate→graded conversion, idempotency, case-id limiting, CLI flows); ruff/ty/D12 clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
load_public_key_from_hex was a module-level helper, not a test, but its non-test_ name made the Custodian N2 detector read it as an uncollected test. Renamed to _pubkey_from_hex (clearly private). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Owner
Author
Failed checks: code_quality |
Dead code — defined but never called (call sites use key.public_key() directly). Removing it (and the now-unused Ed25519PublicKey import) resolves the reviewer's code_quality concern. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds the `operations-center-eval-sign` console script to [project.scripts] so `operations_center.eval.sign:main` is reachable as a production CLI — resolving the D12 finding (symbol tested but never wired). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ProtocolWarden
added a commit
that referenced
this pull request
Jun 21, 2026
Records the merged EVAL machinery (#369 + #370) in HARNESS_TRUST_HARDENING.md's Phase-4 section — corpus hash-chain tamper-evidence, Ed25519 answer-key signing + offline CLI, deterministic replay blocking gate, different-family drift monitor, monotonic constitution + required integrity check, 7 seeded candidates — and spells out what is still deferred (the operator key-anchor, the Component-2 outcome flagger, D-EVAL-4 over-flag attribution, the live drift-monitor model adapter) so the doc reads scaffolding-done, not phase-complete. Docs-only. Co-authored-by: ProtocolWarden <ProtocolWarden@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
operations_center.eval.sign— the tool the operator runs offline to perform the one irreducibly-human step in the trust-hardening spec: anchoring the EVAL answer key so the gate can graduate from report-only to blocking.This is the deliberate completion of the human-anchor seam left by #369. It is not automated and not called by any fleet/controller path — the private key it consumes must never touch a fleet host (a key reachable by a compromised agent could forge answer-key labels, which is the exact attack the design prevents).
Subcommands
keygen— generate an Ed25519 keypair. Writes the PEM private key (you keep it offline) and prints the public-key hex to paste intoeval/constitution/operator_pubkey.ed25519.sign— convert unsigned candidate cases into signed graded cases with your offline private key (idempotent;--case-idto limit which). Re-chains the ledger via the newcorpus.write_ledger(adding a signature changes an entry's hash, so the chain after it is recomputed) —verify_chainstill validates the result.Why a CLI instead of just doing it
I declined to generate or hold the signing key in-session. A key generated on a fleet-reachable host, by the agent that builds the eval, would collapse the un-forgeable anchor the whole design rests on (self-dealing + a label-forging key sitting next to the attacker). The tooling keeps key generation and custody with the operator while making their manual step a single command.
Verified end-to-end (throwaway ephemeral key, never committed)
41 eval unit tests (keygen roundtrip, candidate→graded conversion, idempotency, case-id limiting, CLI flows). ruff / ty / Custodian D12 clean.
Operator runbook (to graduate the real gate, when ready)
python -m operations_center.eval.sign keygen --private-out operator_priv.pem→ keepoperator_priv.pemoff all fleet hosts.eval/constitution/operator_pubkey.ed25519.python -m operations_center.eval.sign sign --private operator_priv.pem(after adding cases to reachmin_graded_cases).🤖 Generated with Claude Code