Skip to content

feat(eval): operator offline answer-key signing CLI - #370

Merged
ProtocolWarden merged 4 commits into
mainfrom
feat/eval-operator-signing-cli
Jun 21, 2026
Merged

feat(eval): operator offline answer-key signing CLI#370
ProtocolWarden merged 4 commits into
mainfrom
feat/eval-operator-signing-cli

Conversation

@ProtocolWarden

Copy link
Copy Markdown
Owner

What

Adds operations_center.eval.sign — the tool the operator runs offline to perform the one irreducibly-human step in the trust-hardening spec: anchoring the EVAL answer key so the gate can graduate from report-only to blocking.

This is the deliberate completion of the human-anchor seam left by #369. It is not automated and not called by any fleet/controller path — the private key it consumes must never touch a fleet host (a key reachable by a compromised agent could forge answer-key labels, which is the exact attack the design prevents).

Subcommands

  • keygen — generate an Ed25519 keypair. Writes the PEM private key (you keep it offline) and prints the public-key hex to paste into eval/constitution/operator_pubkey.ed25519.
  • sign — convert unsigned candidate cases into signed graded cases with your offline private key (idempotent; --case-id to limit which). Re-chains the ledger via the new corpus.write_ledger (adding a signature changes an entry's hash, so the chain after it is recomputed) — verify_chain still validates the result.

Why a CLI instead of just doing it

I declined to generate or hold the signing key in-session. A key generated on a fleet-reachable host, by the agent that builds the eval, would collapse the un-forgeable anchor the whole design rests on (self-dealing + a label-forging key sitting next to the attacker). The tooling keeps key generation and custody with the operator while making their manual step a single command.

Verified end-to-end (throwaway ephemeral key, never committed)

report-only: 0/7 signed → gate report-only, all 7 candidates pass
keygen → sign 7 seeds
after: 7 graded → gate [blocking]: all graded cases pass and floor is cleared
tamper (flip a signed answer in place) → TAMPER: entry_hash mismatch, RESULT FAIL

41 eval unit tests (keygen roundtrip, candidate→graded conversion, idempotency, case-id limiting, CLI flows). ruff / ty / Custodian D12 clean.

Operator runbook (to graduate the real gate, when ready)

  1. Offline: python -m operations_center.eval.sign keygen --private-out operator_priv.pem → keep operator_priv.pem off all fleet hosts.
  2. Paste the printed public hex as the sole first line of eval/constitution/operator_pubkey.ed25519.
  3. Review the candidate cases, then python -m operations_center.eval.sign sign --private operator_priv.pem (after adding cases to reach min_graded_cases).
  4. Commit the pubkey + signed ledger. CI flips the gate to blocking.

🤖 Generated with Claude Code

ProtocolWarden and others added 2 commits June 21, 2026 07:38
Adds operations_center.eval.sign — the tool the operator runs OFFLINE to perform
the one irreducibly-human step in the trust-hardening spec: anchoring the EVAL
answer key. Nothing automated calls it; the private key it consumes must never
touch a fleet host (a key reachable by a compromised agent could forge answer-key
labels — the exact attack the design prevents).

- `keygen`: generate an Ed25519 keypair; writes the PEM private key (kept offline),
  prints the public-key hex to paste into eval/constitution/operator_pubkey.ed25519.
- `sign`: convert unsigned candidate cases into signed graded cases with the
  offline private key (idempotent; --case-id to limit). Re-chains the ledger via
  the new corpus.write_ledger (adding a signature changes an entry's hash, so the
  chain after it is recomputed) — verify_chain still validates the result.

Verified end-to-end with a throwaway ephemeral key (never committed): report-only
→ sign seeds → gate graduates to blocking, all pass; an in-place edit of a signed
answer is caught (entry_hash mismatch).

41 eval unit tests (keygen roundtrip, candidate→graded conversion, idempotency,
case-id limiting, CLI flows); ruff/ty/D12 clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
load_public_key_from_hex was a module-level helper, not a test, but its
non-test_ name made the Custodian N2 detector read it as an uncollected test.
Renamed to _pubkey_from_hex (clearly private).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@ProtocolWarden

ProtocolWarden commented Jun 21, 2026

Copy link
Copy Markdown
Owner Author

Resolved: superseded by new push — re-review resumed

Self-review concerns — auto-fixing (up to 6 attempts; re-queued if still unresolved):

Failed checks: code_quality

ProtocolWarden and others added 2 commits June 21, 2026 07:54
Dead code — defined but never called (call sites use key.public_key()
directly). Removing it (and the now-unused Ed25519PublicKey import) resolves
the reviewer's code_quality concern.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds the `operations-center-eval-sign` console script to [project.scripts]
so `operations_center.eval.sign:main` is reachable as a production CLI —
resolving the D12 finding (symbol tested but never wired).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@ProtocolWarden
ProtocolWarden merged commit 3b49daf into main Jun 21, 2026
23 checks passed
@ProtocolWarden
ProtocolWarden deleted the feat/eval-operator-signing-cli branch June 21, 2026 11:59
ProtocolWarden added a commit that referenced this pull request Jun 21, 2026
Records the merged EVAL machinery (#369 + #370) in HARNESS_TRUST_HARDENING.md's
Phase-4 section — corpus hash-chain tamper-evidence, Ed25519 answer-key signing +
offline CLI, deterministic replay blocking gate, different-family drift monitor,
monotonic constitution + required integrity check, 7 seeded candidates — and
spells out what is still deferred (the operator key-anchor, the Component-2
outcome flagger, D-EVAL-4 over-flag attribution, the live drift-monitor model
adapter) so the doc reads scaffolding-done, not phase-complete. Docs-only.

Co-authored-by: ProtocolWarden <ProtocolWarden@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant