Add RUNNER-CONTRACT.md and nine-scenario conformance suite#19
Merged
Conversation
ROADMAP Part 3 #6: make loop.resolved.json an interchange format instead of an internal detail, so ports (the Hermes signal) become allies under a contract rather than forks. RUNNER-CONTRACT.md is the normative v1 contract for runners: inputs, path safety, model invocation, gate/verdict semantics (including which configurations make judge criteria unreachable), caps and termination, fail-closed consent with the leak warning visible before the consent question, two-layer redaction (path-based non-send + content scrub with surfacing, host included, unscrubbable files reported), state and run-log obligations, and exit codes. conformance/check_runner.py proves a runner honors it: nine scenarios with self-contained deterministic fixtures (fake host/judge, no model CLIs) - happy path, judge-degrade, consent fail-closed, prompt redaction, host-prompt scrub, context non-send, cmd-output scrub, workspace escape refusal, revision cap. A .py runner runs under the harness interpreter; anything else executes directly. The reference templates/run-loop.py is held to the suite via a new unittest wrapper, so CI enforces the contract on every change (48 tests total). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The high-effort review proved four of five redaction scenarios could false-PASS (a runner with all scrub layers disabled passed them) and found normative text contradicting the reference runner. All 10 findings addressed: Harness (now 10 scenarios; sabotage-tested): - The fake host gains a "leak file" mode: flagged content enters plan.md through the normal draft flow instead of a pre-seeded artifact no contract clause requires runners to reuse. - All redaction scenarios assert on the host capture as well as the judge capture (flagged context/cmd content lands in the HOST prompt first - asserting only on the judge was vacuous). - New default-redactions scenario: the .env default glob set must apply with an empty privacy.egress. - revision-cap uses a stall-proof judge (unique blocking issue per round) so a no-progress detector cannot mask a missing cap, and asserts the exact judge call count and that no delivery is drafted. - judge-degrade asserts review artifacts from real revision rounds (degrade-to-revise, not crash-on-bad-output). - Verified against a sabotaged runner (redaction disabled): the five redaction scenarios all FAIL, everything else still passes. Contract: - Default redaction globs enumerated normatively (.env, .env.*, secrets/**, **/*.key) - they are absent from the resolved spec and existed only in code. - Invocation conventions written down (spec path as sole positional arg, loop dir as cwd, subprocess cwd = loop dir). - Context-path escape: MUST NOT read; abort or blocked-marker both conformant (matches the reference, which continues with a marker). - Exit codes: 0=passed is the only MUST; the 1/2 split is SHOULD, with the state file as the normative failure record (the reference maps wall-clock aborts to 2). Runner: - Any crash - not just RunnerError - now leaves state.json terminal instead of a phantom "running" (contract section 5 was right and the runner was wrong; non-UTF-8 context files reproduced it). 49 tests total; conformance 10/10 on the reference runner. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
ROADMAP Part 3 #6 — the host-portability bet.
loop.resolved.jsonbecomes an interchange format with a published contract and a way to prove conformance, so third-party runners (the Hermes port signal) become allies under a contract instead of forks.RUNNER-CONTRACT.md (v1)
Normative MUST/SHOULD contract for anything that executes a compiled Looper spec, grounded in (and enforced against) the reference runner's actual behavior: inputs, path safety, model invocation, gate/verdict semantics, caps and termination, fail-closed consent (with the leak warning shown before the consent question), two-layer redaction with surfacing (host included; unscrubbable flagged files reported as blind spots), state/run-log obligations, exit codes. Versioned with the spec format (v1).
conformance/check_runner.py
Nine scenarios, each scaffolding a fixture loop with deterministic fake host/judge scripts (no real model CLIs), invoking the candidate runner, and asserting the contract's observable obligations:
happy-path,judge-degrade,consent-fail-closed,prompt-redaction,host-prompt-scrub,context-non-send,cmd-output-scrub,workspace-escape,revision-capAssertions are runner-agnostic (exit codes, state file contents, artifacts, and what reached the fake models' stdin) — no reference-runner log-format coupling. Notably,
cmd-output-scrubandhost-prompt-scrubfail against the pre-#18 runner, so the suite demonstrably has teeth.CI
A new unittest wrapper holds
templates/run-loop.pyto the suite on every change (48 tests total). README gains a "Building your own runner" section.🤖 Generated with Claude Code