Add the harness guides, widen ACP capabilities, refuse stray top-level keys - #186
Merged
VeryComplexAndLongName merged 4 commits intoSep 2, 2026
Merged
Conversation
…l keys Three changes land together, implemented into one working tree. HARNESS.md and LIMITS.md exist now, and README.md contains the word harness for the first time. The reference table has a row per agent id read out of HARNESS_AGENT_CAPABILITIES rather than recalled, LIMITS.md separates the chain-wide budget evaluated between stages from the per-stage cap passed to one invocation, and says outright that no wall-clock limit exists - the question that prompted the document assumed one did. The standalone screenshots are generated by a Playwright spec now, so the settings image can no longer show a screen that stopped existing three commits ago, which is what the committed one did. acp-agent-capabilities gives copilot-cli-acp and claude-cli-acp the capabilities their plain counterparts have, because they run the same binaries with the same flags. codex-cli-acp and gemini-cli-acp get explicit empty entries, so an omission stops being spelled the same way as a decision. harness-config-top-level-keys refuses a top-level key the schema does not define, which is what would have caught the harness.json that sat in this repository doing nothing. Two human-only tasks are now done rather than deferred. The top-level-key check was verified by writing the deleted file byte-for-byte and reading it through the real loader: it now answers "unrecognized top-level key apply ... Did you mean stepAgents.apply?" where it used to return silently. The ACP capabilities claim was verified along the whole path rather than at its ends. The entry resolves through a real config file, builds the argv copilot --acp --effort high --max-ai-credits 30, passes the allowlist, and the real binary spawned through cross-spawn with that argv accepts the flags and waits on stdin. An unknown flag exits immediately - which is how gh pr create --json was caught. The standalone half of the documentation check is done against the generated images: every control the document places is where it says, archive shows no picker, and the checkpoint offers Continue and Cancel and nothing else. The VS Code half cannot be done here and stays open. Two new proposals come out of doing that check. harness-git-stage-no-agent: stepAgents.git is accepted and offered by both settings surfaces, and nothing reads it. HARNESS.md already documents this honestly, but a defect described in prose with no tracked change is how it becomes permanent. It is the same defect harness-mechanical-checks removed for archive, missed because git was not in CHAIN_STAGES when that change narrowed the type - both landed in #179. core-test-worker-contention: git.push.test.ts takes 2.6 s alone and hangs past 20 s beside one other worker, and both pass in 7.7 s under a single fork. The cause is contention over real git subprocesses and temp directories, not duration. Raising the ceiling to 20000 ms was tried and produced a 20 s failure instead of a 5 s one. Two task notes had recorded this as timeout flakiness and one had worked around it with a pool flag without recognising the flag as the diagnosis. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Task 6.7 asks for the document to be read beside the running surface. Doing that for VS Code, against the implementation rather than a screen, found the document overstating a defect it had otherwise described well. HARNESS.md said "both UIs offer an agent picker" for stepAgents.git. Only the standalone one does. HARNESS_TEMPLATE_STAGES in commands.ts lists propose, review, apply, verify and archive, and the wizard never asks about git, so VS Code already behaves the way harness-git-stage-no-agent proposes to make both hosts behave. It is right by accident rather than by decision: git is absent from that list because the list predates the git stage. That change's task 5.4 - every stage in CHAIN_STAGES without an entry in CHAIN_STAGE_COMMAND is excluded from HarnessStepAgentStage - is what turns the coincidence into a property, and its task 3.2 now says to confirm the wizard rather than edit it. The rest of the VS Code column checks out: both command titles match package.json exactly, and the wizard is the sequential Quick Pick the document describes, offering effort and budget only where HARNESS_AGENT_CAPABILITIES accepts them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The second settings image was a full-page capture taken after loading the per-change override, so it contained the whole global section as well - a strict superset of the first image, presented in HARNESS.md as a separate illustration of a different thing, and 2131 pixels tall for a reader who only needed the half that differs. It is now scoped to the override section: 36 KB instead of 394 KB. `.last()` on the locator because the view nests one section inside another and the filter matches the outer wrapper too. Two further problems the images surfaced are left alone here, because this change's own task 6.5 forbids touching non-test source, and both are webui strings rather than screenshots. The autonomy select reads "semi-autonomous (not yet implemented)" and "autonomous (not yet implemented)". Both are implemented - HarnessChainRunner has a test suite for each level, and the very next screenshot in the document shows a semi-autonomous run paused at a checkpoint. The document now carries a label contradicting the image below it. The panel's own description points at openspec/changes/agentic-harness/, which was archived and no longer exists at that path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The ACP section said copilot-cli-acp and claude-cli-acp refuse effort and budget, and pointed at acp-agent-capabilities as the change that would fix it. That change is implemented in this same branch, so the section described a state that no longer exists by the time anyone could read it. HARNESS_AGENT_CAPABILITIES now carries both rows, and the document says so instead. This is the reconciliation both changes asked whichever landed second to perform. The guide also claimed VS Code screenshots below were captured by hand and labelled with their date. There are none - no image under docs/images/extension is referenced anywhere in the file. A sentence describing the provenance of images that do not exist is worse than silence, because it reads as a guarantee. Replaced with the plain fact that no VS Code capture exists yet and that adding one is the open human-only task. Both images are now links to their own full-resolution PNG with a caption saying what to look at, and a navigation table sits at the top. The per-change caption names the (not yet implemented) suffix visible in the screenshot as stale UI copy and points at the section describing the semi-autonomous behaviour that suffix denies - the label is wrong in the product, and a caption is the only place this change is permitted to say so, its own task 6.5 forbidding non-test source edits. Verified before committing: no extension image is referenced anywhere in the file, the two ACP rows exist in harness-step-agent.ts, and every internal anchor the new navigation table introduces resolves to a heading that exists. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
VeryComplexAndLongName
deleted the
feat/harness-docs-and-config-strictness
branch
September 3, 2026 11:15
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three changes implemented into one working tree, plus two proposals that came out of doing the human-only checks.
agentic-harness-documentationHARNESS.mdandLIMITS.mdexist, andREADME.mdcontains the word "harness" for the first time.The agent reference table is read out of
HARNESS_AGENT_CAPABILITIESrather than recalled.LIMITS.mdseparates the chain-widebudgetevaluated between stages from the per-stage cap passed to one invocation, and says outright:The question that prompted this document assumed one existed, which is exactly why it has to be stated.
Standalone screenshots are now produced by a Playwright spec, so the settings image can no longer show a screen that stopped existing three commits ago — which is what the committed one did.
acp-agent-capabilitiescopilot-cli-acpandclaude-cli-acpget the capabilities their plain counterparts have, because they run the same binaries with the same flags.codex-cli-acpandgemini-cli-acpget explicit empty entries, so an omission stops being spelled the same way as a decision.harness-config-top-level-keysA top-level key the schema does not define is refused — what would have caught the
harness.jsonthat sat in this repository doing nothing.Two human-only tasks done, not deferred
harness-config-top-level-keys3.6. The deleted file written back byte-for-byte and read through the real loader:It used to return silently.
acp-agent-capabilities3.7 — verified along the whole path, not at its ends, which is whatharness-step-modelsneeded three rounds to learn:agent-harness.json→resolveHarnessConfigbuildInvocationcopilot --acp --effort high --max-ai-credits 30checkAllowlist{"allowed": true}cross-spawnAn unknown flag exits immediately and non-zero — which is how
gh pr create --jsonwas caught.agentic-harness-documentation6.7, standalone half. Checked against the generated images: every control the document places is where it says, effort and budget appear only where the agent accepts them, no model control,archiveshows "runs mechanically — no agent", and the checkpoint offers Continue and Cancel and nothing else. The VS Code half cannot be done here and stays open, with 3.4.Two proposals from doing that check
harness-git-stage-no-agentThe settings screenshot shows an agent picker on the
gitstage. Nothing readsstepAgents.git—CHAIN_STAGE_COMMANDcovers only propose/review/apply/verify, andrunStageroutesgitstraight torunGitStage.HARNESS.mdalready documents this honestly as a known defect. But a defect described in prose with no tracked change is how it becomes permanent. It is the same defectharness-mechanical-checksremoved forarchive, missed becausegitwas not yet inCHAIN_STAGESwhen that change narrowed the type — both landed in #179.core-test-worker-contentiongit.push.test.tstakes 2.6 s alone and hangs past 20 s beside one other worker; both pass in 7.7 s under a single fork. The cause is contention over realgitsubprocesses and temp directories, not duration.Raising the ceiling to 20000 ms was tried and produced a 20 s failure instead of a 5 s one — the clearest possible evidence that duration is not the problem. Two task notes had recorded this as "timeout flakiness", and one had worked around it with
--pool=forks --poolOptions.forks.singleFork=truewithout recognising the flag as the diagnosis.git.push.test.tsalso movesafterEach→afterAllfor its cleanup, which is what ends itsEBUSY ... rmdirfailures — the fixgit-fixture-test-costalready applied to the other two git-fixture files.Test plan
npm run typecheck— cleannpm run lint— cleanpackages/core— 512/512 under a single fork (the contention above is why the flag is needed; it is the subject of the new proposal, not a claim that the suite is fine)openspec change validate --strict— all five changes validnpm run lint:english— passed🤖 Generated with Claude Code