Skip to content

Add the harness guides, widen ACP capabilities, refuse stray top-level keys - #186

Merged
VeryComplexAndLongName merged 4 commits into
mainfrom
feat/harness-docs-and-config-strictness
Sep 2, 2026
Merged

Add the harness guides, widen ACP capabilities, refuse stray top-level keys#186
VeryComplexAndLongName merged 4 commits into
mainfrom
feat/harness-docs-and-config-strictness

Conversation

@VeryComplexAndLongName

Copy link
Copy Markdown
Owner

Three changes implemented into one working tree, plus two proposals that came out of doing the human-only checks.

agentic-harness-documentation

HARNESS.md and LIMITS.md exist, and README.md contains the word "harness" for the first time.

The agent reference table is read out of HARNESS_AGENT_CAPABILITIES rather than recalled. LIMITS.md separates the chain-wide budget evaluated between stages from the per-stage cap passed to one invocation, and says outright:

There is no wall-clock or duration limit on a harness run, and no per-stage timeout.

The question that prompted this document assumed one existed, which is exactly why it has to be stated.

Standalone screenshots are now produced by a Playwright spec, so the settings image can no longer show a screen that stopped existing three commits ago — which is what the committed one did.

acp-agent-capabilities

copilot-cli-acp and claude-cli-acp get the capabilities their plain counterparts have, because they run the same binaries with the same flags. codex-cli-acp and gemini-cli-acp get explicit empty entries, so an omission stops being spelled the same way as a decision.

harness-config-top-level-keys

A top-level key the schema does not define is refused — what would have caught the harness.json that sat in this repository doing nothing.


Two human-only tasks done, not deferred

harness-config-top-level-keys 3.6. The deleted file written back byte-for-byte and read through the real loader:

Invalid harness config: unrecognized top-level key "apply" (accepted keys: stepAgents, autonomyLevel, reviewGate, checkpoints, budget, gitStageAllowlist). Did you mean "stepAgents.apply"?

It used to return silently.

acp-agent-capabilities 3.7 — verified along the whole path, not at its ends, which is what harness-step-models needed three rounds to learn:

link result
real agent-harness.jsonresolveHarnessConfig loads (was refused)
buildInvocation copilot --acp --effort high --max-ai-credits 30
checkAllowlist {"allowed": true}
→ real binary via cross-spawn accepts the flags, waits on stdin for ACP

An unknown flag exits immediately and non-zero — which is how gh pr create --json was caught.

agentic-harness-documentation 6.7, standalone half. Checked against the generated images: every control the document places is where it says, effort and budget appear only where the agent accepts them, no model control, archive shows "runs mechanically — no agent", and the checkpoint offers Continue and Cancel and nothing else. The VS Code half cannot be done here and stays open, with 3.4.

Two proposals from doing that check

harness-git-stage-no-agent

The settings screenshot shows an agent picker on the git stage. Nothing reads stepAgents.gitCHAIN_STAGE_COMMAND covers only propose/review/apply/verify, and runStage routes git straight to runGitStage.

HARNESS.md already documents this honestly as a known defect. But a defect described in prose with no tracked change is how it becomes permanent. It is the same defect harness-mechanical-checks removed for archive, missed because git was not yet in CHAIN_STAGES when that change narrowed the type — both landed in #179.

core-test-worker-contention

git.push.test.ts takes 2.6 s alone and hangs past 20 s beside one other worker; both pass in 7.7 s under a single fork. The cause is contention over real git subprocesses and temp directories, not duration.

Raising the ceiling to 20000 ms was tried and produced a 20 s failure instead of a 5 s one — the clearest possible evidence that duration is not the problem. Two task notes had recorded this as "timeout flakiness", and one had worked around it with --pool=forks --poolOptions.forks.singleFork=true without recognising the flag as the diagnosis.

git.push.test.ts also moves afterEachafterAll for its cleanup, which is what ends its EBUSY ... rmdir failures — the fix git-fixture-test-cost already applied to the other two git-fixture files.

Test plan

  • npm run typecheck — clean
  • npm run lint — clean
  • packages/core512/512 under a single fork (the contention above is why the flag is needed; it is the subject of the new proposal, not a claim that the suite is fine)
  • openspec change validate --strict — all five changes valid
  • npm run lint:english — passed
  • CI green

🤖 Generated with Claude Code

…l keys

Three changes land together, implemented into one working tree.

HARNESS.md and LIMITS.md exist now, and README.md contains the word
harness for the first time. The reference table has a row per agent id
read out of HARNESS_AGENT_CAPABILITIES rather than recalled, LIMITS.md
separates the chain-wide budget evaluated between stages from the
per-stage cap passed to one invocation, and says outright that no
wall-clock limit exists - the question that prompted the document assumed
one did. The standalone screenshots are generated by a Playwright spec
now, so the settings image can no longer show a screen that stopped
existing three commits ago, which is what the committed one did.

acp-agent-capabilities gives copilot-cli-acp and claude-cli-acp the
capabilities their plain counterparts have, because they run the same
binaries with the same flags. codex-cli-acp and gemini-cli-acp get
explicit empty entries, so an omission stops being spelled the same way
as a decision.

harness-config-top-level-keys refuses a top-level key the schema does not
define, which is what would have caught the harness.json that sat in this
repository doing nothing.

Two human-only tasks are now done rather than deferred.

The top-level-key check was verified by writing the deleted file
byte-for-byte and reading it through the real loader: it now answers
"unrecognized top-level key apply ... Did you mean stepAgents.apply?"
where it used to return silently.

The ACP capabilities claim was verified along the whole path rather than
at its ends. The entry resolves through a real config file, builds the
argv copilot --acp --effort high --max-ai-credits 30, passes the
allowlist, and the real binary spawned through cross-spawn with that argv
accepts the flags and waits on stdin. An unknown flag exits immediately -
which is how gh pr create --json was caught.

The standalone half of the documentation check is done against the
generated images: every control the document places is where it says,
archive shows no picker, and the checkpoint offers Continue and Cancel
and nothing else. The VS Code half cannot be done here and stays open.

Two new proposals come out of doing that check.

harness-git-stage-no-agent: stepAgents.git is accepted and offered by
both settings surfaces, and nothing reads it. HARNESS.md already
documents this honestly, but a defect described in prose with no tracked
change is how it becomes permanent. It is the same defect
harness-mechanical-checks removed for archive, missed because git was not
in CHAIN_STAGES when that change narrowed the type - both landed in #179.

core-test-worker-contention: git.push.test.ts takes 2.6 s alone and hangs
past 20 s beside one other worker, and both pass in 7.7 s under a single
fork. The cause is contention over real git subprocesses and temp
directories, not duration. Raising the ceiling to 20000 ms was tried and
produced a 20 s failure instead of a 5 s one. Two task notes had recorded
this as timeout flakiness and one had worked around it with a pool flag
without recognising the flag as the diagnosis.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Task 6.7 asks for the document to be read beside the running surface.
Doing that for VS Code, against the implementation rather than a screen,
found the document overstating a defect it had otherwise described well.

HARNESS.md said "both UIs offer an agent picker" for stepAgents.git. Only
the standalone one does. HARNESS_TEMPLATE_STAGES in commands.ts lists
propose, review, apply, verify and archive, and the wizard never asks
about git, so VS Code already behaves the way harness-git-stage-no-agent
proposes to make both hosts behave.

It is right by accident rather than by decision: git is absent from that
list because the list predates the git stage. That change's task 5.4 -
every stage in CHAIN_STAGES without an entry in CHAIN_STAGE_COMMAND is
excluded from HarnessStepAgentStage - is what turns the coincidence into
a property, and its task 3.2 now says to confirm the wizard rather than
edit it.

The rest of the VS Code column checks out: both command titles match
package.json exactly, and the wizard is the sequential Quick Pick the
document describes, offering effort and budget only where
HARNESS_AGENT_CAPABILITIES accepts them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The second settings image was a full-page capture taken after loading the
per-change override, so it contained the whole global section as well -
a strict superset of the first image, presented in HARNESS.md as a
separate illustration of a different thing, and 2131 pixels tall for a
reader who only needed the half that differs. It is now scoped to the
override section: 36 KB instead of 394 KB.

`.last()` on the locator because the view nests one section inside
another and the filter matches the outer wrapper too.

Two further problems the images surfaced are left alone here, because
this change's own task 6.5 forbids touching non-test source, and both are
webui strings rather than screenshots.

The autonomy select reads "semi-autonomous (not yet implemented)" and
"autonomous (not yet implemented)". Both are implemented -
HarnessChainRunner has a test suite for each level, and the very next
screenshot in the document shows a semi-autonomous run paused at a
checkpoint. The document now carries a label contradicting the image
below it.

The panel's own description points at openspec/changes/agentic-harness/,
which was archived and no longer exists at that path.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The ACP section said copilot-cli-acp and claude-cli-acp refuse effort and
budget, and pointed at acp-agent-capabilities as the change that would fix
it. That change is implemented in this same branch, so the section
described a state that no longer exists by the time anyone could read it.
HARNESS_AGENT_CAPABILITIES now carries both rows, and the document says so
instead. This is the reconciliation both changes asked whichever landed
second to perform.

The guide also claimed VS Code screenshots below were captured by hand and
labelled with their date. There are none - no image under
docs/images/extension is referenced anywhere in the file. A sentence
describing the provenance of images that do not exist is worse than
silence, because it reads as a guarantee. Replaced with the plain fact
that no VS Code capture exists yet and that adding one is the open
human-only task.

Both images are now links to their own full-resolution PNG with a caption
saying what to look at, and a navigation table sits at the top. The
per-change caption names the (not yet implemented) suffix visible in the
screenshot as stale UI copy and points at the section describing the
semi-autonomous behaviour that suffix denies - the label is wrong in the
product, and a caption is the only place this change is permitted to say
so, its own task 6.5 forbidding non-test source edits.

Verified before committing: no extension image is referenced anywhere in
the file, the two ACP rows exist in harness-step-agent.ts, and every
internal anchor the new navigation table introduces resolves to a heading
that exists.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@VeryComplexAndLongName
VeryComplexAndLongName merged commit 348ee61 into main Sep 2, 2026
7 checks passed
@VeryComplexAndLongName
VeryComplexAndLongName deleted the feat/harness-docs-and-config-strictness branch September 3, 2026 11:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant