Propose two harness guides, and one defect found while writing them - #181
Merged
Conversation
README.md does not contain the word "harness". The Agentic Harness is the largest capability this repository has built, and a reader arriving here has no document saying what it is or how to configure it. What exists is spread across nineteen ADRs and forty change proposals, which record why each decision was made and are the wrong shape for someone who wants to use the thing. HARNESS.md will answer how to set it up: the stage sequence, both configuration files and their merge rule, every key with its accepted values, where each is edited in both interfaces, what mechanical checks are, and one table with a row per agent id saying whether it accepts a model, a reasoning effort and which values, a spending cap and in which unit, and whether any binary for it has ever run here. LIMITS.md will answer what stops a run and when. Two independent levels: a chain-wide budget evaluated between stages, which cannot stop a stage already running, and a per-stage cap passed to one CLI invocation as its own flag. It also has to say what does not exist - there is no wall-clock or duration limit on a harness run and no per-stage timeout. The request that prompted this asked about time limits as though there were. The screenshots get generated rather than captured. The committed docs/images/standalone/harness-settings.png is from 2026-08-31 and shows a settings screen that stopped existing three commits ago, before the effort and budget controls, the chat target and the mechanical archive row. packages/server already runs Playwright, so the standalone images become a product of the code the tests exercise. VS Code's cannot be automated - that host has not been startable here even for its own integration tests - so those stay manual and carry the date and version they show, which is the honest version of the same guarantee. One thing the request asked for is not written as asked. ACP is the protocol between the harness and one agent process: structured progress instead of scraped text, and a permission gate where the agent offers one. It is not a channel between two agents, and no agent here sends a message to another. Documenting it as agent-to-agent communication would put a capability in the documentation that does not exist in the product, which is the class of defect this repository has spent the week removing from its own configuration surface. Writing the agent table found a real one. HARNESS_AGENT_CAPABILITIES has no rows for the four ACP ids, and an absent row means "no mechanism at all" by the table's own contract - so effort and budget are refused for all four. For two of them that is wrong: copilot-acp.ts renders --effort and --max-ai-credits, claude-acp.ts renders --effort and --max-budget-usd, and default-runners.ts already permits every one of those flags. The adapter builds the flag, the allowlist admits it, and the validator refuses the configuration that would produce it. That is the mirror image of what harness-config-strictness removed: there the validator accepted a setting nothing could honour, here it refuses one the runtime honours. The practical cost is specific - copilot-cli is the cheap agent, --max-ai-credits is its only spending cap, and copilot-cli-acp is the adapter this project recommends. Today those three cannot be combined. acp-agent-capabilities fixes the table and nothing else; the adapters and the allowlist are already correct, which is what makes it a one-table change. It also requires an explicit empty entry for the two ACP adapters that genuinely render nothing, so an omission stops being spelled the same way as a decision. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two proposals. No implementation yet.
agentic-harness-documentationREADME.mddoes not contain the word "harness". The largest capability this repository has built has no document saying what it is or how to configure it — what exists is spread across nineteen ADRs and forty change proposals, which record why each decision was made and are the wrong shape for someone who wants to use the thing.HARNESS.md— how to set it up. The stage sequence, both configuration files and their merge rule, every key with its accepted values, where each is edited in both interfaces, mechanical checks, and one table with a row per agent id: model, effort and which values, spending cap and in which unit, and whether any binary for it has ever run here.LIMITS.md— what stops a run, and when. Two independent levels: a chain-widebudgetevaluated between stages (so it cannot stop a stage already running), and a per-stagebudgetpassed to one CLI invocation as its own flag. Plus what does not exist:The durations in the code are not user settings —
external-waiter.ts'smaxDurationMs,gh-pr-gateway.ts's five-minute check poll, agent detection's timeout, the CI job ceilings. The request that prompted this asked about time limits as though they existed, which is exactly why the document has to say so outright.Screenshots get generated
docs/images/standalone/harness-settings.pngis from 2026-08-31 and shows a settings screen that stopped existing three commits ago — before the effort and budget controls, the chat target, and the mechanicalarchiverow.grep -rn "docs/images"over the repository's code, CI and tooling returns nothing: all seventeen images are manual captures, and nothing announced that one had rotted.packages/serveralready runs Playwright with four specs and a CI job, so the standalone images become a product of the code the tests exercise. VS Code's cannot be automated — that host has not been startable in this environment even for its own integration tests (seeaudit-log-persistencetask 4.2) — so those stay manual and carry the date and version they show. That is the honest version of the same guarantee, rather than claiming an automation that a person will keep performing by hand.One thing is not written as asked
The request asks to document that agents communicate over ACP. The accurate statement is narrower: ACP is the protocol between the harness and one agent process — structured
session/updateprogress instead of scraped text, and asession/request_permissiongate where the agent offers one. It is not a channel between two agents, and no agent here sends a message to another. ADR 0018's event-driven orchestration is between stages, which is a different mechanism and also not agent-to-agent messaging.Writing it the other way would put a capability in the documentation that does not exist in the product — the class of defect this repository has spent the week removing from its own configuration surface. The document says what ACP gives and says plainly what it does not.
acp-agent-capabilities— found while writing the tableHARNESS_AGENT_CAPABILITIEShas rows for the five CLI adapters andvscode-chat. It has no rows for the four ACP ids, and by the table's own documented contract an absent row means "no mechanism at all" — soeffortandbudgetare refused for all four.For two of them that is wrong, and the evidence is three files apart:
copilot-cli-acp--effort,--max-ai-creditsclaude-cli-acp--effort,--max-budget-usdThe adapter builds the flag, the allowlist admits the flag, and the validator refuses the configuration that would produce it.
This is the mirror image of what
harness-config-strictnessremoved. There the validator accepted a setting nothing could honour; here it refuses one the runtime honours. Same failure — validator and runtime disagreeing — and the refusal is the more confusing direction, because it presents as a deliberate decision.The practical cost is specific:
copilot-cliis the cheap agent,--max-ai-creditsis its only spending cap, andcopilot-cli-acpis the adapter this project recommends for structured output. Today those three cannot be combined — ACP or the cap, not both.The fix is the table and nothing else. Adapters and allowlist are already correct. It also requires explicit empty entries for
codex-cli-acpandgemini-cli-acp, which genuinely render nothing, so an omission stops being spelled the same way as a decision — and a test that every registered id has a row, since that is what failed silently here.Test plan
openspec change validate --strict— both validnpm run lint:english— passeddocs/images/standalone/harness-settings.pngstaleness confirmed againstgit log🤖 Generated with Claude Code