Skip to content

Propose two harness guides, and one defect found while writing them - #181

Merged
VeryComplexAndLongName merged 1 commit into
mainfrom
docs/harness-guides
Sep 2, 2026
Merged

Propose two harness guides, and one defect found while writing them#181
VeryComplexAndLongName merged 1 commit into
mainfrom
docs/harness-guides

Conversation

@VeryComplexAndLongName

Copy link
Copy Markdown
Owner

Two proposals. No implementation yet.

agentic-harness-documentation

README.md does not contain the word "harness". The largest capability this repository has built has no document saying what it is or how to configure it — what exists is spread across nineteen ADRs and forty change proposals, which record why each decision was made and are the wrong shape for someone who wants to use the thing.

HARNESS.md — how to set it up. The stage sequence, both configuration files and their merge rule, every key with its accepted values, where each is edited in both interfaces, mechanical checks, and one table with a row per agent id: model, effort and which values, spending cap and in which unit, and whether any binary for it has ever run here.

LIMITS.md — what stops a run, and when. Two independent levels: a chain-wide budget evaluated between stages (so it cannot stop a stage already running), and a per-stage budget passed to one CLI invocation as its own flag. Plus what does not exist:

There is no wall-clock or duration limit on a harness run, and no per-stage timeout.

The durations in the code are not user settings — external-waiter.ts's maxDurationMs, gh-pr-gateway.ts's five-minute check poll, agent detection's timeout, the CI job ceilings. The request that prompted this asked about time limits as though they existed, which is exactly why the document has to say so outright.

Screenshots get generated

docs/images/standalone/harness-settings.png is from 2026-08-31 and shows a settings screen that stopped existing three commits ago — before the effort and budget controls, the chat target, and the mechanical archive row. grep -rn "docs/images" over the repository's code, CI and tooling returns nothing: all seventeen images are manual captures, and nothing announced that one had rotted.

packages/server already runs Playwright with four specs and a CI job, so the standalone images become a product of the code the tests exercise. VS Code's cannot be automated — that host has not been startable in this environment even for its own integration tests (see audit-log-persistence task 4.2) — so those stay manual and carry the date and version they show. That is the honest version of the same guarantee, rather than claiming an automation that a person will keep performing by hand.

One thing is not written as asked

The request asks to document that agents communicate over ACP. The accurate statement is narrower: ACP is the protocol between the harness and one agent process — structured session/update progress instead of scraped text, and a session/request_permission gate where the agent offers one. It is not a channel between two agents, and no agent here sends a message to another. ADR 0018's event-driven orchestration is between stages, which is a different mechanism and also not agent-to-agent messaging.

Writing it the other way would put a capability in the documentation that does not exist in the product — the class of defect this repository has spent the week removing from its own configuration surface. The document says what ACP gives and says plainly what it does not.

acp-agent-capabilities — found while writing the table

HARNESS_AGENT_CAPABILITIES has rows for the five CLI adapters and vscode-chat. It has no rows for the four ACP ids, and by the table's own documented contract an absent row means "no mechanism at all" — so effort and budget are refused for all four.

For two of them that is wrong, and the evidence is three files apart:

renders allowlist permits validator says
copilot-cli-acp --effort, --max-ai-credits both no mechanism
claude-cli-acp --effort, --max-budget-usd both no mechanism

The adapter builds the flag, the allowlist admits the flag, and the validator refuses the configuration that would produce it.

This is the mirror image of what harness-config-strictness removed. There the validator accepted a setting nothing could honour; here it refuses one the runtime honours. Same failure — validator and runtime disagreeing — and the refusal is the more confusing direction, because it presents as a deliberate decision.

The practical cost is specific: copilot-cli is the cheap agent, --max-ai-credits is its only spending cap, and copilot-cli-acp is the adapter this project recommends for structured output. Today those three cannot be combined — ACP or the cap, not both.

The fix is the table and nothing else. Adapters and allowlist are already correct. It also requires explicit empty entries for codex-cli-acp and gemini-cli-acp, which genuinely render nothing, so an omission stops being spelled the same way as a decision — and a test that every registered id has a row, since that is what failed silently here.

Test plan

  • openspec change validate --strict — both valid
  • npm run lint:english — passed
  • Agent ids, capabilities, allowlist entries and adapter flags read out of the source, not recalled
  • docs/images/standalone/harness-settings.png staleness confirmed against git log
  • CI green

🤖 Generated with Claude Code

README.md does not contain the word "harness". The Agentic Harness is the
largest capability this repository has built, and a reader arriving here
has no document saying what it is or how to configure it. What exists is
spread across nineteen ADRs and forty change proposals, which record why
each decision was made and are the wrong shape for someone who wants to
use the thing.

HARNESS.md will answer how to set it up: the stage sequence, both
configuration files and their merge rule, every key with its accepted
values, where each is edited in both interfaces, what mechanical checks
are, and one table with a row per agent id saying whether it accepts a
model, a reasoning effort and which values, a spending cap and in which
unit, and whether any binary for it has ever run here.

LIMITS.md will answer what stops a run and when. Two independent levels:
a chain-wide budget evaluated between stages, which cannot stop a stage
already running, and a per-stage cap passed to one CLI invocation as its
own flag. It also has to say what does not exist - there is no wall-clock
or duration limit on a harness run and no per-stage timeout. The request
that prompted this asked about time limits as though there were.

The screenshots get generated rather than captured. The committed
docs/images/standalone/harness-settings.png is from 2026-08-31 and shows
a settings screen that stopped existing three commits ago, before the
effort and budget controls, the chat target and the mechanical archive
row. packages/server already runs Playwright, so the standalone images
become a product of the code the tests exercise. VS Code's cannot be
automated - that host has not been startable here even for its own
integration tests - so those stay manual and carry the date and version
they show, which is the honest version of the same guarantee.

One thing the request asked for is not written as asked. ACP is the
protocol between the harness and one agent process: structured progress
instead of scraped text, and a permission gate where the agent offers
one. It is not a channel between two agents, and no agent here sends a
message to another. Documenting it as agent-to-agent communication would
put a capability in the documentation that does not exist in the product,
which is the class of defect this repository has spent the week removing
from its own configuration surface.

Writing the agent table found a real one. HARNESS_AGENT_CAPABILITIES has
no rows for the four ACP ids, and an absent row means "no mechanism at
all" by the table's own contract - so effort and budget are refused for
all four. For two of them that is wrong: copilot-acp.ts renders --effort
and --max-ai-credits, claude-acp.ts renders --effort and
--max-budget-usd, and default-runners.ts already permits every one of
those flags. The adapter builds the flag, the allowlist admits it, and
the validator refuses the configuration that would produce it.

That is the mirror image of what harness-config-strictness removed: there
the validator accepted a setting nothing could honour, here it refuses one
the runtime honours. The practical cost is specific - copilot-cli is the
cheap agent, --max-ai-credits is its only spending cap, and
copilot-cli-acp is the adapter this project recommends. Today those three
cannot be combined.

acp-agent-capabilities fixes the table and nothing else; the adapters and
the allowlist are already correct, which is what makes it a one-table
change. It also requires an explicit empty entry for the two ACP adapters
that genuinely render nothing, so an omission stops being spelled the same
way as a decision.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@VeryComplexAndLongName
VeryComplexAndLongName merged commit 5b85f3c into main Sep 2, 2026
7 checks passed
@VeryComplexAndLongName
VeryComplexAndLongName deleted the docs/harness-guides branch September 3, 2026 11:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant