Skip to content

docs: runbooks, README, and design catalogs contradict the code (MCP runbook, resume story, bind default, event catalog, dead links) #1379

Description

@dennisonbertram

[parent] #1369

Work type

Bug / regression

Observed behavior

A read-only audit of README, CLAUDE.md, docs/runbooks and docs/design against the code at a0d1521 found these operator-facing docs make claims that send an operator to a failing command or describe behavior that does not exist:

  • docs/runbooks/mcp.md:86-108,188-201,212 describes an internal/mcpserver HTTP/SSE server with 10 tools mounted at /mcp; /mcp is actually harnessmcp.NewHTTPHandler (25 tools, POST only, cmd/harnessd/runtime_container.go:348-352, internal/harnessmcp/httptransport.go:29-31 returns 405 for GET). mcp.md:67 tool-name format {server}__{tool} vs code mcp_{server}_{tool} (tools/deferred/mcp.go:109); :61 collision "400" vs in-run failure; :69 nonexistent file path.
  • docs/runbooks/subagent-debugging.md:233-242,130 says max_steps yields run.completed with reason max_steps_reached; it is run.failed (runner.go:4655-4671). :89 ask_user_questionAskUserQuestion.
  • docs/runbooks/golden-path-deployment.md:48,70 bind default :8080 (now 127.0.0.1:8080, internal/config/config.go:254, bind_guard.go); :247-249 nonexistent profiles_dir config key; :262 nonexistent tools.allow config.toml key; :59 max_steps default 30 (profile value; daemon default becomes unlimited under bug(harnessd): implicit max_steps=8 default fails runs after eight tool calls; harness must have no default run cap #1376).
  • docs/runbooks/distribution.md:242,75 workspace_path checklist step and :8080 default (workspace_path becomes honored under bug(server): workspace_path sent by harnesscli --workspace and the TUI is silently ignored; tools run in the daemon workspace #1372; state that).
  • docs/runbooks/observational-memory.md:114 Postgres "planned" but implemented (store_postgres.go, wired main.go:1573); :19 model default only in openai mode.
  • docs/runbooks/profile-authoring.md:320-322 contradicts :60-62 on explicit false; :26-27 omits extends, max_turns. profile-operations.md:76-82 layering order reversed vs internal/config/config.go:3-8.
  • docs/runbooks/remote-cronsd.md:136-140 says Execution.RunID empty; populated (cron/types.go:124, scheduler.go:741).
  • docs/runbooks/terminal-bench-periodic-suite.md:13-31,50 task count 3 vs 21, model default, max-steps default 12 vs 100, upload path.
  • docs/runbooks/tool-usability-testing.md:47 wildcard bind now refused; :182-269 stale source paths; :317 nonexistent git_commit tool.
  • docs/runbooks/harnesscli-live-testing.md:17-31,90 missing flags, stale path. benchmark-smoke.md:218 task count.
  • docs/runbooks/INDEX.md:15,16,27 links to three nonexistent runbooks; two existing runbooks unlisted. README.md:189-190 same dead links; README.md:52 continue-as-resume; README.md:166 wrong run-request field names.
  • docs/design/event-catalog.md documents 23 of 81 events, says only two events are terminal (IsTerminalEvent includes run.cancelled, events.go:477-479), wrong tool name in the waiting_for_user payload. system-prompt-architecture.md:74 says skills are ignored; they are injected (systemprompt/engine.go:46-60). tool-roadmap.md:36, observational-memory-architecture.md:37, native-macos-app.md:29,66,153, plugins.md:163,210 stale paths. docs/INDEX.md omits four folders. CLAUDE.md:78 test count.

Expected behavior

Every route, field, event, tool, env var, flag, path, default and link in these files matches the code. Docs for behavior changed by #1372 (workspace_path honored), #1374 (harnesscli input, correct blocked hint), #1376 (no default step cap) describe the post-fix behavior and this PR merges after those.

Reproduction

Follow any high-severity item above, e.g. curl -N $BASE/mcp per mcp.md → 405; grep run.completed for max_steps_reached per subagent-debugging.md → nothing.

User and operational impact

Operators and agents (AGENTS.md routes them to these runbooks) act on wrong instructions; the MCP runbook and resume instructions fail outright.

Suspected seam and search evidence

Docs drifted after #1328 (bind default), the harnessmcp mount, event additions, and the profile/config cascade changes. Search commands: rg per claim (see attached audit).

Blast-radius impact map

Docs only. Callers: none. Config/env/defaults: docs state current values. API/CLI: none. Persistence: none. Clients: none. Tests: none. Documentation: the files listed.

Regression test first

Documentation-only slice: verification is the docs/runbooks/documentation-maintenance.md link check plus a manual re-verification of every corrected claim against the cited code line. Any doc claim that turns out to require a code change is filed as its own issue, not fixed here.

Fix boundaries

In scope: the files and claims listed above. Out of scope: rewriting prose style, archiving historical directories (separate child), any runtime change.

Diagnostic and observability evidence

Audit report attached to epic #1369 (comment), each finding cites doc:line and code file:line.

Verification plan

rg each corrected route/flag/env/tool/event against the code; run the repo link checker if present; go test ./... -run XXX builds nothing but confirms no accidental code edits; reviewer spot-checks 10 random corrected claims.

Rollout and rollback

None — docs only; revert PR.

Documentation and handoff

Engineering-log entry summarizing the audit and corrections; docs/runbooks/INDEX.md and docs/INDEX.md updated.

Definition of done

  • Every listed claim corrected with the code-verified value
  • No dead links in touched files
  • Engineering log entry present

Process acknowledgement

  • I will preserve the failing regression test, update the issue if the root cause changes, and record any newly discovered bug as its own issue.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingdocumentationImprovements or additions to documentation

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions