Skip to content

Run the git stage against a real repository, and record what it did not cover - #184

Merged
VeryComplexAndLongName merged 1 commit into
mainfrom
test/git-stage-live-verification
Sep 2, 2026
Merged

Run the git stage against a real repository, and record what it did not cover#184
VeryComplexAndLongName merged 1 commit into
mainfrom
test/git-stage-live-verification

Conversation

@VeryComplexAndLongName

Copy link
Copy Markdown
Owner

Closes agentic-harness-git-stage task 4.4, which had been marked done with a note saying it had not been executed.

What ran

Against VeryComplexAndLongName/TestRepo, the scratch repository the owner provided. Seeded with a workflow whose single job fails when a SHOULD_FAIL marker file is present, so both outcomes could be produced on demand.

Driven through the real GitWrapper and the real PullRequestGatewayreal git, real gh, no mocks.

Passing path — PR #1, MERGED

[live] opened https://github.com/VeryComplexAndLongName/TestRepo/pull/1
[live] checks: {"state":"pass"}
[live] merged https://github.com/VeryComplexAndLongName/TestRepo/pull/1

A branch with no upstream pushed through push("origin", branch); gh pr create opened the pull request and the number came from the URL it printed; checks passed; gh pr view reported MERGED.

Refusing path — PR #2, still OPEN

[live] opened .../pull/2 (expected to fail its check)
[live] checks: {"state":"fail","reason":"check failed: Verify (failure)"}

The merge was never attempted, and the pushed branch and open pull request were left for a person — as ADR 0014 requires. That pull request is deliberately still open in the scratch repository, as the evidence.

Why this run and not a mocked one

Each of the three defects review found is one this would have caught and no mocked test could:

Defect Why only a live run finds it
--json on gh pr create the flag exists in no version of gh; only the real binary refuses it
SKIPPED read as a failure SKIPPED appears only in real check output
bare git push after an allowlisted git push <remote> <branch> a branch with no upstream is the only place it fails

The temporary test file that drove it was deleted rather than committed: it pushes to a live repository, and a suite CI runs is the wrong home for that.

What it did not cover — new task 4.4a, unchecked

The chain's own wiring around the stage: shouldRunGitStage re-reading the per-change harness.json, the allowlist check and audit entry for each of the three actions, and the completed event after the merge. Reaching those live needs archive to succeed first, which needs the disposable repository to be an OpenSpec project — a larger setup than the task described.

They have unit coverage with a stubbed gateway (under agent-sufficient plus allowlist, runs git push -> pr create -> merge, blocks a non-allowlisted git target before any push/pr/merge call, and the refusal cases). That is the right level for wiring, and it is not the same as having watched it. Better to say which is which than to let one checkmark imply both.

Test plan

  • Live run, both paths, against a real GitHub repository
  • openspec change validate --strict agentic-harness-git-stage — valid
  • npm run lint:english — passed
  • Temporary live test removed; no source change in this PR
  • CI green

🤖 Generated with Claude Code

…ot cover

Task 4.4 was marked done with a note saying it had not been executed. It
has now been executed, against VeryComplexAndLongName/TestRepo, a scratch
repository the owner provided. The repository was seeded with a workflow
whose one job fails when a SHOULD_FAIL marker is present, so both
outcomes could be produced on demand, and the run went through the real
GitWrapper and the real PullRequestGateway - real git, real gh, no mocks.

Passing path, PR #1: a branch with no upstream pushed through
push("origin", branch); gh pr create opened the pull request and the
number came from the URL it printed; waitForChecks returned pass; the
merge succeeded and gh pr view reported MERGED.

Refusing path, PR #2: the same sequence with the marker committed.
waitForChecks returned {"state":"fail","reason":"check failed: Verify
(failure)"}, the merge was never attempted, and gh pr view reported OPEN
- the pushed branch and open pull request left for a person, as ADR 0014
requires. That pull request is deliberately still open as the evidence.

Each of the three defects review found is one this run would have caught
and no mocked test could. --json on gh pr create is refused by the real
binary. SKIPPED appears only in real check output. A branch with no
upstream is the only place a bare git push fails.

The temporary test file that drove it was deleted rather than committed:
it pushes to a live repository, and a suite CI runs is the wrong home for
that.

A new task 4.4a records what the run did not cover, unchecked. The
chain's own wiring around the stage - shouldRunGitStage re-reading the
per-change harness.json, the allowlist check and audit entry for each
action, the completed event after the merge - needs archive to succeed
first, which needs the disposable repository to be an OpenSpec project.
Those junctions have unit coverage with a stubbed gateway, which is the
right level for wiring and is not the same as having watched it. Better
to say which is which than to let one checkmark imply both.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@VeryComplexAndLongName
VeryComplexAndLongName merged commit 43bb63e into main Sep 2, 2026
7 checks passed
@VeryComplexAndLongName
VeryComplexAndLongName deleted the test/git-stage-live-verification branch September 3, 2026 11:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant