Skip to content

test(ci): prove the widened nightly notice actually fires for a red DAST run (BACKLOG #318) - #860

Merged
wshallwshall merged 2 commits into
mainfrom
worktree-agent-a02a6c84791dc7620
Sep 5, 2026
Merged

test(ci): prove the widened nightly notice actually fires for a red DAST run (BACKLOG #318)#860
wshallwshall merged 2 commits into
mainfrom
worktree-agent-a02a6c84791dc7620

Conversation

@wshallwshall

Copy link
Copy Markdown
Collaborator

BACKLOG #318, the known gap increment 1 carried. Scope was narrow on purpose: the notice, not increment 2.

What I found before touching anything

The watch-list widening the item asks for is already on main. Re-measured at a2eef0f37:

claim at HEAD
nightly-notice.yml watches only ["CI"] FALSE. It watches ["CI", "Security", "DAST"].
tests/test_nightly_notice.py pins that name Present, plus a per-name schedule: trigger check.
Increment 1 is on disk scripts/security/dast_auth_sweep.py, route_gates.py, dast-policy.json, .github/workflows/dast.yml, tests/test_dast_auth_sweep.py, tests/test_dast_claims.py, ADR 0155 all present.
Increment 1 passes pytest tests/test_dast_auth_sweep.py tests/test_dast_claims.py tests/test_nightly_notice.py -> 55 passed in 76.83s.

I could not attribute WHEN the widening landed: this worktree's clone is shallow (110 commits), so git log --diff-filter=A on that path reports the file as added at the shallow boundary rather than at its real birth. That is an instrument limit, not a finding.

So what I built instead

The widening was never the whole fix. Every assertion pinning it read YAML and matched strings. "The watch list contains DAST" says nothing about what happens when a DAST run completes, which is the same class of claim as the defect #318 filed, one level up.

The decision half is now driven:

  • The job's if: gate is evaluated against workflow_run payloads. The evaluator accepts exactly the <github path> == '<literal>' shape and refuses anything else, so rewriting the gate reds here rather than passing against an approximation. It is deliberately not a general Actions expression engine.
  • The step's run: body is executed VERBATIM under bash -e -- the shell Actions applies -- against a gh stub that journals every call. Title matching lives inside gh --jq, so the stub reads the searched-for title out of the expression the script built. "Which issue did this run look for?" becomes an observation, and the stub exits non-zero rather than answering a query it cannot parse.

Twelve rows. A red DAST nightly opens a DAST-titled issue with the run link and commit in the body; a second comments instead of duplicating; a green one closes it; cancelled/skipped/timed_out write nothing; and, the control that matters, a green CI nightly may not close the issue a red DAST run opened -- asserted in both directions.

How I proved it fires, rather than asserting it

Six defects injected into the shipped workflow, tests re-run, workflow reverted. Each is caught by the rows that name it and by no others:

injected defect result
drop DAST from the watch list 1 failed -- test_it_also_watches_the_dast_workflow
one shared issue title for all three workflows 4 failed -- create, dedupe, close, and the CI-cannot-close-DAST control
gate on push instead of schedule 3 failed -- both gate rows plus the structural one
silence the failure branch (open no issue) 2 failed -- both create rows
never close a recovered nightly 1 failed -- the close row
rename the jq selector past what the stub reads 8 failed -- the stub refuses loudly instead of faking "no match"

The stub's PATH shadowing is itself controlled. A real gh sits at C:\Program Files\GitHub CLI\gh.EXE on the measuring box; _assert_the_stub_won passes with the prepend and fails without it, so the stub genuinely displaced a live binary rather than filling a vacuum.

Checks run

Run with a python -m venv on cp314 (standard, not free-threaded), extras installed as CI installs them.

  • ruff check . -> All checks passed!
  • ruff format --check . -> 1241 files already formatted
  • mypy messagefoundry -> Success: no issues found in 267 source files
  • pre-commit run actionlint --files .github/workflows/nightly-notice.yml -> Lint GitHub Actions workflow files...Passed
  • pytest tests/test_nightly_notice.py tests/test_dast_auth_sweep.py tests/test_dast_claims.py tests/test_required_workflow_state.py tests/test_gate_ci_mirror_parity.py tests/test_gate_liveness.py tests/test_failure_signal.py tests/test_ci_red_reader.py tests/test_doc_guards_lane.py tests/test_link_resolution.py -> 173 passed in 204.58s
  • pytest tests/test_tooling_partition.py tests/test_lint_scope_parity.py tests/test_required_contexts.py -> 33 passed in 8.08s
  • pytest tests/test_backlog_status_check.py tests/test_backlog_hygiene_claim_extraction.py tests/test_dangling_citation_advisory.py tests/test_link_resolution.py -> 69 passed in 103.03s
  • Every pre-commit hook passed on both commits, actionlint included.

I did not run the full suite. It takes about twelve minutes and my turn is one turn; I ran the modules that touch what I changed plus the workflow-reading gates.

Legs a reader must check after my process exits

  • The three required test legs on ubuntu-latest, windows-2022 and windows-2025. The new rows execute bash, so the Windows legs are the ones that matter here -- tests/_bash_resolver.py is the shared resolver and it fails loudly rather than skipping, by design.
  • windows-service-smoke (NSSM), which never runs locally.
  • The web console leg on windows-2025, which the brief flags as able to red with 405 passed and zero failures.
  • tests/test_worktree_gate_control_plane.py is flagged as failing across unrelated PRs today. Not mine.

Left open

  1. gh issue list --limit 50 is a latent duplicate-opener. If more than 50 open issues carry the bug label, the notice's own issue can fall off the page it searches, and it opens a duplicate every night with nothing reporting a problem. Out of my scope, unallocated, and I did not file a number for it -- naming the subject rather than citing a #N I have not allocated.
  2. tests/tooling_manifest.txt needed no edit. tests/test_nightly_notice.py is already on line 118, and I added no new file.
  3. Branch name. I kept the worktree's own branch rather than renaming to a claude/... form, to avoid perturbing the worktree and collision gates.
  4. Two scoring-table rows in docs/BACKLOG.md still read workflows: ["CI"]. They are dated 2026-08-20 measurements and were true then. I left them as records and stated the current answer once, in the item body, rather than rewriting a dated figure. Say so if you would rather they were corrected in place.
  5. Increment 2 is untouched, as briefed -- no fuzzer, no OpenAPI overlay, no DICOM DIMSE, no /ui plane, no TLS black-box target, no non-GET reach. I judged none of the cheap pieces reachable inside one turn without a mutator or a fifth DEP-1 lock, so I took none.
  6. The delivery half remains unverifiable by construction. GitHub matches the completed run's name: and dispatches, and a workflow_run workflow only triggers from the default branch, so no PR can fire it. The first scheduled DAST run after this merges is the end-to-end evidence. The test module states this rather than papering over it.

Item NOT closed, no label applied, auto-merge not armed. Claim taken with scripts/coord/claim.ps1 -Take 318; no live lane held it.

🤖 Generated with Claude Code

wshallwshall and others added 2 commits September 4, 2026 17:50
…t (BACKLOG #318)

The DAST widening the item asks for is already on main -- nightly-notice.yml
watches CI, Security and DAST. What it has no coverage of is whether that
widening DOES anything: every assertion in tests/test_nightly_notice.py reads
YAML and matches strings, and "the watch list contains DAST" is the same class
of claim as the defect #318 filed, one level up.

So the decision half is now driven. The job's `if:` gate is evaluated against
workflow_run payloads, and the step's `run:` body is executed VERBATIM under
`bash -e` -- the shell Actions applies -- against a `gh` stub that journals
every call. The stub reads the title out of the `--jq` expression the script
built, so "which issue did this run look for?" is an observation rather than an
inference, and it exits non-zero rather than answering a query it cannot parse.

Twelve rows: a red DAST nightly opens a DAST-titled issue, a second comments
instead of duplicating, a green one closes it, cancelled/skipped/timed_out write
nothing, and -- the control that matters -- a GREEN CI nightly may not close the
issue a RED DAST run opened, in both directions.

Proven by mutation, not assertion. Six defects injected into the shipped
workflow, each caught by the rows that name it: dropping DAST from the watch
list, one shared issue title, gating on push, silencing the create, never
closing on recovery, and renaming the jq selector. The gh stub's shadowing is
itself controlled -- a real gh is on PATH and the check fails without the
prepend.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ts present tense

The Known-gap paragraph asserted that nightly-notice.yml watches only CI. That
stopped being true before this branch: it watches CI, Security and DAST on main
today. The paragraph now records what is present, what this branch adds, and
what is still increment 2.

The two scoring rows above the item carry the same superseded claim. They are
dated measurements from 2026-08-20 and are left standing rather than rewritten;
the item body is the current answer and says so.

Item NOT closed. Its closing act is `code` and increment 2 is untouched.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@wshallwshall wshallwshall added the reviewed A reviewer has read this. Removed automatically when new commits arrive. label Sep 4, 2026
@wshallwshall

Copy link
Copy Markdown
Collaborator Author

Disclosure: the reviewed label on this pull request was applied by the Manager seat that commissioned the diff.

It is not an independent read of the scope. COMMON.md is explicit that the label proves a step happened, not that an independent party looked, and that a seat labelling work it commissioned must say so. The owner has since disabled the Reviewer role and its tag requirement; this disclosure stands regardless, so a later auditor can tell the two cases apart.

What I actually checked, so the record is specific rather than a blanket claim:

  • Verified independently that origin/main already carries workflows: ["CI", "Security", "DAST"], so the fix this item asked for was already landed and my brief's premise was stale.
  • Confirmed the workflow change here is comment-only, and that the ledger row is its own last commit.
  • Glyph scan clean on added lines, with an injected control.

I did NOT reproduce the six injected-defect runs; those row counts are the Builder's.

@wshallwshall
wshallwshall added this pull request to the merge queue Sep 5, 2026
Merged via the queue into main with commit fc28a7e Sep 5, 2026
46 of 47 checks passed
@wshallwshall
wshallwshall deleted the worktree-agent-a02a6c84791dc7620 branch September 5, 2026 08:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

reviewed A reviewer has read this. Removed automatically when new commits arrive.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant