test(ci): prove the widened nightly notice actually fires for a red DAST run (BACKLOG #318) - #860
Merged
Merged
Conversation
…t (BACKLOG #318) The DAST widening the item asks for is already on main -- nightly-notice.yml watches CI, Security and DAST. What it has no coverage of is whether that widening DOES anything: every assertion in tests/test_nightly_notice.py reads YAML and matches strings, and "the watch list contains DAST" is the same class of claim as the defect #318 filed, one level up. So the decision half is now driven. The job's `if:` gate is evaluated against workflow_run payloads, and the step's `run:` body is executed VERBATIM under `bash -e` -- the shell Actions applies -- against a `gh` stub that journals every call. The stub reads the title out of the `--jq` expression the script built, so "which issue did this run look for?" is an observation rather than an inference, and it exits non-zero rather than answering a query it cannot parse. Twelve rows: a red DAST nightly opens a DAST-titled issue, a second comments instead of duplicating, a green one closes it, cancelled/skipped/timed_out write nothing, and -- the control that matters -- a GREEN CI nightly may not close the issue a RED DAST run opened, in both directions. Proven by mutation, not assertion. Six defects injected into the shipped workflow, each caught by the rows that name it: dropping DAST from the watch list, one shared issue title, gating on push, silencing the create, never closing on recovery, and renaming the jq selector. The gh stub's shadowing is itself controlled -- a real gh is on PATH and the check fails without the prepend. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ts present tense The Known-gap paragraph asserted that nightly-notice.yml watches only CI. That stopped being true before this branch: it watches CI, Security and DAST on main today. The paragraph now records what is present, what this branch adds, and what is still increment 2. The two scoring rows above the item carry the same superseded claim. They are dated measurements from 2026-08-20 and are left standing rather than rewritten; the item body is the current answer and says so. Item NOT closed. Its closing act is `code` and increment 2 is untouched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Collaborator
Author
|
Disclosure: the It is not an independent read of the scope. COMMON.md is explicit that the label proves a step happened, not that an independent party looked, and that a seat labelling work it commissioned must say so. The owner has since disabled the Reviewer role and its tag requirement; this disclosure stands regardless, so a later auditor can tell the two cases apart. What I actually checked, so the record is specific rather than a blanket claim:
I did NOT reproduce the six injected-defect runs; those row counts are the Builder's. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
BACKLOG #318, the known gap increment 1 carried. Scope was narrow on purpose: the notice, not increment 2.
What I found before touching anything
The watch-list widening the item asks for is already on
main. Re-measured ata2eef0f37:nightly-notice.ymlwatches only["CI"]["CI", "Security", "DAST"].tests/test_nightly_notice.pypins that nameschedule:trigger check.scripts/security/dast_auth_sweep.py,route_gates.py,dast-policy.json,.github/workflows/dast.yml,tests/test_dast_auth_sweep.py,tests/test_dast_claims.py, ADR 0155 all present.pytest tests/test_dast_auth_sweep.py tests/test_dast_claims.py tests/test_nightly_notice.py->55 passed in 76.83s.I could not attribute WHEN the widening landed: this worktree's clone is shallow (110 commits), so
git log --diff-filter=Aon that path reports the file as added at the shallow boundary rather than at its real birth. That is an instrument limit, not a finding.So what I built instead
The widening was never the whole fix. Every assertion pinning it read YAML and matched strings. "The watch list contains
DAST" says nothing about what happens when a DAST run completes, which is the same class of claim as the defect #318 filed, one level up.The decision half is now driven:
if:gate is evaluated againstworkflow_runpayloads. The evaluator accepts exactly the<github path> == '<literal>'shape and refuses anything else, so rewriting the gate reds here rather than passing against an approximation. It is deliberately not a general Actions expression engine.run:body is executed VERBATIM underbash -e-- the shell Actions applies -- against aghstub that journals every call. Title matching lives insidegh --jq, so the stub reads the searched-for title out of the expression the script built. "Which issue did this run look for?" becomes an observation, and the stub exits non-zero rather than answering a query it cannot parse.Twelve rows. A red DAST nightly opens a DAST-titled issue with the run link and commit in the body; a second comments instead of duplicating; a green one closes it;
cancelled/skipped/timed_outwrite nothing; and, the control that matters, a green CI nightly may not close the issue a red DAST run opened -- asserted in both directions.How I proved it fires, rather than asserting it
Six defects injected into the shipped workflow, tests re-run, workflow reverted. Each is caught by the rows that name it and by no others:
DASTfrom the watch listtest_it_also_watches_the_dast_workflowpushinstead ofscheduleThe stub's PATH shadowing is itself controlled. A real
ghsits atC:\Program Files\GitHub CLI\gh.EXEon the measuring box;_assert_the_stub_wonpasses with the prepend and fails without it, so the stub genuinely displaced a live binary rather than filling a vacuum.Checks run
Run with a
python -m venvon cp314 (standard, not free-threaded), extras installed as CI installs them.ruff check .->All checks passed!ruff format --check .->1241 files already formattedmypy messagefoundry->Success: no issues found in 267 source filespre-commit run actionlint --files .github/workflows/nightly-notice.yml->Lint GitHub Actions workflow files...Passedpytest tests/test_nightly_notice.py tests/test_dast_auth_sweep.py tests/test_dast_claims.py tests/test_required_workflow_state.py tests/test_gate_ci_mirror_parity.py tests/test_gate_liveness.py tests/test_failure_signal.py tests/test_ci_red_reader.py tests/test_doc_guards_lane.py tests/test_link_resolution.py->173 passed in 204.58spytest tests/test_tooling_partition.py tests/test_lint_scope_parity.py tests/test_required_contexts.py->33 passed in 8.08spytest tests/test_backlog_status_check.py tests/test_backlog_hygiene_claim_extraction.py tests/test_dangling_citation_advisory.py tests/test_link_resolution.py->69 passed in 103.03sI did not run the full suite. It takes about twelve minutes and my turn is one turn; I ran the modules that touch what I changed plus the workflow-reading gates.
Legs a reader must check after my process exits
testlegs onubuntu-latest,windows-2022andwindows-2025. The new rows executebash, so the Windows legs are the ones that matter here --tests/_bash_resolver.pyis the shared resolver and it fails loudly rather than skipping, by design.windows-service-smoke(NSSM), which never runs locally.windows-2025, which the brief flags as able to red with 405 passed and zero failures.tests/test_worktree_gate_control_plane.pyis flagged as failing across unrelated PRs today. Not mine.Left open
gh issue list --limit 50is a latent duplicate-opener. If more than 50 open issues carry thebuglabel, the notice's own issue can fall off the page it searches, and it opens a duplicate every night with nothing reporting a problem. Out of my scope, unallocated, and I did not file a number for it -- naming the subject rather than citing a#NI have not allocated.tests/tooling_manifest.txtneeded no edit.tests/test_nightly_notice.pyis already on line 118, and I added no new file.claude/...form, to avoid perturbing the worktree and collision gates.docs/BACKLOG.mdstill readworkflows: ["CI"]. They are dated 2026-08-20 measurements and were true then. I left them as records and stated the current answer once, in the item body, rather than rewriting a dated figure. Say so if you would rather they were corrected in place./uiplane, no TLS black-box target, no non-GETreach. I judged none of the cheap pieces reachable inside one turn without a mutator or a fifth DEP-1 lock, so I took none.name:and dispatches, and aworkflow_runworkflow only triggers from the default branch, so no PR can fire it. The first scheduled DAST run after this merges is the end-to-end evidence. The test module states this rather than papering over it.Item NOT closed, no label applied, auto-merge not armed. Claim taken with
scripts/coord/claim.ps1 -Take 318; no live lane held it.🤖 Generated with Claude Code