Fix the two reasons a consumer hand-rolled a control we publish - #11
Merged
Conversation
On 2026-08-02 the scheduled consumer-drift sweep went red for the first time: `30 hand-rolled control(s) across 7 repo(s); 29 allowlisted; 1 new`. The day before it read `29 ... 0 new`. The new control is aml-filter/ci.yml/secret-scan, from aml-filter#89. That PR closed a genuine hole -- aml-filter's gitleaks scan ran only in its weekly security-audit.yml sweep and never on a pull request, so a secret could merge and sit in public history for up to seven days. It closed that hole by inlining gitleaks/gitleaks-action: the exact control this repo publishes as secret-scan.yml, at the identical pinned action SHA. A security fix shipped by hand-rolling the shared brick. The brick was a drop-in -- secret-scan.yml takes no required inputs, and examples/aml-filter/security-audit.yml already showed aml-filter calling it. So part of the cause is habit. Two parts were ours: 1. examples/aml-filter/ci.yml carried no secret-scan job. The worked example this repo publishes for the exact file being edited offered nothing to copy. Fixed: the example now has the caller. 2. Nothing warned that adopting a reusable workflow RENAMES its check run to "<caller job> / <called job>". aml-filter's branch protection requires a context named literally "gitleaks"; adopting makes it "Secret scan / gitleaks" and blocks every merge until protection is updated. That cost is paid by the adopter and invisible to whoever published the brick -- one concrete reason inlining keeps winning. Fixed: it has its own README section, and the example carries the warning inline. Also pinned the property the finding depended on. aml-filter/ci.yml/frontend-gate has been allowlisted since 2026-07-26; had the allowlist key been read at file granularity, that older entry would have swallowed the new secret-scan control and the sweep would have reported a clean run on the day it mattered most. The new fixture asserts an entry covers one control, not the file it lives in. Proven by mutation, not by assertion: widening allowlist_index to compare <repo>/<workflow> makes the detector report `2 allowlisted; 0 new` on a fixture holding one known and one brand-new control -- it swallows the new one and exits 0 -- and the new cases go red. Restored byte-identical afterwards. The detector was NOT weakened. It still exits 1 on any un-allowlisted control. The one new allowlist entry is a pointer to an open converging PR on aml-filter, carries the full reason above, and is marked for deletion when that PR lands; a stale entry warns and never fails, so deleting it early is free. The allowlist header also stops claiming "nothing here is new drift", which was true for exactly seven days. Claim touched: "new drift with no entry fails the build" -- unchanged and now regression-tested. "Adoption is the fix, not an exemption" -- upheld: the consumer is converging, and both ci-side reasons it did not converge the first time are fixed here. On Dagger: the 2026-07-31 decision set a disconfirming test -- attempt the convergence sweep, and if new hand-rolled controls reappear within 60 days, the model is the problem. This is NOT that test firing. Its precondition was never met: no drafted caller had been adopted, so nothing could reappear after converging. The narrower true statement, recorded in the README, is that drift accrued while convergence had not started. Evidence: security-policy.sh passed consumer-drift-cases.sh passed (3 new cases) example-fidelity-cases.sh passed lineage-guard-cases.sh passed shellcheck -x clean actionlint (examples) clean zizmor 1.26.1 (examples) no findings example fidelity 173 resolved, 0 MISSING, 0 UNVERIFIABLE, 7 repos Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0186xrrT9mYfCZTVkk8HN12p
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
Converging aml-filter (aml-filter#93) turned up a cause the desk analysis missed, because it only appears when someone tries to adopt. This repo tags releases `ci-vX.Y.Z`. Third-party actions tag `vN`. So the trailing comment on a correct first-party pin reads `# ci-v3.0.0`. aml-filter's own supply-chain test lints pinned-`uses:` comments against `^v\d` and therefore REJECTED the correct pin -- the converging PR went red on the consumer's own guard, and the only way to satisfy that regex would have been to write a comment naming a tag that does not exist. The fix belongs in the consumer's guard and it tightens rather than loosens: key the expected scheme off the ref, so neither convention is accepted for the other. Documented here with that shape, so the next adopter reads it before hitting it. That makes three separate taxes on doing the right thing -- a missing example, a renamed check run, and a rejected pin comment -- and none of them is charged to the person who inlines the action instead. That asymmetry is the finding. Also swapped the allowlist entry's branch reference for aml-filter#93 now that the converging PR exists. Evidence: consumer-drift-cases.sh passed security-policy.sh passed README anchors 3 new cross-links resolve Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0186xrrT9mYfCZTVkk8HN12p
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What happened
On 2026-08-02 the scheduled
Consumer driftsweep went red for the first time since it shipped:run 30739082151. The 08-01 run read
29 … 0 new, so the window is one day.The new control:
aml-filter/.github/workflows/ci.yml/secret-scan, introduced by aml-filter#89 (2026-08-01).That PR closed a real hole. aml-filter's gitleaks scan ran only in its weekly
security-audit.ymlsweep and never on a pull request, so a secret could merge and sit in public history for up to seven days. It closed that hole by inlininggitleaks/gitleaks-action— the exact control this repo publishes assecret-scan.yml, at the identical pinned action SHA. A security fix shipped by hand-rolling the shared brick.Why — and it is partly our fault
secret-scan.ymlwas a genuine drop-in: no required inputs, same action SHA, andexamples/aml-filter/security-audit.ymlalready showed aml-filter calling it. So part of the cause is habit. But two parts were ours, and both are fixed here.ci-side causeexamples/aml-filter/ci.ymlhad no secret-scan job. The worked example we publish for the exact file being edited offered nothing to copy.<caller job> / <called job>. aml-filter's branch protection requires a context named literallygitleaks; adopting makes itSecret scan / gitleaksand blocks every merge until protection is updated.Cause #2 is the interesting one. It is a real cost of adoption, paid by the person converging and invisible to the person who published the brick. Inlining never renames anything. Verified against this repo's own dogfood job, whose check run is literally
Secret scan (own brick) / gitleaks.Adoption is the fix, not an exemption
The consumer is converging — see the aml-filter PR linked below. The allowlist entry added here is a pointer to that open PR, not an exemption: it carries the full reason, and is marked for deletion when that PR lands. A stale entry warns and never fails, so deleting it early is free.
The detector was not weakened
It still exits 1 on any un-allowlisted control. What this PR adds is a regression test for the property the finding depended on.
aml-filter/ci.yml/frontend-gatehas been allowlisted since 2026-07-26. Had the allowlist key been read at file granularity, that older entry would have swallowed the newsecret-scancontrol and the sweep would have reported a clean run on the day it mattered most.Proven by mutation, not asserted. Widening
allowlist_indexto compare<repo>/<workflow>:and the new cases go red:
Detector restored byte-identical; suite green again.
On the Dagger question
The 2026-07-31 decision not to adopt dagger.io set a disconfirming test: attempt the convergence sweep, and if new hand-rolled controls reappear within 60 days, the model is the problem, not the backlog.
This is not that test firing. Its precondition was never met — no drafted caller had been adopted, so nothing could "reappear" after converging, and one new control cannot tell "the model is wrong" apart from "nobody has run the sweep yet". The narrower true statement, recorded in the README: drift accrued while convergence had not started. One new control out of thirty, from a PR fixing a real security gap, does not re-open Dagger on its own. It is recorded so the next one lands on a record instead of a blank page.
Evidence
tests/security-policy.shtests/consumer-drift-cases.shtests/example-fidelity-cases.shtests/lineage-guard-cases.shshellcheck -xNot merging
Paired with the converging PR on
hseshadr/aml-filter. Note that the aml-filter PR requires a branch-protection update (thegitleaksrequired context becomesSecret scan / gitleaks) — that is an owner action and is spelled out in that PR.🤖 Generated with Claude Code
https://claude.ai/code/session_0186xrrT9mYfCZTVkk8HN12p