Skip to content

Fix the two reasons a consumer hand-rolled a control we publish - #11

Merged
hseshadr merged 2 commits into
mainfrom
fix/secret-scan-adoption-friction
Aug 2, 2026
Merged

Fix the two reasons a consumer hand-rolled a control we publish#11
hseshadr merged 2 commits into
mainfrom
fix/secret-scan-adoption-friction

Conversation

@hseshadr

@hseshadr hseshadr commented Aug 2, 2026

Copy link
Copy Markdown
Owner

What happened

On 2026-08-02 the scheduled Consumer drift sweep went red for the first time since it shipped:

30 hand-rolled control(s) across 7 repo(s); 29 allowlisted; 1 new

run 30739082151. The 08-01 run read 29 … 0 new, so the window is one day.

The new control: aml-filter / .github/workflows/ci.yml / secret-scan, introduced by aml-filter#89 (2026-08-01).

That PR closed a real hole. aml-filter's gitleaks scan ran only in its weekly security-audit.yml sweep and never on a pull request, so a secret could merge and sit in public history for up to seven days. It closed that hole by inlining gitleaks/gitleaks-action — the exact control this repo publishes as secret-scan.yml, at the identical pinned action SHA. A security fix shipped by hand-rolling the shared brick.

Why — and it is partly our fault

secret-scan.yml was a genuine drop-in: no required inputs, same action SHA, and examples/aml-filter/security-audit.yml already showed aml-filter calling it. So part of the cause is habit. But two parts were ours, and both are fixed here.

# ci-side cause Fix in this PR
1 examples/aml-filter/ci.yml had no secret-scan job. The worked example we publish for the exact file being edited offered nothing to copy. The example now carries the caller.
2 Nothing warned that adopting renames the check run. A caller job reports as <caller job> / <called job>. aml-filter's branch protection requires a context named literally gitleaks; adopting makes it Secret scan / gitleaks and blocks every merge until protection is updated. New README section Adopting a reusable workflow renames its check run, plus the warning inline in the example.

Cause #2 is the interesting one. It is a real cost of adoption, paid by the person converging and invisible to the person who published the brick. Inlining never renames anything. Verified against this repo's own dogfood job, whose check run is literally Secret scan (own brick) / gitleaks.

Adoption is the fix, not an exemption

The consumer is converging — see the aml-filter PR linked below. The allowlist entry added here is a pointer to that open PR, not an exemption: it carries the full reason, and is marked for deletion when that PR lands. A stale entry warns and never fails, so deleting it early is free.

The detector was not weakened

It still exits 1 on any un-allowlisted control. What this PR adds is a regression test for the property the finding depended on.

aml-filter/ci.yml/frontend-gate has been allowlisted since 2026-07-26. Had the allowlist key been read at file granularity, that older entry would have swallowed the new secret-scan control and the sweep would have reported a clean run on the day it mattered most.

Proven by mutation, not asserted. Widening allowlist_index to compare <repo>/<workflow>:

halfknown  ci.yml  dependency-audit  DRIFT (allowlisted)
2 hand-rolled control(s) across 1 repo(s); 2 allowlisted; 0 new     <- new control swallowed, exit 0

and the new cases go red:

FAIL: an allowlisted control does not cover a second control in the same file (exit 0, expected 1)
FAIL: the un-allowlisted control in a partly-allowlisted file is named as NEW

Detector restored byte-identical; suite green again.

On the Dagger question

The 2026-07-31 decision not to adopt dagger.io set a disconfirming test: attempt the convergence sweep, and if new hand-rolled controls reappear within 60 days, the model is the problem, not the backlog.

This is not that test firing. Its precondition was never met — no drafted caller had been adopted, so nothing could "reappear" after converging, and one new control cannot tell "the model is wrong" apart from "nobody has run the sweep yet". The narrower true statement, recorded in the README: drift accrued while convergence had not started. One new control out of thirty, from a PR fixing a real security gap, does not re-open Dagger on its own. It is recorded so the next one lands on a record instead of a blank page.

Evidence

Check Result
tests/security-policy.sh passed
tests/consumer-drift-cases.sh passed (3 new cases)
tests/example-fidelity-cases.sh passed
tests/lineage-guard-cases.sh passed
shellcheck -x clean
actionlint (examples) clean
zizmor 1.26.1 (examples, online) no findings
example fidelity 173 resolved, 0 MISSING, 0 UNVERIFIABLE, 7 repos
guard can fail shown red above

Not merging

Paired with the converging PR on hseshadr/aml-filter. Note that the aml-filter PR requires a branch-protection update (the gitleaks required context becomes Secret scan / gitleaks) — that is an owner action and is spelled out in that PR.

🤖 Generated with Claude Code

https://claude.ai/code/session_0186xrrT9mYfCZTVkk8HN12p

On 2026-08-02 the scheduled consumer-drift sweep went red for the first time:
`30 hand-rolled control(s) across 7 repo(s); 29 allowlisted; 1 new`. The day
before it read `29 ... 0 new`.

The new control is aml-filter/ci.yml/secret-scan, from aml-filter#89. That PR
closed a genuine hole -- aml-filter's gitleaks scan ran only in its weekly
security-audit.yml sweep and never on a pull request, so a secret could merge and
sit in public history for up to seven days. It closed that hole by inlining
gitleaks/gitleaks-action: the exact control this repo publishes as
secret-scan.yml, at the identical pinned action SHA. A security fix shipped by
hand-rolling the shared brick.

The brick was a drop-in -- secret-scan.yml takes no required inputs, and
examples/aml-filter/security-audit.yml already showed aml-filter calling it. So
part of the cause is habit. Two parts were ours:

1. examples/aml-filter/ci.yml carried no secret-scan job. The worked example this
   repo publishes for the exact file being edited offered nothing to copy. Fixed:
   the example now has the caller.

2. Nothing warned that adopting a reusable workflow RENAMES its check run to
   "<caller job> / <called job>". aml-filter's branch protection requires a
   context named literally "gitleaks"; adopting makes it "Secret scan / gitleaks"
   and blocks every merge until protection is updated. That cost is paid by the
   adopter and invisible to whoever published the brick -- one concrete reason
   inlining keeps winning. Fixed: it has its own README section, and the example
   carries the warning inline.

Also pinned the property the finding depended on. aml-filter/ci.yml/frontend-gate
has been allowlisted since 2026-07-26; had the allowlist key been read at file
granularity, that older entry would have swallowed the new secret-scan control
and the sweep would have reported a clean run on the day it mattered most. The
new fixture asserts an entry covers one control, not the file it lives in.

Proven by mutation, not by assertion: widening allowlist_index to compare
<repo>/<workflow> makes the detector report `2 allowlisted; 0 new` on a fixture
holding one known and one brand-new control -- it swallows the new one and exits
0 -- and the new cases go red. Restored byte-identical afterwards.

The detector was NOT weakened. It still exits 1 on any un-allowlisted control.
The one new allowlist entry is a pointer to an open converging PR on aml-filter,
carries the full reason above, and is marked for deletion when that PR lands; a
stale entry warns and never fails, so deleting it early is free. The allowlist
header also stops claiming "nothing here is new drift", which was true for
exactly seven days.

Claim touched: "new drift with no entry fails the build" -- unchanged and now
regression-tested. "Adoption is the fix, not an exemption" -- upheld: the
consumer is converging, and both ci-side reasons it did not converge the first
time are fixed here.

On Dagger: the 2026-07-31 decision set a disconfirming test -- attempt the
convergence sweep, and if new hand-rolled controls reappear within 60 days, the
model is the problem. This is NOT that test firing. Its precondition was never
met: no drafted caller had been adopted, so nothing could reappear after
converging. The narrower true statement, recorded in the README, is that drift
accrued while convergence had not started.

Evidence:
  security-policy.sh          passed
  consumer-drift-cases.sh     passed (3 new cases)
  example-fidelity-cases.sh   passed
  lineage-guard-cases.sh      passed
  shellcheck -x               clean
  actionlint (examples)       clean
  zizmor 1.26.1 (examples)    no findings
  example fidelity            173 resolved, 0 MISSING, 0 UNVERIFIABLE, 7 repos

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0186xrrT9mYfCZTVkk8HN12p
@cursor

cursor Bot commented Aug 2, 2026

Copy link
Copy Markdown

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

Converging aml-filter (aml-filter#93) turned up a cause the desk analysis missed,
because it only appears when someone tries to adopt.

This repo tags releases `ci-vX.Y.Z`. Third-party actions tag `vN`. So the trailing
comment on a correct first-party pin reads `# ci-v3.0.0`. aml-filter's own
supply-chain test lints pinned-`uses:` comments against `^v\d` and therefore
REJECTED the correct pin -- the converging PR went red on the consumer's own
guard, and the only way to satisfy that regex would have been to write a comment
naming a tag that does not exist.

The fix belongs in the consumer's guard and it tightens rather than loosens: key
the expected scheme off the ref, so neither convention is accepted for the other.
Documented here with that shape, so the next adopter reads it before hitting it.

That makes three separate taxes on doing the right thing -- a missing example, a
renamed check run, and a rejected pin comment -- and none of them is charged to
the person who inlines the action instead. That asymmetry is the finding.

Also swapped the allowlist entry's branch reference for aml-filter#93 now that
the converging PR exists.

Evidence:
  consumer-drift-cases.sh   passed
  security-policy.sh        passed
  README anchors            3 new cross-links resolve

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0186xrrT9mYfCZTVkk8HN12p
@hseshadr
hseshadr merged commit e86dee8 into main Aug 2, 2026
4 checks passed
@hseshadr
hseshadr deleted the fix/secret-scan-adoption-friction branch August 2, 2026 13:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant