fix: the scanner recovers a secret an open-ended pattern's greedy quantifier swallowed (iss-188)

.abcd/work/DECISIONS.md

@@ -924,3 +924,5 @@ parallel-agent merge contention bites.
 - 2026-08-05 — iss-43, CORRECTION to the entry immediately above, from a merge-gate record-accuracy review on PR #196. That entry stands unedited; four of its claims are corrected here rather than in place, because it is merged history. (1) It opens "the three-claim corpus is closed as OVERTAKEN rather than fixed", and the corpus does not close as a unit: claims 1 and 2 — the Phase 0 status claim and the native-capability surface list — close as overtaken via `73428b6`, while claim 3's surviving half, the Layout line, is FIXED by that same PR. The entry knows this later in its own text and in the issue's `resolution:` field, so the opening verdict was a second over-claim of the same shape as the attribution it goes on to retract: a single verdict asserted over a corpus whose members had different fates. (2) "a SHALLOW clone whose truncation window opened well after 2026-07-17" over-claims. The evidence bounds the boundary to the interval after 2026-07-17 and no later than the oldest visible substantive README rewrite; "well after" asserts a distance nothing measured. Read it as "after 2026-07-17". (3) "writing the doc to describe a packaging mechanism that does not exist" is refuted by the entry's own closing paragraph. The clause scopes to the all-channel curated publish path, which genuinely does not exist and which no release has ever taken; the namespace deny itself exists and is tested. (4) The call edge is wrong in the closing paragraph: "reachable from the wired `abcd launch ship`; it is unwired, because `Ship` stops at `WouldPublish`" reads as one chain, implying the verb runs through `launch.Ship`. It does not. `launch.Ship` has NO production caller — the only calls are three in `dryrun_test.go` — while the wired verb exercises the deny through `launch.PrecheckPayload` and `launch.RenderPayload`, which resolve the bundle at `internal/core/launch/render.go:192`, reached from `internal/surface/cli/ship.go`. `release.yml` invokes neither. Both facts stand; the edge joining them did not. The lesson generalises past this entry: a summary sentence written before the detail paragraphs are finished tends to keep the verdict the round started with, and a call chain assembled from two true sentences about neighbouring functions is not evidence that either calls the other — `grep` for the callers.
 - 2026-08-05 — iss-118: DECISIONS.md and ACKNOWLEDGEMENTS.md gain `merge=union` in `.gitattributes`, the same remedy already in place for CHANGELOG.md. Both are the identical shape — append-only ledgers of anonymous, dated, order-independent entries that never need identity — so a concurrent-append conflict has nothing to actually disagree on; the union driver keeps both sides instead of stopping the merge. Prompted directly by the bug-hunt loop's `bugfix/iss-184-…` branch hitting exactly this conflict against `.abcd/work/DECISIONS.md` on PR #199, which iss-118 had already diagnosed (filed after an earlier merge hit the same conflict) but left unresolved pending this design call. Full atomicisation (per-decision records, a new id family, an armed uniqueness detector) was the other option iss-118 named; not pursued here as disproportionate to a minor, low-traffic hotspot. Impact recorded as `internal`, no CHANGELOG entry — matching the iss-80 precedent for a process/tooling change with no user-facing behaviour.
 - 2026-08-05 — iss-185 (bug-hunt loop, round 3): `ScanText`'s leading-`\b`-anchored secret patterns could never detect a second fixed-length token immediately abutting a first with no separator — the byte before the second token is itself a word character (the first token's own last byte), a word/word transition where `\b` never holds, so `FindAllStringIndex` silently returns only the first match. `Redact` then left the second token's whole body raw, and `history`'s stage-two fail-closed residual re-scan reported the redacted text clean anyway, so a concatenated pair of secrets reached disk with one still live. Landed after three rounds of pre-PR/pre-merge adversarial review, each catching a real problem in the pass before it: (1) the first pass's own-pattern-only probe missed a mixed pair of different fixed-length patterns, and double-counted a `google_api_key` match whose dash-terminated 35th char already satisfied `\b` on its own; (2) the fix for that, an unanchored `FindStringIndex` probe, cost O(matches × patterns × remaining line length) — measured at 14–49s on large single-line input; (3) anchoring the probe with `\A` fixed the *restart-at-every-offset* cost but not the cost of one attempt against two patterns (`net_lan_hostname`, `net_device_hostname`) that carry their own unbounded internal quantifier, still quadratic on many back-to-back fixed-length tokens. The landed shape: `adjacencyProbe` compiles an `\A`-anchored, `\b`-stripped (past any inline flag group) variant of each pattern; `scanAllPatterns` probes every pattern's variant, within a small fixed window (`maxAdjacencyProbeWindow`, 512 bytes) so one attempt can never cost more than a constant regardless of what follows, at the byte offset immediately after each already-found match, deduplicated against matches already found. A pattern with an open-ended quantifier can still greedily consume into a following token before this recovery ever runs — a separate, broader gap, captured as iss-188 rather than folded in here. Repro: `internal/adapter/scanner/adjacency_test.go`, `TestConcatenatedSecretsBothDetected` (the original bug), `TestConcatenatedDifferentFixedLengthSecretsBothDetected` and `TestGoogleAPIKeyDashJunctionNotDoubleCounted` (review round 1), `TestAdjacencyProbeStaysLinearOnLongLines` and `TestAdjacencyProbeWindowIsBounded` (review rounds 2 and 3).
+- 2026-08-05 — iss-188 (bug-hunt loop, follow-on to iss-185): a secret pattern whose quantifier is open-ended (`\bghp_[A-Za-z0-9]{36,}` and the seven other `{n,}` families) greedily consumes a following token's own leading bytes when those fall inside its trailing character class, so the true junction sits BEFORE the match's reported end and iss-185's forward adjacency probe — which only ever looks at that end — never runs where the second token actually starts. `"ghp_"+36×a+"ghp_"+36×b` reported one finding covering `ghp_aaa…aaaghp` and left `_bbb…bbb` raw through `Redact`, with `history`'s fail-closed residual re-scan reporting the output clean: the same live-secret-reaches-disk violation iss-185 closed for the fixed-length family. Fixed in `internal/adapter/scanner/scanner.go:465` (`stolenJunctions`): before a match's reported end is accepted as final, a bounded backward search walks candidate cuts and reports each one where the shortened prefix is STILL a whole match for its own pattern AND a token can begin, and `scanAllPatterns` probes those cuts exactly as it probes the forward end. Open-ended is detected from the compiled regexp — a match that cannot still match one byte shorter has a rigid length and is skipped after one test — not from hand-annotated metadata, so a pattern added later inherits the classification. Three things keep the cost per match rather than per match LENGTH, which the ledger entry named as the trap (an unbounded backward scan would trade the leak for a resource-exhaustion cliff on exactly the huge single-line input a scanner must handle): that one-test early-out, `maxAdjacencyBacktrack` (512, mirroring `maxAdjacencyProbeWindow`), and a single combined `junctionProbe` alternation over the whole pattern set that generates candidate cuts in one linear pass instead of one pass per pattern — measured 40s unbounded against 0.3s bounded on a 200KB line. The over-long match is deliberately left untrimmed: it only over-reports the first token's span, and `sealLine` already forces every overlap byte to `*`. Repro: `internal/adapter/scanner/adjacency_test.go`, `TestConcatenatedOpenEndedSecretsBothDetected` (the ledger entry's literal reproducer) and `TestOpenEndedSecretSwallowingDifferentFamilyBothDetected` (a whole `AKIA` key swallowed by an alnum class run), both watched failing on pre-fix code for the claimed reason; cost guard `TestJunctionBacktrackIsBounded`.
+- 2026-08-05 — iss-188 follow-up (two independent adversarial reviews of the fix above, both landing on the same defect): the bounded backward search in `internal/adapter/scanner/scanner.go` (`stolenJunctions`) could step OVER the junction it was looking for. `junctionProbe` is unanchored, so one of its hits can SPAN a real junction — begin before it and end after it — without beginning at it; the loop nevertheless resumed at that hit's END (`off += loc[1]`) even when the hit's own offset had just been REJECTED by `wholeMatch`, skipping every byte in between, the true junction among them. Nothing revisits a skipped range, so the search returned fewer cuts than it should and iss-188's own failure mode reopened through a narrower trigger: `"ghp_"+32×a+"AIza"+4×b+"ghp_"+36×c` reported one finding and left the second token's tail raw, and `"ghp_"+10×a+"AIza"+22×z+"sk-proj-"+40×i` — where the skipped-over decoy is itself a syntactically valid secret — left 48 raw bytes of an OpenAI project key through `Redact` with the fail-closed residual re-scan reporting the output clean. Fixed by always resuming at `cut + 1`, dropping the jump-to-hit-end fast path entirely: a rejected candidate can hide a real junction one byte later, and an accepted one is not worth a special case. This stays bounded for the reason the whole feature exists: the loop is capped at `maxAdjacencyBacktrack` (512) iterations per match regardless of how long the match is, and each iteration's search is capped at `maxAdjacencyBacktrack + maxAdjacencyProbeWindow` bytes by RE2's linear guarantee — so the per-match cost is a constant factor independent of the REST of the line, and the whole scan stays linear in line length rather than quadratic. Measured: doubling and quadrupling an adversarial 247KB line doubled and quadrupled the time (2.3s / 4.6s / 9.2s), which is the property that matters. Repro: `internal/adapter/scanner/adjacency_test.go`, `TestStolenJunctionSearchDoesNotSkipPastRejectedCandidate` and `TestStolenJunctionSearchSkipsPastValidDecoySecret`, both watched failing on pre-fix code for the claimed reason; cost guard `TestJunctionBacktrackIsBounded`, extended with `dense_rejected_candidates_in_backtrack_window` — every match's window packed with candidates that all fail validation, the worst case for advancing one byte at a time. Two further gaps the same reviews found were CAPTURED, not folded in: iss-190 (a recovered match longer than `maxAdjacencyProbeWindow` is truncated, and the misaligned artificial end breaks the chain that would recover a THIRD abutting token — the pre-existing iss-185 window trade-off reached through a new path, whose real fix is an adaptive window that risks reintroducing the unbounded per-match cost) and iss-191 (`junctionProbe`'s `(?s).` compile fallback would validate once per byte instead of once per candidate, unreachable with the bundled pattern set). Two comment claims were also corrected as inaccurate: cost is bounded independently of the rest of the line but IS proportional to the match's own length, since `wholeMatch` re-runs the probe over the prefix; and a window-exceeding recovery can miss a token ENTIRELY, not merely truncate it, when the pattern's required structural markers (`jwt_shaped`'s two `.` separators) both fall outside the window.

.abcd/work/issues/open/iss-190-scanner-adjacency-recovery-is-capped-by-maxadjacencyprobewin.md

@@ -0,0 +1,12 @@
+---
+schema_version: 1
+id: "iss-190"
+slug: "scanner-adjacency-recovery-is-capped-by-maxadjacencyprobewin"
+severity: "minor"
+category: "bug"
+source: "agent-finding"
+found_during: "bug-hunt loop round 3, pre-PR review of iss-188's fix"
+found_at: "internal/adapter/scanner/scanner.go"
+---
+
+scanner adjacency recovery is capped by maxAdjacencyProbeWindow, so a recovered token longer than the window is truncated — and the truncation can break the recovery CHAIN, hiding a third abutting token entirely. probeAt (scanAllPatterns, internal/adapter/scanner/scanner.go) runs its anchored probe against line[at:at+maxAdjacencyProbeWindow] (512 bytes), so a recovered open-ended match longer than that is recorded with an artificial end at the window edge. Two manifestations. (a) Known and accepted since iss-185: the recovered token's far tail past the window stays raw in Redact's output; the token's prefix is still masked, so this is a partial-redaction trade, not a total miss. (b) NEW, found by two independent adversarial reviews of iss-188's fix: because the artificial end does not align with the token's real end, the next iteration's probeAt(m.end) and stolenJunctions both start from a byte offset that is inside the token rather than at its junction, so the chain that would otherwise unwind a run of three or more abutting tokens breaks — the third token becomes entirely invisible, not merely truncated. Repro: line := "ghp_" + strings.Repeat("a",36) + "ghp_" + strings.Repeat("m",600) + "ghp_" + strings.Repeat("z",36) yields exactly 2 token:github_pat findings where an unbounded search would find 3; the third token's 36 raw bytes survive Redact and the fail-closed residual re-scan reports the output clean. A sharper variant of (a) affects jwt_shaped specifically: its regex requires two literal '.' separators, so a JWT-shaped token whose header and payload together exceed the window has NEITHER separator inside the probe's slice and the anchored probe finds no match at all — the token is missed entirely rather than truncated, e.g. a fixed-length match immediately followed by a JWT whose first two segments total more than ~502 bytes. This is NOT a defect introduced by iss-188's backward search; it is the same pre-existing maxAdjacencyProbeWindow trade-off from iss-185, now reachable through a second code path because the backward search feeds more candidates into probeAt. Fixing it properly needs a different mechanism (a growing or adaptive window, or re-probing chained from a recovered match's real end) that must not reintroduce the unbounded per-match cost the window exists to prevent — the exact risk iss-188's own ledger entry warned against. Deliberately not folded into iss-188's fix pass to keep that diff scoped to the confirmed junction-search defect.

.abcd/work/issues/open/iss-191-junctionprobe-s-compile-fallback-turns-stolenjunctions-from.md

@@ -0,0 +1,12 @@
+---
+schema_version: 1
+id: "iss-191"
+slug: "junctionprobe-s-compile-fallback-turns-stolenjunctions-from"
+severity: "minor"
+category: "bug"
+source: "agent-finding"
+found_during: "bug-hunt loop round 3, pre-PR review of iss-188's fix"
+found_at: "internal/adapter/scanner/scanner.go"
+---
+
+junctionProbe's compile fallback turns stolenJunctions from a bounded candidate walk into a per-byte one, a latent cost cliff. junctionProbe (internal/adapter/scanner/scanner.go) builds one combined alternation over every pattern's boundary-free body; if that alternation fails to compile it falls back to regexp.MustCompile(`(?s).`), which matches at EVERY byte offset. The fallback was chosen so the candidate generator can over-produce but never under-produce — correct for detection, but it changes stolenJunctions' cost class: instead of one wholeMatch validation per real candidate junction in the backtrack window, wholeMatch runs once per byte offset in the window, and each of those validations is itself proportional to the match's own length (it re-runs the anchored probe over line[m.start:cut]). Cost per match goes from O(window) validations to O(window x match length) work. A reviewer measured roughly a 500x slowdown substituting the fallback directly — 4.21s against 8.25ms on one 200KB match. Not reachable with the current bundled pattern set: every bundled body alternates cleanly, and triggering the compile failure needs a pathological custom .abcd/config/pii.json override whose added pattern makes the joined alternation invalid (the per-pattern regexes are each compiled and validated on merge, so this is hard but not provably impossible to reach). Latent rather than urgent. Options if it is ever worth closing: cap the fallback's candidate count the way the window caps the search, drop patterns from the alternation one at a time until it compiles rather than abandoning the whole set, or mark the scanner unavailable (fail-closed) when the combined probe cannot be built — the last is consistent with how the package already treats a config fault.

.abcd/work/issues/open/iss-188-scanner-adjacency-greedy-steal-open-ended.md → .abcd/work/issues/resolved/iss-188-scanner-adjacency-greedy-steal-open-ended.md

@@ -7,6 +7,8 @@ category: "bug"
 source: "agent-finding"
 found_during: "bug-hunt loop round 3, pre-PR security review of iss-185's fix"
 found_at: "internal/adapter/scanner/scanner.go:scanAllPatterns"
+resolution: "scanAllPatterns now backtracks a bounded window from an open-ended match's reported end (stolenJunctions) and probes each candidate cut where the shortened prefix is still a whole match for its own pattern, so a second token whose leading bytes the greedy quantifier swallowed is detected and redacted instead of surviving raw"
+impact: fix
 ---
 
 scanner adjacency recovery misses greedy over-consumption in open-ended-quantifier secret patterns. iss-185's fix (scanAllPatterns, scanner.go) recovers a second token that immediately abuts a first with no separator ONLY when the first pattern is fixed-length (its match always ends exactly where the second token begins, so probing every pattern's boundary-free variant at that one byte offset is sufficient). A pattern whose quantifier is open-ended (e.g. `\bghp_[A-Za-z0-9]{36,}`, `\bsk-ant-[A-Za-z0-9_-]{40,}`, `\bsk_live_[A-Za-z0-9]{20,}`) instead greedily consumes as many class-matching bytes as are available, including a following token's own leading bytes when those happen to fall inside the same character class — shifting the true junction earlier than the reported match end, before scanAllPatterns' adjacency probe ever runs. Concrete repro: `"ghp_" + strings.Repeat("a",36) + "ghp_" + strings.Repeat("b",36)` — the first `ghp_` pattern's `{36,}` class run consumes the 36 a's AND the second token's leading "ghp" (all alnum), stopping only at the second token's `_` (not in `[A-Za-z0-9]`); ScanText reports exactly ONE finding covering `ghp_aaaa...aaaghp`, and the second token's `_bbbb...bbb` tail survives Redact completely raw with the fail-closed residual re-scan reporting clean — the same "can never contain a live secret" violation iss-185 was filed for, just for the open-ended-quantifier pattern family instead of the fixed-length one. Distinct from iss-185 (closed): that fix's mechanism (probe every pattern at a match's reported end) is necessary but not sufficient — this needs a bounded backward search (shrink the greedy match's end toward its start, checking at each candidate cut whether the shortened match is still valid for its own pattern AND some pattern's boundary-free variant matches there) without becoming O(line length) per match, since these patterns' matches can legitimately be very long (base64 blobs, minified files) and an unbounded backward scan across every byte would be a new resource-exhaustion risk on exactly the kind of large single-line input a secret scanner must handle. Surfaced by an adversarial pre-PR security review of iss-185's fix (bug-hunt loop round 3), not by the round's own hunt; not fixed in that PR to keep its diff scoped to the exact bug iss-185 named and to avoid shipping an unbounded-cost search under review pressure. Affected pattern families: github_pat (ghp_/ghs_/gho_/ghu_/ghr_), anthropic, openai_project, openai_svcacct, stripe_live, stripe_test, slack, jwt_shaped — every bundled pattern with an open (`{n,}`) rather than fixed (`{n}`) quantifier.