Document round 4 (production hardening) in evals + README summary - #19
Merged
Merged
Conversation
Adds the fourth review round to evals/README.md: adversarial re-audit that found and fixed #15 (mask_line_all() skips masking on partial secret overlap) and #16 (builtin_scan() reads full file before the 5MB size cap check), the 10->24 edge-case test expansion, the CHANGELOG.md/v0.1.0 release, and a second real-execution pass against a 6-fake-secret-type fixture -- including an honest accounting of a $1.9477 cost mistake (scan run without --model haiku). Updates README.md's Real-world validation summary to the new totals: 9/9 issues closed, $2.7627 total real LLM spend, 24-test suite, and a link to the v0.1.0 release.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
evals/README.md: adversarial re-audit that found/fixed mask_line_all() leaks nearly all of a secret when two matches partially overlap without either containing the other #15 (mask_line_all()skips masking on partial secret overlap) and builtin_scan() reads a file's full contents into memory before checking the 5MB size cap #16 (builtin_scan()reads the full file before the 5MB size-cap check), the 10→24 edge-case test expansion,CHANGELOG.md/v0.1.0release, and a real-execution pass against a 6-fake-secret-type fixture.scancall was run without--model haiku, landing on the backend's default (non-haiku) model and costing $1.9477 vs. $0.0304 for the correctly-flaggeddescribecall — recorded rather than folded quietly into the total.README.md's "Real-world validation" summary to the new totals: 9/9 issues closed (Secret suffix leaks into masked_preview/context_line for generic_high_entropy_assignment rule #2–README masking formula is stale after #5's fix — still documents the buggy clamp(3, 6) #8, mask_line_all() leaks nearly all of a secret when two matches partially overlap without either containing the other #15, builtin_scan() reads a file's full contents into memory before checking the 5MB size cap #16), $2.7627 total real LLM spend, 24-test regression suite, and a link to thev0.1.0release.Docs-only change, no code touched.
Test plan
gh issue list/gh pr list/git loggrep -rn '#\[test\]' src/ | wc -lv0.1.0tag/release exists viagh release list