1.085M real commands through the full gate: 0 real recall misses #6
BGMLAI
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Today the full 6-stage gate (the real
ActionPipeline, not the flatcheck_actionpath) was run against 1,085,159 unique real agent commands streamed from five public trajectory datasets (Nemotron, SWE-Zero, SWE-Hero, Kwai SWE-smith-mini, nebius), plus a deterministic sweep of all 43 known danger classes.Result: 0 real recall misses.
pip install gate-cat && python scripts/recall_danger_axis.py.rm -f test*.csv *.pyc && rm -rf __pycache__in a repo workspace). The delete-analyzer allowed them asproven-disposable, and the identical shape with a real target blocks 5/5 (rm -rf *on real files,/etc/*,*.db,~/.ssh/*). So it's a catalog over-match, not a gate blind spot.Full method + artifact: RECALL.md, results/million_recall_2026-07-08.json. Every number is in FACTS.md (F1a/F1b).
Honest caveat kept front and center: this measures detection of known-dangerous shapes. The gate is certain only about what it blocks; an unmatched action is unchecked, not safe. That's exactly why axis 2 exists — to hunt for shapes we didn't enumerate — and why the bypass suite prints its own known gaps. If you find a shape that slips through, that's the veto story worth more than a star.
Beta Was this translation helpful? Give feedback.
All reactions