Repository navigation
epistemic-guard: Catch unexecuted tool claims in Python on CPU (~11 ms, 0 VRAM, MIT) #3171
Replies: 2 comments 3 replies
|
@AlienatedAliens RollSolo, your point about the 11ms latency and zero VRAM usage for the verification layer is spot on. The common pitfall here is assuming that cross-encoders like MiniLM or SNLI models can reliably distinguish between 'tool not called' and 'tool called but failed' without explicit execution traces. These models often hallucinate intent because they rely on semantic similarity rather than cryptographic proof of execution, leading to high false positives when the LLM confidently lies about a skipped step. Without the receipt mechanism you implemented, the agent's narrative remains unanchored from reality. The key is that semantic verification fails when the model hasn't actually executed the action, regardless of how fluent the lie sounds. |
|
The failure mode you are solving is the one we see constantly: the narration is fluent, confident, and false, and nothing in the transcript lets a reader separate what ran from what was merely said. Your guard covers the in-process half: catch the unsupported claim while the run is still yours to retry. The half we work on at Zambo (I run ops there) is what happens after the process ends, when the reader is a client, a teammate, or another AI and the runtime is gone. For that reader, the attempt itself needs a record they can open. I ran a small job a few minutes ago so the contrast is clickable. Step one executed: a live price fetch, saved as a public receipt (https://zambo.dev/run/8869c25f-a500-4502-ac54-6b208f052a33). Step two was an attempted memory save carrying the content "the old staging database was deleted." It was rejected before execution on a missing required argument, and the rejection is its own public receipt: executed: false, the exact arguments received, the reason (https://zambo.dev/run/9e5d5513-bcd0-4f0f-b6cd-8bebf9cd7f77). A summary claiming the deletion happened would collide with that record in one click. A summary that tells the truth matches it. Two details from running this in production that may be useful for epistemic-guard:
Underneath is AER-1 (IETF draft-zambo-aer1-26), an open format with a public conformance corpus, so a receipt written by one tool can be checked by someone else's verifier. If epistemic-guard ever wants its verdicts to travel outside the process that produced them, that is the layer we built it for. |
Uh oh!
There was an error while loading. Please reload this page.
When running smaller or quantized tool-calling models locally, a common bug happens after tool execution: a
@toolcall raises an exception (or the model skips calling it), and the final answer still tells the user "I saved your backup and deleted the old staging DB."I pulled the local verification layer out of my agent testbed into a standalone MIT package (
pip install epistemic-guard):Using it with
smolagents@tool@recorder.toolusesfunctools.wraps, so stacking@tooloutside@recorder.toolpreserves signature and docstring inspection while logging Ed25519-signed receipts (includingok=Falsewhen a tool raises):Benchmark (
N=250held-out turns, incl. 100 live LLM completions)Off-the-shelf SNLI/MNLI cross-encoders score
"Zero tools ran"vs."I saved checklist_backup.md"asNEUTRALon ~75% of phantom actions (52.8%accuracy). Fine-tuning an 83 MB MiniLM cross-encoder (action_claim_int8) on 10 speech-act classes gave better accuracy than local 12B-14B LLM judges without competing with the agent model for GPU VRAM:95% Wilson CI)epistemic-guard(action_claim_int8)0 MiBVRAM (82.7 MBRAM)236/250 (94.4% [90.8%, 96.6%])96.7%/7.8%27.4 msMistral-Nemo-Instruct-12B(Q5_K_MJudge)8,844 MiBVRAM217/250 (86.8% [82.0%, 90.4%])87.7%/14.1%77.9 msMicrosoft Phi-4-14B(Q6_KJudge)12,214 MiBVRAM216/250 (86.4% [81.6%, 90.1%])100.0%/26.6%75.1 msDeBERTa-v3-xsmallNLI0 MiBVRAM (91.1 MBRAM)132/250 (52.8% [46.6%, 58.9%])24.6%/20.3%27.5 msThe package also includes
ProvenanceGate+SQLiteProvenanceStore(< 0.3 mspublic-key-only Ed25519 memory gate so untrusted tool outputs or model prose cannot forgeUSER_EXPLICIT/TOOL_VERIFIEDrecords) andcorroborate_sources(SimHash + 11-word verbatim span deduplication for syndicated web RAG snippets).Known limits
Phi-4-14Bcaught 4 subtle false negatives thataction_claim_int8missed (100.0%vs96.7%recall), at the cost of12.2 GiBVRAM and a26.6%false-positive rate on benign offers and idioms.real_web_v2,N=64),corroborate_sourcesdrops to47/64 (73.4%)burden accuracy and56/64 (87.5%)dispute accuracy. Implicit cross-study disagreements without lexical or single-sentence NLI contradiction account for ~26% of misses.If you run into agent outputs that break the clause splitter or receipt matcher, feel free to post them here so I can add them to the test suite.
All reactions