Repository navigation
epistemic-guard (MIT): Post-execution agent verification using 4 INT8 encoders on OpenVINO CPU/NPU (0 MiB dGPU VRAM, ~10 ms) #39819
AlienatedAliens
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Sharing an open-source MIT Python package I just shipped (
epistemic-guard) built directly on the OpenVINO Runtime (openvino+tokenizersonly — zero PyTorch/Transformers runtime dependency) so local LLM agents can keep 100% of discrete GPU VRAM free for the main generator while running post-execution verification on CPU or Intel NPU.Why we built it on OpenVINO INT8
Pre-execution tool firewalls (MCP permissions, human-in-the-loop approval) check what tools an agent tries to call, not the post-execution failure where a tool errors out (or is never called) and the LLM still tells the user "I refunded order #8821 and emailed your receipt."
Running a second 12B–14B quantized LLM-as-a-judge on GPU consumes 8.8–12.2 GiB of VRAM alongside the primary agent model. Instead,
epistemic-guardships four SHA-256-pinned INT8 cross-encoders (openvino_model.xml/.bin,226.8 MBtotal compressed download viaepistemic-guard download-models):action_claim_int8(10-class speech-act classifier distilled/fine-tuned onnli-MiniLM2-L6-H768,82.7 MB): splits assistant replies into clauses (10.9 msp50 on CPU) and checks concrete action claims against signedToolReceiptrecords in an RFC 6962-style Merkle log.hazard_int8(nli-MiniLM2-L6-H768,82.7 MB,3.2 msp50): scores multi-clause query hazard (LOW/ELEVATED/HIGH) to scale required independent RAG domains.web_evidence_int8(ms-marco-MiniLM-L-6-v2,23.1 MB,3.0 msp50): stance/relevance filtering for scraped passages.nli_int8(nli-deberta-v3-xsmall,91.1 MB,9.9 msCPU p50,99.2%FP32 parity): pairwise cross-source contradiction detection (NLIWrapper(device="CPU" | "NPU" | "AUTO")).It also includes
ProvenanceGate, a< 0.3 msEd25519 public-key-only SQLite memory write-gate so prompt-injected assistant narration cannot forgeUSER_EXPLICITorTOOL_VERIFIEDmemory envelopes (0/200false rejections on benign memory continuity).OpenVINO INT8 CPU vs. Local Quantized GPU LLM Judges (
N=250held-out supervision benchmark)Evaluated across
N=250strictly held-out supervision turns (including 100 live organic multi-sentence LLM completions with fenced code blocks, polite offers, and figurative idioms):N=250)epistemic-guard(action_claim_int8)0 MiB236/250 (94.4%)96.7%/7.8%27.4 ms(10.9 ms/clause)Mistral-Nemo-Instruct-2407-12B(Q5_K_M)8,844 MiB217/250 (86.8%)87.7%/14.1%77.9 msMicrosoft Phi-4-14B(Q6_K)12,214 MiB216/250 (86.4%)100.0%/26.6%75.1 msDeBERTa-v3-xsmallSNLI0 MiB132/250 (52.8%)24.6%/20.3%27.5 msQuickstart
All four encoders pad to static token shapes (
128/256tokens) so they also compile cleanly on Intel AI BoostNPU(device="NPU") without dynamic-shape recompilation stalls.Known Limits
real_web_v2,N=64), end-to-end burden accuracy is47/64 (73.4%)and dispute accuracy is56/64 (87.5%)— implicit cross-study disagreements without single-sentence NLI opposition still account for ~26% of open-web misses.Feedback, bug reports, and tips on optimal
ov.Coreexecution hints (LATENCYvsTHROUGHPUT) when interleaving 3–4 micro-batches of 128-token encoder calls per chat turn on Core Ultra CPU+NPU are very welcome!All reactions