Repository navigation
v0.1.0 -- First measured release
First measured release. Aegis Vector plants forged policy documents in a local RAG pipeline, measures whether small local models believe them, and measures what each defense buys and what it costs. Everything runs on one workstation: Ollama, llama.cpp, ChromaDB, and a FastAPI dashboard.
What's in it
- Live dashboard (
server.py,/): eight streamed stages, each with a live timer. It shows the 3D vector manifold (query-centered, with the true top-k retrieval sphere), first-token refusal probabilities, per-token surprisal, the Palimpsest memory gate switched off, on hold, or on flag, gate-off vs gate-on answers side by side, and a per-run cost table with editable prices. - Experiment battery (
experiments.py): six forged-document payloads against five small local models, reporting retrieval rank, hijack rate, and payload perplexity, plus first-token refusal probing under four framings. - Defenses battery (
defenses.py): ten defenses measured on adoption, utility, and cost per query and per document, with every answer labeled by a JSON-schema-constrained judge (qwen2.5:7b) cross-checked against a regex. - Memory gate (
memory_gate.py): Palimpsest v0.2.0 in front of retrieval. New documents are filed under a registered fact and quarantined when they collide with the verified record or change a figure as an exception. - Black Hat vs White Hat (
/episodes.html): the defenses battery as an animated comic-strip arms race. Every win rate and quoted answer comes from the run. - Memory guard (
resource_guard.py): free RAM and VRAM preflight before each model load, a short keep-alive, and bounded retry.
Headline results (five models, six payloads, four questions each)
| Defense | Adoption of the forged limit | Utility | Cost per query vs none |
|---|---|---|---|
| none | 46% | 85% | |
| Perplexity filter | 46% | 85% | +0% |
| Provenance labels | 18% | 90% | +21% |
| Provenance, forged doc via trusted channel | 43% | 90% | +19% |
| Spotlighting | 29% | 95% | +51% |
| Layered (provenance + spotlighting) | 5% | 90% | +70% |
| Palimpsest memory gate, hold | 0% | 85% | -6% |
- A naturally worded forgery never ranked first, yet every model adopted it every time it reached the top 3 (15 of 15).
- The effective forgeries read as more natural than the real policy, so a perplexity filter would reject legitimate documents first.
- The memory gate quarantined all six forgeries. It's the only defense measured that reached 0% adoption without refusing answers: the grounding check also reached 0%, but blocked half the answers to get there.
Limits, stated plainly: four questions per payload, so rates move in 25-point steps. The gate only protects facts in its registry, and forgeries of unregistered facts or rules with no number are untested. The first-token refusal metric can't measure phi3:mini. Full details are in the README.