Skip to content

v0.5.0 -- A local judge, agent-driven memory, and a benchmark with its limits stated

Choose a tag to compare

@ScottColeSW ScottColeSW released this 03 Oct 15:59
· 8 commits to main since this release

Agent-driven memory, a local judge that needs no chat model, and a labeled benchmark with its limits stated.

What changed since v0.4.0:

  • Agent-driven memory. Memory and learn() let an agent decide what a new claim is to what is held (new, reinforces, coexists, collides, exception, supersedes); the library enforces the boundary whoever judges. An outside claim never supersedes a held belief, and one the rules or the judge flag is held for a person. Works standalone with a local model, or as an MCP server (palimpsest-mcp, pip install "palimpsest[mcp]") for any MCP client.
  • Embeddings. Held claims are found by meaning, not only wording (palimpsest/embed.py, a local Ollama embedder is included). Rules remain the deterministic floor.
  • An NLI hybrid judge (pip install "palimpsest[nli]", CPU only, transformers below 5). A pretrained NLI model plus embeddings decides the relation; a small model may only add precision on what kind of conflict it is. A no-refiner mode uses no language model at all and turns every conflict into a plain collision, both claims kept live. Quantities are unit-aware, and a figure conflict is a raise-only signal.
  • bench/. Labeled judge and framing cases split into development and held-out, a runner, and a running leaderboard. Each held-out batch was frozen before any model saw it, scored once, and then moved to development; errors are published. The labels were written by one author and reviewed by one other person; three ambiguous cases were excluded and two relabeled after review, and the corrections are recorded in bench/GUIDE.md.

Measured (held-out, 71 judge cases, 95% ranges are wide): the hybrid with the stronger refiners scores 73 to 77% on the relation, against 34% for the fixed rules and 17 to 58% for small chat models asked directly. The hybrids miss 0 to 17% of true conflicts depending on the refiner. A newer small model (gemma3:4b) scored highest as a refiner but missed more conflicts and flagged none of the forgeries in the set, so it is not recommended. No claim is made that this transfers to other domains.

Used in practice: Project Aegis Vector's gate holds all ten forged payloads (0% adoption, 90% utility) and has a stated limit: it protects only registered facts. Void Marauders can use the NLI judge for trust memory; a paired test of memory on and off found no measurable benefit on one scenario.

700 tests pass, 3 skipped. store.py's Elasticsearch backend is still unverified against a live cluster. Nothing acts on a resolved collision beyond recording it, by design.