JIT Context OS: Sub-3ms Epistemic Memory Cascade for Autonomous Agents (EXP-009: Local 9B beats Gemini Flash)

[Wojciech Wiesner (wojciechwiesner)]

Problem: The Haystack Tax & Infinite Context Degeneracy

In multi-agent systems and autonomous coding loops, as tasks expand across multiple files and turns, the message history accumulates terminal outputs, file contents, and assistant reasoning. This triggers:

1. Lost-in-the-Middle Degradation: Frontier models miss subtle architectural constraints buried in 30k+ token histories.
2. Exponential Token Costs & Rate Limits: Every subsequent turn re-sends the entire history, triggering 429 rate limit errors on high-frequency workflows.
3. Epistemic Self-Poisoning: The model treats its own past speculative comments or hallucinated intermediate variables as ground truth in later turns.

---

Proposed Solution: JIT Context OS Architecture

Instead of appending raw conversational history into prompts, we propose an Epistemic Context Cascade governed by deterministic invariants (I1–I10):

• L0 Hot-Path (<3ms SQLite WAL): A local in-memory SQLite WAL tracks atomic events, verified tool executions (write_file, execute_command exit codes), and the active working set.
• Epistemic Authority Weighting:
  • User instructions: Weight 1.0 (Invariant I1)
  • Deterministic tool execution proofs (exit: 0, test pass): Weight 1.0
  • Assistant conversational reasoning / speculation: Weight 0.0 (preventing self-poisoning)
• Lean Dynamic Working-Set Capsule (<1.8k tokens):
  Each turn passes ONLY:
  1. System Prompt + tool definitions (100% prefix cache hit rate >88%)
  2. The immediate prior tool result
  3. A dynamically compiled <ONA_CONTEXT> capsule (1.2k – 1.8k tokens) containing the active user goal, current module interfaces, and verified runtime assertions.

---

Empirical Proof: EXP-009 Benchmark ($599 Mac Mini vs Cloud API)

We evaluated this architecture in a blind, multi-file bug-fixing task (4 distinct bugs across 3 modules, verified by independent pytest assertions):

Architecture & Model	Hardware / Platform	Turns to Complete	Wall-Clock Time	Pytest Verification	API / Compute Cost
🥇 Local Qwen 3.8 9B + JIT	Mac Mini M2 Pro (Local Ollama 32k)	4 turns	~2 min (126s)	4/4 PASSED (exit 0)	$0.00 (100% local)
🥈 Gemini 3.8 Flash + JIT	Google Cloud API	9 turns	17.7s	4/4 PASSED (exit 0)	Paid Cloud API
🥉 Gemini 3.8 Flash (Standard Chat History)	Google Cloud API	10 turns (exhausted)	34.5s	❌ 0/4 FAILED (lost in tests/)	Paid Cloud API

Key Takeaway: A 9B open-weight model running locally on consumer silicon solved the multi-file task in fewer turns than Gemini 3.8 Flash because its context was kept surgically lean (<1.8k tokens). Small models aren't incapable of complex reasoning—they have been suffocated by chat history.

---

Reference Implementation & Permanent DOI

The reference implementation, 10 formal invariants, and test suites are open source:

• Repository: github.com/wojciechwiesner/jit-context
• CERN Zenodo Permanent Record: 10.5281/zenodo.22649542
• Live Case Study: theones.io/case-studies/

Would the AutoGen community and maintainers be interested in exploring epistemic context governors as a pluggable memory/context layer for multi-agent workflows?