v0.5.9 - Flat Memory Footprint + INT8 Models + Crash Observability

Roampal Core v0.5.9

Release Date: 2026-08-27
Type: Memory footprint fix + INT8 embedder/reranker + crash observability
Triggered by: A resource-exhaustion incident (2026-08-24): with system commit charge at 91%, Windows denied an allocation to the thin MCP stdio process, killing it with an unhandled MemoryError and no persisted trace — consistent with the v0.5.8 server's ~2.4 GB ONNX working set.

Summary

The v0.5.8 server holds two ONNX models (embedder + cross-encoder) and grows to ~2,355 MB working set under normal load. Two mechanisms: FP16 model files up-convert to FP32 at load (+703 MB embedder, +362 MB cross-encoder), and ONNX Runtime's CPU memory arena never returns per-shape scratch buffers to the OS (+892 MB ratchet, almost all cross-encoder).

v0.5.9 disables the arena and memory-pattern cache on both models and ships measured INT8 exports. Measured: ~2,355 MB → ~484 MB steady state (~5x, flat), and both search paths get faster — embed 20.7 → 8.1 ms, rerank 1,388 → 1,190 ms (end-to-end gate run). Production process footprint: under 1,000 MB, flat. Crash observability and degraded-state responses make the next memory incident visible instead of silent.

---

What ships

Item 1 — ONNX memory arena + pattern cache off

Both model loaders construct sessions with enable_cpu_mem_arena=False and enable_mem_pattern=False — per-shape scratch buffers return to the OS instead of ratcheting toward a 2.5–3 GB plateau. Verified by an automated RSS soak gate (real INT8 embedder, 120 varied-shape ops, first-vs-last-fifth drift must stay under 5%). The thread cap was measured 45% slower for zero savings and dropped; ROAMPAL_ORT_THREADS remains available.

Item 2 — Embedder: mpnet FP16 → mpnet INT8

Same model, INT8 export: file 529.2 → 265.8 MB, resident +703 → +285 MB, embed 20.7 → 11.2 ms (isolated). Output stays 768-dim — no schema change. Existing vectors stay semantically valid against INT8 queries (measured cosine agreement 0.990).

e5-base upgrade held back: the planned model-family change failed its accuracy gate hard — 100% baseline → 57.5%. Root cause is the pre-existing v0.5.3 fact-dedup guard, calibrated for mpnet's embedding geometry: under e5-base it silently rejects nearly every new fact write as a false duplicate. Deferred to v0.6.0 behind a threshold recalibration. The attempt surfaced and fixed a real bug: the migration detector compared only the ONNX filename (mpnet-INT8 and e5-INT8 share one relative path), so migration state is now keyed on a composite model::onnx_file string.

Item 3b — Cross-encoder INT8 + shared session

The reranker loads its INT8 export: resident 362 → 156 MB, 1.28× faster than FP16 in isolation (Spearman 0.9879 vs FP16 rankings, top-4 set identical). One shared cross-encoder session across all profiles instead of one per profile — verified under concurrent load (exactly one session, ~29% latency penalty for two simultaneous reranks, not 2x).

Item 2a — Background re-embed machinery

New roampal reembed CLI plus automatic background migration: per-collection resume, smallest collection first, single-runner lock (cross-process PID file + in-process guard, keyed by path so profiles never lock each other out), compare-and-swap against mid-batch edits, and a --dry-run guaranteed side-effect-free. Migration is invisible to the MCP client; search reports per-collection progress while returning normal results from finished collections.

Item 5 — Crash observability

PID-scoped rotating file logs under ~/.roampal/logs/, an RSS heartbeat every 60 seconds, and MemoryError/ExceptionGroup detection that logs a final record before exit. The failure mode that started this release no longer exits silently.

Item 7 — Degraded states

Embedder-down raises an explicit EmbedderUnavailable end to end: HTTP 200 with a degraded body on /api/search, a plain-language notice on the MCP search_memory path, and an explicit injected marker on the hook path. Never a bare [], never a silent empty injection. New /api/status aggregates embedder, reranker, and migration health.

Item 6 — Documentation corrections

ARCHITECTURE.md and README memory figures corrected to match measurement (~484 MB steady state, ~500 MB RAM requirement). Every figure in the release notes is measured, not estimated — method and raw numbers included.

---

Upgrading

• First start after upgrade downloads ~379 MB (INT8 embedder + reranker artifacts) — the one visible wait.
• A one-time background re-embed recomputes stored vectors with the new artifact: per-collection, resumable, never blocking search or the MCP client. It runs once; subsequent starts do nothing. ROAMPAL_REEMBED_DISABLE=1 opts out; roampal reembed runs it on demand.
• No schema changes; 768-dim vectors unchanged.
• Rollback: ROAMPAL_EMBED_MODEL + ROAMPAL_EMBED_ONNX_FILE (FP16 pair) and ROAMPAL_CE_ONNX_FILE=onnx/model_O4.onnx restore the v0.5.8 models; a reverse migration fires automatically on next start.

Verification

• 780 tests passing, 3 skipped (up from 735), including the new RSS soak gate, migration/concurrency, degraded-state, and MemoryError tests.
• Live evidence: server RSS 321.8 MB flat (heartbeat), INT8 artifacts on both models, migration complete, real queries returning correct results.
• Full measured evidence, method, and raw numbers: dev/docs/releases/v0.5.9/RELEASE_NOTES.md