Skip to content

v0.5.6 — Embedder stats persistence + drift indicator refinement

Choose a tag to compare

@zzallirog zzallirog released this 13 May 03:01

The KNN predictor and the meta layer share an embedding space defined by per-feature median/MAD normalization. Until this release those statistics were a daemon-process artifact — a restart re-fit them on the post-boot ring window, drifting the normalization away from vectors already in the chroma index. v0.5.6 promotes them to a persistent file (data/embedder-stats.json) loaded on boot, so the archive and live queries stay in the same vector space across restarts.

New surface

  • Embedder.save_stats() / Embedder.load_stats() persist per-feature median/MAD to data/embedder-stats.json. Daemon reloads on boot and freezes refits while the file is present — newly-embedded frames share the same vector space as vectors already in the chroma index. Remove the file to re-enable adaptation.
  • scripts/reindex_chroma_from_store.py — one-shot helper that rebuilds the chroma index from store.db.frames: reconstructs TelemetryFrame from raw_json, embeds, writes to chroma, then runs backfill_labels against throttle_events. Idempotent — skips ids already present, safe to re-run.
  • drift detectorchroma_no_growth indicator now gates on base_count > 0, distinguishing a deliberately-disabled feature (count stays at 0) from a genuinely-stuck collector (count was growing, then stopped).

Dependency

chromadb venv installation aligned with the pyproject.toml constraint (>=0.5,<1.0) on Python 3.14 — matches the spec already in the source tree.

Tests

  • test_save_load_stats_roundtrip — byte-for-byte vector equivalence across Embedder instances after save/load. The invariant that warm-start reindex depends on: vectors in chroma and queries from the daemon must agree given the same input frame.
  • Missing/corrupt stats file handled gracefully (returns False, embedder stays unfitted).
  • test_evaluate_chroma_disabled_no_false_positive — drift detector remains silent when chroma is intentionally not in use.

700+ tests green.