Skip to content

v0.3.7 — Telemetry integrity fixes

Choose a tag to compare

View all tags
@pluto2060 pluto2060 released this 27 Apr 13:31
v0.3.7
1439017

What's fixed

Three data quality bugs in the telemetry collection pipeline:

1. G1 block name mismatch (bm25-memory.py)
Injection items used "block": "g1" but last-retrieval-meta.json stored metadata under "g1_decisions". Result: all G1 records had query_type=UNKNOWN, retrieval_method=UNKNOWN, candidates_returned=None.

2. CM retrieval_method lost (utility-rate.py)
chat-memory.py writes retrieval_method to a separate file (last-cm-injection.json). The merge code discarded this before the metadata was saved → CM records had retrieval_method=UNKNOWN despite using HYBRID.

3. "UNKNOWN" truthy bypass (utility-rate.py)
block_meta.get("query_type") or _classify_query(prompt) — "UNKNOWN" is a truthy string so the fallback never fired. Fixed with explicit != "UNKNOWN" guard.

Also adds BM25 default for g2_prefetch/g2_grep blocks (codebase-memory-mcp is BM25-only, no meta block written).

Impact

After backfilling 37 historical records:

  • retrieval_method UNKNOWN: 60% → 0%
  • query_type UNKNOWN: 62% → 0%
  • Causal r (BM25 score → utility_rate): +0.639 → +0.667
  • SEMANTIC vs KEYWORD utility gap: 24.8pp (47.8% vs 23.0%)
Assets 2
Loading