Skip to content

v0.9.4 — cold-tier recall decompression fix

Choose a tag to compare

@spranab spranab released this 14 Jul 03:19

v0.9.4 — hotfix

Fixes #62: cold-tier recall scores were silently wrong — garbage similarities always, and a chance NaN that poisoned the response-level confidence.

Root cause (two compounding defects)

  1. archive() stores cold-tier embeddings zstd-compressed in the same column hot rows use for raw f32 — and the recall read path never decompressed (decompress_embedding had exactly one call site: hydrate). Every recall scoring path that touched a cold record reinterpreted compressed bytes as floats. The cold tier has been silently unscoreable in recall since compression was introduced; #60''s panic fix is what made it visible as a wrong number instead of a crash.
  2. consolidate::cosine_similarity — the function recall actually scores through — still carried the pre-#60 NaN-blind guard (== 0.0).

Fix

  • decompress_if_cold() (zstd magic check) after decrypt in both durable embedding readers, mirroring hydrate(). Previously-affected records self-heal on the next read — no per-record hydrate ritual needed.
  • The #60 guard ported into cosine_similarity (!(norm > 0.0)): degenerate vectors degrade to finite 0.0 similarity, never NaN.
  • Defense: the index rebuild scan also decompresses cold-format blobs if tier drift ever routes one there.

Credit: reported by @blue-mtn-dog with line-level root-cause analysis, verified exact on every claim — their second such report.

Published to PyPI (pip install -U yantrikdb) and crates.io.