fix(memory): preserve FTS5 relevance direction

[ScriptedAlchemy]

Summary:

• preserve FTS5 negated-BM25 ordering when converting ranks into bounded relevance scores
• add a real-world regression where exact stale-process evidence must outrank unrelated TraceDecay V2 facts
• verify rare-term coverage, once-per-search retrieval/access counters, analytics funnel totals, and untracked context-memory enrichment

Tests:

• cargo test --lib memory::retrieval::tests::fts5_rank_score_preserves_bm25_direction -- --exact
• cargo test --test mcp_suite mcp_handler_test::fact_search_ranks_exact_operational_evidence_and_tracks_once -- --exact
• focused memory retrieval/counter and MCP context/analytics tests
• cargo fmt --all -- --check
• cargo check --all-targets
• cargo clippy --all-targets -- -D warnings

[changeset-bot]

⚠️ No Changeset found

Latest commit: d4ad019

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c3b7780ea7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Normalize BM25 ranks before mixing with other scores

For small or common-term memory stores, SQLite FTS5 often returns BM25 ranks around -1e-6 even for the best full-text hit, and this conversion maps that to an FTS score of roughly zero. In those searches the 0.40 FTS component in combined_score effectively disappears, so a fact surfaced only by list/entity overlap can tie or outrank an actual content FTS match based on holographic/recency/usage signals. Consider normalizing against the returned rank range, or otherwise calibrating the raw BM25 magnitude, before combining it with the other 0–1 scores.

Useful? React with 👍 / 👎.

[ScriptedAlchemy]

Fixed in b4aa14a: ranks are now normalized against the strongest returned BM25 relevance before cross-signal scoring, so common-term values around -1e-6 retain their FTS weight while order is preserved. Added tiny-rank and lone-match regressions; all ranking evals pass.

[ScriptedAlchemy]

Cross-session diagnosis (Fable) of the Linux/macOS failures: memory_suite::eval_memory_ranking_morphology, well-behaved case install_query_keeps_installing_fact_on_top.

Reproduced on this branch. The direction fix is correct — do not revert it. The eval was previously satisfied partly because of the inverted direction: with BM25 honest, the off-topic one-term exact match on a shorter document ("Use npm to install packages", fts=0.554) now edges the on-topic two-term morphological match ("Use pnpm for installing dependencies…", fts=0.509) by 0.2217 vs 0.2053 — rank 2 instead of 1, gap −1. The backup case still passes (fts=0 there; holographic decides), confirming this is a term-coverage weighting question, not a direction error.

Two sanctioned paths from the repo's own eval conventions (the message_search eval used the same pattern):

1. Encode it as an expected_current_failure/known-gap case with the aspirational expectation documented, keeping the suite green while the gap is visible; or
2. Fix the underlying quality gap: fold distinct-matched-term coverage into the fts component (two matched terms should not lose to one on length normalization alone), then keep the eval as-is — likely the better end state since the eval's intent (morphology beats off-topic exact) is a real ranking property users feel.

Either way the fixture data and both rankings above should save you the repro.

🤖 Diagnosis by Claude Code

[ScriptedAlchemy]

Followed through on option 2 from my diagnosis (pushed to this branch by the parallel Claude session, user-authorized): the morphology eval now passes on merit instead of needing a known-gap encoding.

Two mechanisms, matching semantics end to end:

• build_fts_query emits prefix-form terms ("install"*) for tokens ≥4 chars, so simple morphology actually reaches the FTS index ("install" finds "installing"); short tokens stay exact to avoid over-matching.
• The fts component is weighted by distinct-term coverage (0.5 + 0.5 × matched/total, same exact-or-prefix rule), so a document matching every query term can no longer lose to a shorter one-term document on BM25 length normalization — the exact failure shape in install_query_keeps_installing_fact_on_top. Coverage rides in the why trace.

Verified: all 5 eval_memory_ranking_* evals pass (49/49 memory_suite), your fts5_rank_score_preserves_bm25_direction and fact_search_ranks_exact_operational_evidence_and_tracks_once regressions pass unchanged, plus new unit tests for the prefix query shape and coverage math. fmt + workspace clippy -D warnings clean.

🤖 Addressed by Claude Code