Add optional deterministic ranking signals (lexical overlap, hit reinforcement, decay)

[aurexav]

Context
Reranker scores can be noisy and are not always directly comparable across runs, and pure rerank_score sorting can overreact to small score deltas. ELF already has basic tie-break signals (importance and recency) and optional hit logging, but additional deterministic signals can improve stability.

Goal
Add small, explainable ranking signals that reduce noise and improve stability, without changing default behavior.

Scope

1. Lexical overlap bonus (deterministic)

• Compute a simple overlap signal between query tokens and:
  • stitched snippet tokens
  • structured fields (when Add structured memory fields with field-level embeddings #17 is enabled)
• Add an optional small bonus term when overlap is high.

2. Reinforcement from hits (deterministic)

• Use existing note hit signals (hit_count, last_hit_at) to add a small, saturating reinforcement term.
• Ensure reinforcement is bounded and does not overwhelm rerank.

3. Optional decay (separate from TTL)

• Add an optional decay score that gradually reduces ranking weight for stale notes even before TTL expiration.
• Keep TTL as a hard filter; decay is only a soft ranking term.

Explainability

• Record each term contribution in traces (for example: lexical_bonus, hit_boost, decay_penalty).

Non-goals

• No LLM calls.
• No changes to evidence binding rules.
• No changes to candidate retrieval.

Configuration

• Add ranking knobs with safe defaults (disabled by default, or defaults that preserve current behavior).

Testing and evaluation

• Unit tests for term computation and bounds.
• Harness: measure rank churn reduction across multiple runs of the same query.

Acceptance criteria

• With features enabled, rank churn decreases on representative queries without meaningful quality loss.
• Default configuration preserves current ranking behavior.

References

• OpenMemory: decay/reinforcement signals (conceptual inspiration)

[aurexav]

Design notes from implementation planning (2026-02-07)

Problem framing

• The reranker score can be noisy and not directly comparable across runs. Sorting mostly by raw rerank_score can overreact to small deltas and cause rank churn.
• Retrieval (especially retrieval_rank) is a strong signal but lives on a different scale than rerank_score, so naive mixing is fragile.

Suggested architecture direction (shared with #15 and #28)

• Move ranking to an explicit scoring pipeline with two phases:
  1. Feature computation: compute a stable set of per-candidate features (rerank_score, retrieval_rank, importance, age_days, hit_count, last_hit_age_days, lexical_overlap_ratio, etc.).
  2. Scoring policy: combine features into final_score via a versioned, configurable policy that produces a structured ScoreBreakdown (terms + per-term contribution + any clamping/bounds).
• Explainability/traces should record the ScoreBreakdown directly (term name, contribution, and any relevant intermediate values) so tuning is "change one knob, see the term contributions".

Deterministic signals (this issue)

• General principle: every added signal should be bounded and monotonic where possible, so it cannot overwhelm rerank.

1. Lexical overlap bonus

• Compute query tokens deterministically (ASCII-lowercased alnum tokens, de-duplicated; cap term count).
• Compute overlap_ratio = matched_query_terms / query_terms.
• Apply a bounded bonus only if overlap_ratio >= min_ratio:
  • lexical_bonus = weight * clamp((overlap_ratio - min_ratio) / (1 - min_ratio), 0..1)
• Record lexical_overlap_ratio and lexical_bonus in breakdown.
• Target fields: stitched snippet text; optionally key and structured fields when available.

2. Hit reinforcement (bounded)

• Use hit_count and optionally time since last_hit_at.
• Example bounded boost:
  • hit_saturation = hit_count / (hit_count + half_saturation)
  • recency = if last_hit_tau_days > 0: exp(-last_hit_age_days / last_hit_tau_days) else 1
  • hit_boost = weight * hit_saturation * recency
• Record hit_count, last_hit_age_days, hit_boost.

3. Soft staleness decay (separate from TTL)

• Keep TTL as a hard filter. Add an optional soft penalty for staleness prior to TTL expiration.
• Example bounded penalty:
  • staleness = 1 - exp(-age_days / tau_days)
  • decay_penalty = -weight * staleness
• Record age_days, decay_penalty.

Where to apply (chunk vs note)

• Decide explicitly whether these terms should affect chunk selection within a note (chunk-level) or only re-rank already-selected note representatives (note-level). Chunk-level gives better snippet sensitivity; note-level reduces behavior churn.

Experiment UX

• Consider allowing per-request policy override on an admin/debug endpoint so tuning can happen without restarting the service, while still writing the exact policy snapshot into traces for reproducibility.

Note on learned scorers

• A learned scorer (LTR) is a possible future upgrade to replace the hand-tuned policy, but it is not required for this issue. The feature + breakdown architecture keeps that option open.

[aurexav]

Implemented optional deterministic ranking signals and v2-only additive ranking explain.

What changed

• Added optional deterministic ranking terms (lexical overlap bonus, hit reinforcement, soft decay) behind config, disabled by default to preserve existing behavior.
• Switched ranking explain to schema search_ranking_explain/v2 with an additive decomposition (final_score == sum(terms[].value)).
• Search responses omit per-term inputs; traces/eval retain inputs for diagnostics and replay.
• Expanded trace candidate capture with candidate_snapshot to support stable replay.
• SQL table files are schema-only (no migration-style ALTER statements).
• Added a spec-level version registry at docs/spec/system_version_registry.md.

Verification

• cargo test
• cargo make fmt-check

Commit

• 954ae81