feat: heuristic pre-screening before LLM consolidation

[salishforge]

Summary

Use text-overlap detection and polarity checking before invoking the LLM for consolidation. Reduces cost and latency for simple cases.

Design

• Before LLM summarization, run heuristic checks:
  • Text overlap >35% between memories → merge candidate (concat without LLM)
  • Polarity mismatch detection (enabled/disabled, true/false, before/after) → flag contradiction
  • Exact duplicate detection → skip
• Only invoke LLM for complex batches that need semantic understanding
• Track LLM cost savings from heuristic bypass

Inspiration

Inspired by hippo-memory (MIT) 35% overlap merge threshold and claude-code-toolkit (MIT) deterministic-vs-probabilistic philosophy.

Closes #53

[salishforge]

Implementation Decisions

1. Screening placement: Per-batch diversity check (Option B)

Options considered:

• A) Pre-batching: Compare individual hot-tier rows before grouping. Better consolidation quality but requires rethinking the batch loop — high complexity.
• B) Post-batching: Evaluate each batch's internal diversity after the existing grouping. Skip LLM for high-overlap batches. ← Selected

Rationale: Option B is minimally invasive — inserts between batch grouping and LLM call with no structural changes. Option A is better but should be evaluated via A/B testing once we have baseline metrics.

2. Overlap threshold: 70% word-level Jaccard

35% (from hippo-memory) is too aggressive for conversational memory where topic words naturally recur. 70% catches true near-duplicates without false-positiving on topically related but distinct memories.

3. LLM skip behavior: Raw concatenation (existing fallback)

When we skip LLM, we use the existing concat path. A lightweight merge (dedup sentences, sort chronologically) is a future enhancement — adds complexity without proven benefit yet.

4. Polarity contradictions: Flag as metadata

Contradictions detected heuristically are stored as _contradictions in warm-tier metadata. They don't block consolidation — the reflection phase handles conflict resolution. This avoids adding a new flow.

5. Comparison scope: Within-batch only

Per security review: pairwise comparisons limited to batch boundaries (max 50 rows). No cross-batch comparison. Prevents O(N²) DoS on large backlogs.

Future A/B testing opportunities

• Option A (pre-batch screening) vs Option B at scale
• Overlap threshold tuning (35% vs 50% vs 70%)
• Lightweight merge vs raw concat quality comparison
• Cross-batch polarity detection value