KS68: recall fixes — 80% micro-benchmark (IE-3, KU-3, TR-3, ME-4, PT-3)

[Liorrr]

Summary

• TR-3 fixed: Patent deadline now ranks KS67: Schema-driven fact extraction + Greptile AI review #1 with consolidation — apply_temporal_boost() (+0.015) fires for temporal:future/temporal:past labeled memories on temporal queries
• ME-4 fixed: PostgreSQL/ClickHouse in top-3 — co_occurrence_boost() (+0.05) fires for multi-entity database/language co-mentions
• IE-3 fixed: UBC ranks KS68: recall fixes — 80% micro-benchmark (IE-3, KU-3, TR-3, ME-4, PT-3) #2 with consolidation — Pipe B topic alignment gate prevents cross-topic false rescues
• KU-3 improved: enforce_subject_diversity tracks (subject, primary_topic) tuples, not just subject; prevents identity memories from crowding out role/career results
• PT-3 (labels): topic:language split into topic:language:natural + topic:language:programming with backward-compat OR fallback; temporal:future/temporal:past extended with date pattern detection
• New labels: topic:tools:editor (IDE/editor memories) + query boost +0.06; action:learning (language learning); memtype:intro (identity intro memories); memtype:preference_update ("I switched to X")
• New boosts: label_topic_boost(), apply_temporal_boost(), co_occurrence_boost(), career_intro_adjustment(), preference_update_boost()
• Consolidation quality: is_near_dup_child() helper (G1), Tier 2 label enrichment test (G3), apply_tier2_labels() shared helper (G4)
• Safety: deduplicate_parent_child() in final results — prevents parent+child from both occupying result slots
• Supersession: flat demotion at full strength (0.15) for superseded parent entries

Benchmark

Mode	Score	vs KS67
Embedding-only	13/20 (65%)	—
With consolidation	16/20 (80%)	0 regressions, TR-3 + ME-4 newly fixed

Workspace: 380 tests passing, 0 failures, clippy clean (all 12 crates).

Remaining failures (KS68.2 / KS69)

Query	Root cause	Next step
KU-1	Shopify raw sim advantage too large for demotion	KS69: needs Supersedes edge at parent level or query-time exclusion
KU-3	Neovim embedding gap ~0.485	KS68.3: child_rescue_only=false + parent-child dedup
PT-3	M18 embedding diluted by travel context	KS68.3: child_rescue_only=false + JLPT child extraction
IE-1	Identity memory Hebbian inflation	KS69: career query → intro demotion needs higher factor

Test plan

• cargo test --workspace — 380/0
• cargo clippy --workspace -- -D warnings — clean (all 12 crates)
• CI — green (3 consecutive runs)
• echo_micro_benchmark consolidation mode — 16/20, 0 regressions vs KS67
• QA validated on multiple commits, confirmed no regression from KS67 baseline

🤖 Generated with Claude Code

[greptile-apps]

Greptile Summary

This PR lands KS68: five new query-time scoring boosts (apply_temporal_boost, co_occurrence_boost, label_topic_boost, preference_update_boost, career_intro_adjustment), a deduplicate_parent_child safety pass, (subject, primary_topic) subject-diversity tuples, an expanded label taxonomy (topic:language:natural/programming split, topic:tools:editor, memtype:intro, memtype:preference_update, action:learning), and a refactored apply_tier2_labels shared helper that fixes both previously-flagged bugs (phantom-label index + missing surviving-set intersection). The benchmark moves from 65 % → 80 % with zero regressions against KS67.

• label_topic_boost shared break (P1, echo.rs:2982-2995): The break inside the inner for ql in query_labels loop is shared between the topic:tools:* and action:learning branches. When both labels are present in a query (e.g. "What editor am I learning?"), only the first matching boost fires; the second is silently skipped. Fix: use per-category boolean guards.
• apply_tier2_labels duplicate posting-list entries (P2, consolidation.rs:974-988): Labels already present from Tier 1 that also appear in the Tier 2 LabelSet are re-inserted into the label_index posting list. query_labels dedup() masks the effect at retrieval, but posting lists grow unnecessarily. Fix: snapshot pre-existing labels before mutation and gate indexing on !pre_existing.contains(label).
• The contains_future_date concern from the prior review is resolved: the call site is now gated behind co-occurring deadline/due context keywords.
• The phantom-label index concern from the prior review is resolved: the surviving-set intersection correctly restricts which labels reach the index."

Confidence Score: 4/5

Safe to merge after a targeted fix to label_topic_boost; the shared break causes silent ranking degradation but no data loss or crash.

Strong benchmark improvement (65%→80%, 0 regressions), 380 tests pass, clippy clean. One P1 logic fix required (label_topic_boost break guard) and one P2 index-cleanliness fix. Prior review concerns are addressed. Score reflects one targeted fix remaining before merge.

crates/shrimpk-memory/src/echo.rs (label_topic_boost break), crates/shrimpk-memory/src/consolidation.rs (apply_tier2_labels duplicate indexing)

Important Files Changed

Filename	Overview
crates/shrimpk-memory/src/echo.rs	Five new ranking boosts, parent-child dedup, (subject, topic) subject diversity — label_topic_boost has a shared break that silently suppresses the second boost category when both match
crates/shrimpk-memory/src/consolidation.rs	apply_tier2_labels refactored to shared helper with post-truncation surviving-set check (prior phantom-label bug fixed); pre-existing Tier 1 labels that reappear in Tier 2 still create duplicate posting-list entries
crates/shrimpk-memory/src/labels.rs	topic:language split into :natural/:programming with backward-compat OR fallback; contains_future_date now correctly gated behind deadline/due co-occurrence keywords; new label prototypes and Tier 1 rules added cleanly

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A["echo() query"] --> B["Brute-force cosine scoring"]
    B --> C["+ co_occurrence_boost\n+0.05 if 2+ DB or lang keywords"]
    C --> D["+ parent supersession demotion"]
    D --> E["collect EchoResult vec"]
    E --> F["apply_temporal_boost\n+0.015 if temporal:* label on temporal query"]
    F --> G["label_topic_boost\n+0.06 topic:tools:* / +0.025 action:learning"]
    G --> H["preference_update_boost\nx1.05 for memtype:preference_update on current-state query"]
    H --> I["career_intro_adjustment\n-0.10 memtype:intro / +0.03 topic:career on career query"]
    I --> J["sort by final_score"]
    J --> K["enforce_subject_diversity\ncap=3 per (subject, primary_topic) tuple"]
    K --> L{"reranker enabled?"}
    L -->|yes| M["cross-encoder rerank"]
    L -->|no| N["community summary fallback"]
    M --> N
    N --> O["deduplicate_parent_child\nremove lower-scoring of parent+child pair"]
    O --> P["superseded-parent exclusion"]
    P --> Q["return top-K results"]

Loading

_{Reviews (3): Last reviewed commit: "fix: gate contains_future_date behind de..." | Re-trigger Greptile}