feat(smart-search): index semantic facts, procedural skills, crystals, insights, facets

[rohitg00]

Problem

mem::smart-search indexes compressed observations + lessons. It does not index:

• KV.semantic (semantic facts produced by consolidation pipeline)
• KV.procedural (procedural skills extracted from repeated patterns)
• KV.crystals (densified high-confidence memory clusters)
• KV.insights (cluster-derived insights from reflect)
• KV.facets (entity facet decompositions)

These are the densest, highest-confidence memory in the system. They get written by mem::consolidate-pipeline + mem::reflect + mem::crystallize, then sit dark until a query specifically targets them via memory_crystal_recall / memory_lesson_recall.

This is the search-side parallel to #732 (which addresses the inject-time gap in mem::context). #732 fixes the SessionStart / PreCompact auto-inject path. This issue fixes the on-demand smart-search recall path.

Proposed shape

Extend mem::smart-search to optionally include high-order tiers in the candidate pool. Score each tier with the same RRF fusion of BM25 + vector channels.

iii composition:

• mem::smart-search reads existing 5 KV scopes in addition to observations
• Per-tier scoring: BM25 on fact / name+steps / narrative / content text + vector cosine on tier's embedding (need to ensure each tier has embeddings; some don't today)
• Per-tier weight in fusion: high-confidence tiers (crystals, facts confidence > 0.85) get a fusion boost
• Output format: existing compact result list adds a tier field per result (observation | lesson | semantic | procedural | crystal | insight | facet)
• expandIds honors all tiers (currently observation-only); response shape unified as { tier, id, sessionId?, content, score }

Embedding coverage gap

Audit which high-order tiers have embeddings:

• Lessons: yes (already searchable via memory_lesson_recall)
• Semantic facts: NO. Need to add embedding: Float32Array on creation
• Procedural skills: NO
• Crystals: NO
• Insights: NO
• Facets: NO

For tiers without embeddings, do one-time backfill via mem::backfill-embeddings::high-order on first enable. Subsequent writes embed inline.

Filtering + scoping

• Project filter: each tier may or may not have project field. Audit + enforce consistent scoping.
• Agent filter: agentId field where applicable. Cross-agent leak prevention.
• Confidence floor: skip tier entries below 0.3 confidence by default; configurable via AGENTMEMORY_SMART_SEARCH_CONFIDENCE_FLOOR.

Budget impact

Indexing 5 additional scopes per smart-search call increases retrieval cost. Mitigations:

• Tiers are small (~10x fewer entries than observations on typical workloads)
• BM25 + vector are in-process, microsecond-scale per entry
• Cap per-tier candidate pool: max 50 entries per tier before fusion
• Total cap on returned results unchanged (existing top-k applies)

Measured cost increase target: <10% on a 1000-observation workload.

Edge cases

• Tier with no entries — skip tier entirely, no empty-block in response
• expandIds with cross-tier IDs — handler dispatches by ID prefix (crystal_xxx → KV.crystals, sem_xxx → KV.semantic, etc.)
• Backfill on enable — first call with AGENTMEMORY_HIGH_ORDER_SEARCH=true triggers backfill. Block search until backfill done (could take minutes on large stores). Better: backfill in background, smart-search uses tiers as-they-become-embeddable.
• Embedding model switch — if embeddings backfilled with model A, then user switches to model B, vector channel returns zero across the tier. Need version stamp on each embedding, detect mismatch, lazy re-embed.
• Deleted entries — Lesson.deleted: true already exists. Same convention for other tiers. Skip in search.
• Decay interaction — search score should factor strength × confidence for tier entries. Decayed-near-zero entries de-prioritized naturally.

Acceptance

• Smart-search reads all 6 tiers (existing observation pool + 5 new)
• Per-tier BM25 + vector indexes
• Embedding backfill for tiers lacking them
• Embedding version stamp + lazy re-embed on model switch
• Project + agent scoping enforced
• Confidence floor configurable
• expandIds dispatches on ID prefix across tiers
• Response shape unified with tier field
• AGENTMEMORY_HIGH_ORDER_SEARCH=false opts out
• Tests per tier (positive recall + scoping + filter)
• Bench harness shows whether high-order tier inclusion improves multi-session and temporal-reasoning recall categories

Why it matters

Closes the dark-memory gap on the recall surface. The most condensed memory in the system becomes queryable. Pairs with #732 (inject side) to make the consolidation pipeline's output actually reach the agent.

[Tanmay-008]

i thing we exclude facets from the high-order search pool, and only index the remaining 4 tiers (Semantic facts, Procedural skills, Crystals, Insights). Here is why:

Not Semantic Content: Facets are structural metadata tags (e.g., priority:urgent, category:database), not natural language text. BM25 and vector search are designed to operate on natural language — running them on 2-3 word key:value pairs produces meaningless scores.

High False Positive Rate: Including them in BM25 or vector search will introduce significant noise. A full-text query like "urgent database migration" would match every item tagged priority:urgent or category:database, regardless of actual relevance, polluting the search results and pushing down genuinely relevant observations.

Dedicated Tool Already Exists: We already have memory_facet_query (mem::facet-query) specifically designed for exact-match structural filtering on these tags with matchAll / matchAny semantics. Adding facets to full-text search would create a redundant, lower-quality path for the same data.

No Confidence or Strength Scores: Facets lack the confidence or strength fields that the other high-order tiers use for RRF fusion ranking and the configurable AGENTMEMORY_SMART_SEARCH_CONFIDENCE_FLOOR filter. There's no meaningful way to rank or threshold them alongside the other tiers.

Embedding Backfill Waste: Generating vector embeddings for strings like priority:urgent wastes embedding API tokens and produces low-quality vectors. The resulting cosine similarities would be near-random noise, degrading the vector channel for the entire search.

Duplicate Signal: The entities that facets tag (actions, memories, observations) are already in the search pool. A facet just adds a redundant, lower-quality echo of the same item — the observation "Fixed urgent database migration bug" is already searchable; its facet tag category:database adds nothing new.