feat(memory): opt-in hybrid semantic search

[codeaholicguy]

Summary

• add opt-in local hybrid semantic memory search behind memory.semantic: true
• preserve strict-to-broad FTS and fuse lexical/semantic ranks deterministically with RRF
• add pinned MiniLM download/status/re-embedding commands, explain metadata, and lexical-only offline fallback
• store versioned float32 embeddings in an additive migration and invalidate them on content changes

Eval gates

Expanded-100 model gate against current strict-to-broad FTS:

• MiniLM paraphrase nDCG@5: 0.7590 vs 0.6576 (+15.41%)
• MiniLM paraphrase recall@5: 0.76 vs 0.68 (+11.76%)
• overall MRR@5: 0.9275 vs 0.8637 (improved)
• identifier-exact hit@1: 0.96 vs 0.96, with zero per-query regressions
• repeated rankings: identical
• BGE did not pass: +8.75% nDCG@5 and +5.88% recall@5

Built CLI on expanded-100 improved the published 0.57.1 baseline from hit@1 81% / hit@3 91% / hit@5 96% / zero 1% to 88% / 97% / 98% / zero 0%. Judged irrelevant top-three increased from 2.9% to 4.7%, which is a recall/noise trade-off to monitor.

Warm in-process search at 1,000 rows (100 searches after five warmups): median 21.85 ms, p95 29.78 ms, max 33.51 ms.

Packaging

• rejected @huggingface/transformers@3.8.1: 705 MB installed
• selected exact onnxruntime-web@1.22.0 + @huggingface/tokenizers@0.1.3: 100 MB installed, no native/platform-specific package
• pinned q8 Xenova/all-MiniLM-L6-v2 model: 23 MB lazy download, excluded from install footprint
• measured model load 426 ms, first inference 58 ms, warm inference about 12-14 ms

Pre-merge simplification pass

Audited the diff against the repo's simplify-implementation discipline (every addition needs a current caller; guards/retries need a demonstrated trigger; delete-before-add) before requesting review:

• Collapsed loadLocalEmbedder()'s unused modelsRoot/download options and the unreachable offline-inspect-only branch (its only real caller always passed download: true, no modelsRoot); removed the same dead modelsRoot pass-through from storeKnowledgeSemantic/updateKnowledgeSemantic/reembedKnowledge.
• Removed LocalEmbedder.dispose() — implemented but never invoked by any caller.
• Replaced SemanticModelUnavailableError with a plain Error+cause — the subclass was never distinguished via instanceof anywhere; every catch site already falls back to a generic message.
• Added a CLI test for the semantic-enabled memory update path, closing a gap lint had flagged (an unused mock import) and matching the existing store/search coverage.

Kept: getSemanticStatus/downloadSemanticModel's modelsRoot option (genuinely exercised by a test to isolate from the real model cache directory), the embedder DI seam on every semantic handler (used throughout the test suite), and all RRF fusion / degradation / migration behavior — none of it changed.

Reverted on user review: an initial pass here also removed GlobalDevKitConfig.memory.semantic as dead code (nothing read it). That was a misclassification — the field was unwired, not unwanted. It's reinstated and now wired: memory.semantic resolves with project > global > default(false) precedence at the single site every semantic-aware command/server path already calls (ConfigManager#getMemorySemanticEnabled), so a developer can turn semantic search on once in ~/.ai-devkit/.ai-devkit.json instead of repeating it in every project's config. An explicit per-project value (either direction) still overrides the global default. No new config surface — same boolean, one more file, one resolution function. Covered by three new tests (project overrides global, project-unset inherits global, both-unset defaults to false).

None of the eval-gate numbers above changed as a result of this pass.

Lifecycle docs

Replaced the single ad hoc docs/ai/2026-09-01-feature-memory-semantic.md note with the five phase documents under docs/ai/<phase>/2026-09-01-feature-memory-semantic.md, each following the dev-requirements/dev-design/dev-planning/dev-implementation/dev-testing skill structure and grounded in the shipped, simplified state:

• Requirements — goals/non-goals, acceptance criteria traceable to existing tests/gates, and the frozen MiniLM-vs-BGE model-selection gate table.
• Design — architecture, data model, API surface, and design decisions/trade-offs.
• Planning — full task breakdown by milestone, including this simplification pass as its own milestone.
• Implementation — file-by-file notes and what the simplification pass changed and why.
• Testing — gate evidence plus fresh build/test/lint/e2e output captured after the simplification commits.

npx ai-devkit@latest lint --feature memory-semantic passes against the new structure.

Validation

Re-run after the simplification and doc commits:

• npm run build — all 6 projects
• npm test — 1,091 passed (was 1,087; +1 for the added memory update semantic test, +3 for the global/project precedence tests)
• npm run lint — no errors (3 pre-existing, unrelated warnings)
• npx vitest run --config e2e/vitest.config.ts — 41 passed
• npx ai-devkit@latest lint --feature memory-semantic — base structure and all five phase docs passed

Operational notes

Semantic search defaults off. Missing model/network, corrupt or stale vectors, and corpora above the 5,000-row brute-force ceiling degrade to lexical search. Rollback is memory.semantic: false; the additive columns remain inert and row deletion requires no vector-index cleanup.