feat(embedding): embedding works by default on store and retrieve (ticket 6) + hashing fallback rejected (ticket 7 candidate) - #20
Merged
Conversation
…trieve Ticket records the operator-observed incident (embedding config only in a repo .env the daemon never reads; a single 429 marked the index failed and search silently fell back to lexical for days), the agreed fix (per-operation bounded drain without threshold batching, rate-limit = pause not fail, local hashing degrade chain with reason, drop the AUTO_SYNC gate, deployment-layer config persistence), and the industry practice it follows (async post-write embedding in mcp-memory-ts; degrade-on-persistent-failure lifecycle in openclaw).
…cket 6)
Implements the core of ticket 6 so embeddings are present on the normal write
and search paths without a fragile config toggle:
- sync engine: syncEmbeddingTarget accepts { maxBatches } and stops after N
batches, leaving the index mid-flight (records still queued) instead of
blocking on a full backfill or marking it complete prematurely.
- service: #drainEmbeddings now fires on every remember/search via
#signalMaintenance (no writeThreshold/accessThreshold batching), runs one
bounded batch per target, and drops the NMG_EMBED_AUTO_SYNC gate — a
configured provider (+key) implies auto-sync. Provider failures start a 30s
cooldown so a down/rate-limited provider cannot hang every query; a full
successful drain clears it.
- search: searchMemoryContext serves hybrid from a *partial* index (the
vector LEFT JOIN keeps lexical results and lifts only indexed records), so a
previously failed/429'd index is no longer a permanent lexical dead end; and
accepts a degradedReason so a cooldown search reports degraded:true instead
of silently re-attempting the provider.
Tests: bounded-drain (maxBatches), provider-presence-implies-sync hybrid, and
degraded-unreachable-provider cases; sync/chaos/service suites pass.
Ticket 6 checklist updated (local-hashing blend remains as the open slice).
Verified: npm run agent:verify all blocking checks green (check,
test:product 778, build, verify:static incl. verify:packages + complexity).
…al candidate (ticket 7) The local-hashing retrieval fallback was measured (real store) and rejected: 256-d nmg-hashing-v1 blended retrieval is byte-identical to pure lexical (self-recall 45/154 both arms; scores ≈ 0), matching the published dimensionality bottleneck; the NUMEN-style fix (16K-32K dims, beats BM25) is unaffordable for a local SQLite store. Recorded as a rejected decision with an explicit scope note: only the semantic- retrieval role is rejected. Word-level uses (SimHash/feature hashing for near-duplicate candidate recall and spelling-tolerant matching) are a separate candidate, opened as ticket 7 with research precedent (claude-memory-system issue 53: 64-bit SimHash, Hamming <= 3, pre-filter before an LLM dedup judge). Ticket 6 marked done-evaluated; ticket 7 lists the recall-gap measurement gates adoption. docs: 109 files, 0 errors.
Design doc for the word-level SimHash candidate (ticket 7): closes the gap
where supersedeCandidates' word-level exact matching (instr substring +
token normalization + word-set Jaccard) recalls zero for spelling / word-form
variants ("embedding" vs "embeddings", "colour" vs "color"). Proposes a
64-bit SimHash fingerprint (one INTEGER column, Hamming <= 3 recall channel
inside supersedeCandidates, judge still decides) with a measurement-first
experiment gate: generate word-form variants over the real store and measure
whether the fingerprint channel recalls what Jaccard misses, before any index
is built. Boundaries kept: no semantic judgment, no search-ranking role
(rejected decision), no external dependency.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
变更描述
What（改了什么）
实现工单 6（embedding 默认在存储与检索路径生效）核心 + 关闭 hashing 语义检索兜底 + 立项 SimHash 词法层候选（工单 7）。
Why（为什么改 / 解决什么问题）
实测故障：embedding 配置只放仓库
.env（daemon 不读），重启即丢 key；一次 free-tier 429 把索引标failed、last_succeeded_at = null→ 搜索静默退化纯词法数天。修复方向经业界调研确认（mcp-memory-ts async post-write embedding；openclaw degrade-on-persistent-failure）。Changes（关键改动点）
src/core/embedding-sync.ts：sync 引擎支持{ maxBatches }—— 限量补缺，跑到批上限即返回（索引保持 mid-flight、记录仍排队），不一次性全量 backfill 撞限流。src/cli/service.ts：#drainEmbeddings改为每次 remember/search（经#signalMaintenance）触发、每 target 限量 1 批；去掉NMG_EMBED_AUTO_SYNC门（有 provider 即自动 sync）；provider 失败设 30s 冷却（down 的 provider 不卡每次查询），整链成功清除冷却。src/integration/search.ts：部分索引即可 hybrid（vector LEFT JOIN 保 lexical 结果、只对有向量记录加分），failed/429 索引不再是词法死路；新增degradedReason通道，冷却期搜索报degraded: true+ reason。src/core/types.ts：retrieval.reason放宽为 string（冷却原因可自由传）。完成检查项
本地质量检查
npm run agent:verify通过（check / test:product / build / verify:static 含 verify:packages + complexity gate / docs:check 109 files 0 errors）npm run check/lint/format:check通过RCP（Repository Control Plane）
repo-development频道两条（工单 6 实施1788430068893_000011）已 resolvenpm run agent:verify（core-memory / daemon-cli / ci-and-tests 全 blocking 绿）CI 完成确认
nmg-rcp forge-status --pr <PR号>的All checks passed为 SUCCESS