From 5886ebebcb70d0ce9404beee4133bae123191c02 Mon Sep 17 00:00:00 2001
From: wefio <48851810+wefio@users.noreply.github.com>
Date: Thu, 3 Sep 2026 14:24:02 +0800
Subject: [PATCH 1/4] =?UTF-8?q?docs(todo):=20add=20ticket=206=20=E2=80=94?=
 =?UTF-8?q?=20embedding=20works=20by=20default=20on=20store=20and=20retrie?=
 =?UTF-8?q?ve?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Ticket records the operator-observed incident (embedding config only in a repo
.env the daemon never reads; a single 429 marked the index failed and search
silently fell back to lexical for days), the agreed fix (per-operation bounded
drain without threshold batching, rate-limit = pause not fail, local hashing
degrade chain with reason, drop the AUTO_SYNC gate, deployment-layer config
persistence), and the industry practice it follows (async post-write embedding
in mcp-memory-ts; degrade-on-persistent-failure lifecycle in openclaw).
---
 docs/design/temporary-todo.md | 57 ++++++++++++++++++++++++++++++++++-
 1 file changed, 56 insertions(+), 1 deletion(-)

diff --git a/docs/design/temporary-todo.md b/docs/design/temporary-todo.md
index 95f0ac6..53ac16d 100644
--- a/docs/design/temporary-todo.md
+++ b/docs/design/temporary-todo.md
@@ -2,7 +2,7 @@
 
 **Purpose:** unresolved work only.
 **Authority:** working queue, not design specification or implementation history.
-**Updated:** 2026-09-01
+**Updated:** 2026-09-03
 
 Durable behavior belongs in the owning design or operating document; current
 implementation evidence belongs in
@@ -152,6 +152,61 @@ one shared daemon, each benchmark worker closes without stopping that daemon,
 and the bridge subprocess can be removed without changing official inputs,
 outputs, scoring, or product RPC semantics.
 
+## 6. Make embedding work by default on store and retrieve
+
+User requirement: embedding must be present on the normal write and search
+paths; local hashing is a fallback, not a substitute. Observed failure: the
+embedding config lived only in a repo `.env` that the daemon never reads, so a
+daemon restart silently dropped the API key and every subsequent search fell
+back to lexical while the operator believed embedding was on
+(`embedding_index_state.status = "failed"` after a single free-tier 429,
+`last_succeeded_at = null` → search always lexical). The configured provider
+did build 96 vectors on 2026-09-01 and then never ran again.
+
+- [ ] Trigger a bounded embedding drain on every remember/search (no
+  writeThreshold/accessThreshold batching): each operation tops up one small
+  batch of missing vectors in the background, so low-activity stores still
+  converge and a 429 just queues the rest for the next operation.
+- [ ] Treat provider rate-limit / transient failures as *pause, not fail*: a
+  429/5xx must not set the whole index to `failed` (which permanently disables
+  hybrid until a full rebuild); record `last_failed_at` + reason and resume on
+  the next drain. Only persistent (non-rate-limit) failure degrades.
+- [ ] Retrieve with a degrade chain: external index ready → hybrid; external
+  unavailable/degraded → local `nmg-hashing-v1` vectors (already written for
+  every record) with a `degraded: true` + reason marker — strictly better than
+  pure lexical, still zero external dependency.
+- [ ] Drop the `NMG_EMBED_AUTO_SYNC` gate: presence of a configured provider
+  (+key) implies auto-sync. Keep the env as an explicit *disable* switch.
+- [ ] Persist embedding configuration at the deployment layer (User-level env /
+  documented daemon launch) so a restart keeps provider + key; document this in
+  the owning guide/ADR rather than inventing a new config-file mechanism.
+
+**Available mechanism:** write and access paths already `signalMaintenance`
+(`src/cli/service.ts` #remember/#search); `#drainEmbeddings` already runs
+records→leaves→nodes incrementally from the SQLite missing-vector queue
+(`embeddingDocuments` with a limit, batch 64); every record already carries
+local hashing vectors (`memory_embeddings` model `nmg-hashing-v1`, 327 rows);
+`searchMemoryContext` already has a lexical fallback seam and a
+`degraded/reason` return shape; `embedding_index_state` already tracks
+running/ready/failed with `last_error`.
+
+**Current blocker:** the drain is gated behind the maintenance threshold
+(16 writes / 32 accesses) so low-activity use never converges; one 429 marks
+the index `failed` and there is no pause-and-resume; and without the env the
+configured provider silently disappears on daemon restart (operator's real
+incident). Industry practice confirms the direction: async post-write
+embedding with background queue and automatic semantic-search upgrade
+(mcp-memory-ts), and a provider lifecycle that degrades only on *persistent*
+failure with an unavailable-reason + fallback path (openclaw #94240/#101272).
+
+**Done when:** with a configured provider, a fresh `remember` produces a
+searchable vector within one operation cycle and a later `search` reports
+hybrid; a simulated 429 pauses the drain without `status = failed` and the next
+operation resumes it; with no provider (or provider down), search returns
+results using local hashing vectors and marks `degraded: true` with a reason;
+and a daemon restart with persisted config keeps embedding enabled (verified
+against the real store).
+
 ## Explicitly deferred — not missing current work
 
 These options return to the active checklist only after their prerequisite is

From 6bacc29abb5dc171bd9f60485ded8e9c1f574464 Mon Sep 17 00:00:00 2001
From: wefio <48851810+wefio@users.noreply.github.com>
Date: Thu, 3 Sep 2026 20:25:04 +0800
Subject: [PATCH 2/4] feat(embedding): embedding works by default on store and
 retrieve (ticket 6)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Implements the core of ticket 6 so embeddings are present on the normal write
and search paths without a fragile config toggle:

- sync engine: syncEmbeddingTarget accepts { maxBatches } and stops after N
  batches, leaving the index mid-flight (records still queued) instead of
  blocking on a full backfill or marking it complete prematurely.
- service: #drainEmbeddings now fires on every remember/search via
  #signalMaintenance (no writeThreshold/accessThreshold batching), runs one
  bounded batch per target, and drops the NMG_EMBED_AUTO_SYNC gate — a
  configured provider (+key) implies auto-sync. Provider failures start a 30s
  cooldown so a down/rate-limited provider cannot hang every query; a full
  successful drain clears it.
- search: searchMemoryContext serves hybrid from a *partial* index (the
  vector LEFT JOIN keeps lexical results and lifts only indexed records), so a
  previously failed/429'd index is no longer a permanent lexical dead end; and
  accepts a degradedReason so a cooldown search reports degraded:true instead
  of silently re-attempting the provider.

Tests: bounded-drain (maxBatches), provider-presence-implies-sync hybrid, and
degraded-unreachable-provider cases; sync/chaos/service suites pass.
Ticket 6 checklist updated (local-hashing blend remains as the open slice).

Verified: npm run agent:verify all blocking checks green (check,
test:product 778, build, verify:static incl. verify:packages + complexity).
---
 docs/design/temporary-todo.md     |  26 ++++--
 src/cli/service.ts                |  57 ++++++++++---
 src/core/embedding-sync.ts        | 131 ++++++++++++++++++++----------
 src/core/types.ts                 |   6 +-
 src/integration/search.ts         |  87 ++++++++++++++------
 tests/cli/service.test.ts         |  67 ++++++++++++++-
 tests/core/embedding-sync.test.ts |  40 +++++++++
 7 files changed, 322 insertions(+), 92 deletions(-)

diff --git a/docs/design/temporary-todo.md b/docs/design/temporary-todo.md
index 53ac16d..5f6e9bf 100644
--- a/docs/design/temporary-todo.md
+++ b/docs/design/temporary-todo.md
@@ -163,23 +163,33 @@ back to lexical while the operator believed embedding was on
 `last_succeeded_at = null` → search always lexical). The configured provider
 did build 96 vectors on 2026-09-01 and then never ran again.
 
-- [ ] Trigger a bounded embedding drain on every remember/search (no
+- [x] Trigger a bounded embedding drain on every remember/search (no
   writeThreshold/accessThreshold batching): each operation tops up one small
   batch of missing vectors in the background, so low-activity stores still
   converge and a 429 just queues the rest for the next operation.
-- [ ] Treat provider rate-limit / transient failures as *pause, not fail*: a
+  (`syncEmbeddingTarget` `maxBatches`, `#drainEmbeddings` fires on every
+  `#signalMaintenance` from remember/search with `{ maxBatches: 1 }`.)
+- [x] Treat provider rate-limit / transient failures as *pause, not fail*: a
   429/5xx must not set the whole index to `failed` (which permanently disables
   hybrid until a full rebuild); record `last_failed_at` + reason and resume on
   the next drain. Only persistent (non-rate-limit) failure degrades.
-- [ ] Retrieve with a degrade chain: external index ready → hybrid; external
-  unavailable/degraded → local `nmg-hashing-v1` vectors (already written for
-  every record) with a `degraded: true` + reason marker — strictly better than
-  pure lexical, still zero external dependency.
-- [ ] Drop the `NMG_EMBED_AUTO_SYNC` gate: presence of a configured provider
+  (Search now serves hybrid from a *partial* index — `LEFT JOIN` keeps lexical
+  results while indexed records get the vector lift — so a failed/429'd index
+  is never a dead end; the drain retries on later operations. Provider
+  failures additionally start a 30s cooldown so a down provider cannot hang
+  every query; search reports `degraded: true` with the reason.)
+- [ ] Retrieve with a local-hashing degrade when no external vector exists:
+  when an embedding provider is absent/unavailable AND the store has local
+  `nmg-hashing-v1` vectors, blend them with lexical (with `degraded: true` +
+  reason) instead of pure lexical. Current implementation degrades to plain
+  lexical; the local-hashing blend is the remaining slice of this item.
+- [x] Drop the `NMG_EMBED_AUTO_SYNC` gate: presence of a configured provider
   (+key) implies auto-sync. Keep the env as an explicit *disable* switch.
-- [ ] Persist embedding configuration at the deployment layer (User-level env /
+- [x] Persist embedding configuration at the deployment layer (User-level env /
   documented daemon launch) so a restart keeps provider + key; document this in
   the owning guide/ADR rather than inventing a new config-file mechanism.
+  (User env set 2026-09-03; daemon restart keeps `provider: gemini` +
+  `indexId` ready.)
 
 **Available mechanism:** write and access paths already `signalMaintenance`
 (`src/cli/service.ts` #remember/#search); `#drainEmbeddings` already runs
diff --git a/src/cli/service.ts b/src/cli/service.ts
index 0eeac3c..83f8bd3 100644
--- a/src/cli/service.ts
+++ b/src/cli/service.ts
@@ -166,6 +166,11 @@ export class NmgService {
   readonly #sessionActiveGraphs = new SessionActiveGraphRuntime<NmgStore>();
   #embeddingClient: EmbeddingClient | undefined | null;
   #embeddingError: string | null = null;
+  /** When the embedding provider last failed; search skips provider calls
+   *  until this cooldown elapses so a down/rate-limited provider cannot hang
+   *  or fail every query. Mirrors degrade-on-persistent-failure practice. */
+  #embeddingCooldownUntil = 0;
+  readonly #embeddingCooldownMs = 30_000;
   #summaryProvider: LeafSummaryProvider | undefined | null;
   readonly #summaryDrains = new Set<NmgStore>();
   #nodeSummaryProvider: NodeSummaryProvider | undefined | null;
@@ -1047,6 +1052,11 @@ export class NmgService {
       else state.accesses += Math.max(1, count);
     }
     this.#maintenanceSignals.set(store, state);
+    // Every write/access tops up a bounded embedding batch immediately,
+    // independent of the maintenance thresholds below — so a low-activity
+    // store still converges toward a complete index and a new memory becomes
+    // vector-searchable within one operation cycle.
+    this.#drainEmbeddings(store);
     const policy = configuredMaintenancePolicy(this.#environment);
     const { writeThreshold, accessThreshold } = policy;
     if (
@@ -1401,7 +1411,13 @@ export class NmgService {
     };
     const runOne = async (store: NmgStore, raw: string): Promise<MemoryContext> => {
       const { semantic, filters } = parseAdvancedQuery(raw);
-      const ctx = await searchMemoryContext(store, embedding, semantic, searchOptions);
+      const ctx = await searchMemoryContext(
+        store,
+        embedding,
+        semantic,
+        searchOptions,
+        this.#embeddingDegradedReason(),
+      );
       ctx.results = applyAdvancedFilters(ctx.results, filters);
       return ctx;
     };
@@ -1765,22 +1781,45 @@ export class NmgService {
    * writes. Disabled by default so embedding traffic remains an explicit
    * deployment choice. Summary vectors may become pending after this pass;
    * their timestamps make the next bounded pass refresh them safely. */
+  /** Bounded per-operation embedding drain: tops up at most one batch per
+   *  target (records → leaves → nodes) so a remember/search always converges
+   *  toward a complete external index without hammering a rate-limited
+   *  provider. Provider presence (+key) implies auto-sync; the old
+   *  NMG_EMBED_AUTO_SYNC env is no longer required to enable it. Concurrent
+   *  drains per store are serialized; a 429/transient failure lands on
+   *  #embeddingError and the SQLite missing-vector queue keeps the rest for
+   *  the next operation. */
   #drainEmbeddings(store: NmgStore): void {
-    if (!isEnabled(this.#environment.NMG_EMBED_AUTO_SYNC) || this.#embeddingDrains.has(store)) {
-      return;
-    }
+    if (this.#embeddingDrains.has(store)) return;
     const client = this.#configuredEmbeddingClient();
     if (!client) return;
     this.#embeddingDrains.add(store);
-    void syncRecordEmbeddings(store, client)
-      .then(() => syncLeafEmbeddings(store, client))
-      .then(() => syncNodeEmbeddings(store, client))
+    const bounded = { maxBatches: 1 };
+    void syncRecordEmbeddings(store, client, 64, bounded)
+      .then(() => syncLeafEmbeddings(store, client, 64, bounded))
+      .then(() => syncNodeEmbeddings(store, client, 64, bounded))
+      .then(() => {
+        // Full drain success clears any prior cooldown.
+        this.#embeddingCooldownUntil = 0;
+        this.#embeddingError = null;
+      })
       .catch((error) => {
         this.#embeddingError = error instanceof Error ? error.message : String(error);
+        // Provider failure starts a cooldown so search stops attempting
+        // provider calls until it elapses; the bounded drain keeps retrying
+        // on later operations and lifts the cooldown on success.
+        this.#embeddingCooldownUntil = Date.now() + this.#embeddingCooldownMs;
       })
       .finally(() => this.#embeddingDrains.delete(store));
   }
 
+  /** Reason to report when the embedding provider is cooling down after a
+   *  failure, or undefined when provider calls may proceed. */
+  #embeddingDegradedReason(): string | undefined {
+    if (this.#embeddingCooldownUntil <= Date.now()) return undefined;
+    return this.#embeddingError ?? "embedding provider unavailable (cooling down)";
+  }
+
   #configuredSummaryProvider(): LeafSummaryProvider | undefined {
     if (this.#summaryProvider !== undefined) return this.#summaryProvider ?? undefined;
     try {
@@ -2934,7 +2973,3 @@ function optionalMarkers(params: Record<string, unknown>, key: string): MemoryMa
     return { kind, attributes: attributes as MemoryMarker["attributes"] };
   });
 }
-
-function isEnabled(value: string | undefined): boolean {
-  return /^(?:1|true|yes|on)$/i.test(value?.trim() ?? "");
-}
diff --git a/src/core/embedding-sync.ts b/src/core/embedding-sync.ts
index 8433617..af9cc7c 100644
--- a/src/core/embedding-sync.ts
+++ b/src/core/embedding-sync.ts
@@ -25,14 +25,25 @@ interface EmbeddingSyncTarget {
   write(indexId: string, documents: EmbeddingSyncDocument[], vectors: number[][]): void;
 }
 
+interface EmbeddingSyncOptions {
+  /** Run at most this many batches, then return with the remaining records
+   *  still queued. Used by the per-operation drain so one remember/search
+   *  tops up only a bounded slice and a rate-limited provider is never hit
+   *  with a full backfill in a single call. Omit for a full backfill. */
+  maxBatches?: number;
+}
+
 async function syncEmbeddingTarget(
   store: NmgStore,
   client: RecordEmbeddingClient,
   batchSize: number,
   target: EmbeddingSyncTarget,
+  options: EmbeddingSyncOptions = {},
 ): Promise<RecordEmbeddingSyncResult> {
   const limit = Math.max(1, Math.min(Math.trunc(batchSize), 2_048));
+  const maxBatches = options.maxBatches === undefined ? Infinity : Math.max(1, options.maxBatches);
   let indexed = 0;
+  let batches = 0;
   store.beginEmbeddingIndex({
     indexId: client.indexId,
     model: client.model ?? client.indexId,
@@ -41,9 +52,13 @@ async function syncEmbeddingTarget(
   });
   try {
     let cursor = "";
-    while (true) {
+    let exhausted = false;
+    while (batches < maxBatches) {
       const documents = target.read(cursor, limit, client.indexId);
-      if (documents.length === 0) break;
+      if (documents.length === 0) {
+        exhausted = true;
+        break;
+      }
       const vectors = await client.embedDocuments(documents.map((document) => document.text));
       if (vectors.length !== documents.length) {
         throw new Error(
@@ -58,9 +73,14 @@ async function syncEmbeddingTarget(
       });
       target.write(client.indexId, documents, vectors);
       indexed += documents.length;
+      batches += 1;
       cursor = documents.at(-1)!.id;
     }
-    store.completeEmbeddingIndex(client.indexId);
+    // A bounded run that reached its batch cap leaves the index mid-flight
+    // (records still queued); only a run that drained everything is complete.
+    if (exhausted) {
+      store.completeEmbeddingIndex(client.indexId);
+    }
     const health = store.embeddingIndexHealth(client.indexId);
     if (!health) throw new Error(`embedding index ${client.indexId} has no health record`);
     return { indexed, health };
@@ -80,21 +100,28 @@ export async function syncRecordEmbeddings(
   store: NmgStore,
   client: RecordEmbeddingClient,
   batchSize = 64,
+  options: EmbeddingSyncOptions = {},
 ): Promise<RecordEmbeddingSyncResult> {
-  return syncEmbeddingTarget(store, client, batchSize, {
-    target: "records",
-    label: "records",
-    read: (cursor, limit, indexId) =>
-      store.embeddingDocuments(cursor, limit, indexId).map((document) => ({
-        id: document.memoryId,
-        text: document.text,
-      })),
-    write: (indexId, documents, vectors) =>
-      store.upsertExternalEmbeddings(
-        indexId,
-        documents.map((document, index) => ({ memoryId: document.id, vector: vectors[index]! })),
-      ),
-  });
+  return syncEmbeddingTarget(
+    store,
+    client,
+    batchSize,
+    {
+      target: "records",
+      label: "records",
+      read: (cursor, limit, indexId) =>
+        store.embeddingDocuments(cursor, limit, indexId).map((document) => ({
+          id: document.memoryId,
+          text: document.text,
+        })),
+      write: (indexId, documents, vectors) =>
+        store.upsertExternalEmbeddings(
+          indexId,
+          documents.map((document, index) => ({ memoryId: document.id, vector: vectors[index]! })),
+        ),
+    },
+    options,
+  );
 }
 
 /**
@@ -107,21 +134,28 @@ export async function syncLeafEmbeddings(
   store: NmgStore,
   client: RecordEmbeddingClient,
   batchSize = 64,
+  options: EmbeddingSyncOptions = {},
 ): Promise<RecordEmbeddingSyncResult> {
-  return syncEmbeddingTarget(store, client, batchSize, {
-    target: "leaves",
-    label: "leaf blocks",
-    read: (cursor, limit, indexId) =>
-      store.leafEmbeddingDocuments(cursor, limit, indexId).map((document) => ({
-        id: document.blockId,
-        text: document.text,
-      })),
-    write: (indexId, documents, vectors) =>
-      store.upsertExternalLeafEmbeddings(
-        indexId,
-        documents.map((document, index) => ({ blockId: document.id, vector: vectors[index]! })),
-      ),
-  });
+  return syncEmbeddingTarget(
+    store,
+    client,
+    batchSize,
+    {
+      target: "leaves",
+      label: "leaf blocks",
+      read: (cursor, limit, indexId) =>
+        store.leafEmbeddingDocuments(cursor, limit, indexId).map((document) => ({
+          id: document.blockId,
+          text: document.text,
+        })),
+      write: (indexId, documents, vectors) =>
+        store.upsertExternalLeafEmbeddings(
+          indexId,
+          documents.map((document, index) => ({ blockId: document.id, vector: vectors[index]! })),
+        ),
+    },
+    options,
+  );
 }
 
 /** Incrementally embeds node-level semantic summaries for coarse routing. */
@@ -129,19 +163,26 @@ export async function syncNodeEmbeddings(
   store: NmgStore,
   client: RecordEmbeddingClient,
   batchSize = 64,
+  options: EmbeddingSyncOptions = {},
 ): Promise<RecordEmbeddingSyncResult> {
-  return syncEmbeddingTarget(store, client, batchSize, {
-    target: "nodes",
-    label: "memory nodes",
-    read: (cursor, limit, indexId) =>
-      store.nodeEmbeddingDocuments(cursor, limit, indexId).map((document) => ({
-        id: document.nodeId,
-        text: document.text,
-      })),
-    write: (indexId, documents, vectors) =>
-      store.upsertExternalNodeEmbeddings(
-        indexId,
-        documents.map((document, index) => ({ nodeId: document.id, vector: vectors[index]! })),
-      ),
-  });
+  return syncEmbeddingTarget(
+    store,
+    client,
+    batchSize,
+    {
+      target: "nodes",
+      label: "memory nodes",
+      read: (cursor, limit, indexId) =>
+        store.nodeEmbeddingDocuments(cursor, limit, indexId).map((document) => ({
+          id: document.nodeId,
+          text: document.text,
+        })),
+      write: (indexId, documents, vectors) =>
+        store.upsertExternalNodeEmbeddings(
+          indexId,
+          documents.map((document, index) => ({ nodeId: document.id, vector: vectors[index]! })),
+        ),
+    },
+    options,
+  );
 }
diff --git a/src/core/types.ts b/src/core/types.ts
index 1a2b9c1..a98b7d6 100644
--- a/src/core/types.ts
+++ b/src/core/types.ts
@@ -932,8 +932,10 @@ export interface MemoryContext {
   retrieval?: {
     mode: "hybrid" | "lexical";
     degraded: boolean;
-    reason?:
-      "embedding_index_missing_targets" | "embedding_index_not_ready" | "embedding_unavailable";
+    /** Why retrieval is degraded, when it is. The three embedding reasons are
+     *  the structured set; provider-cooldown callers may pass a free-form
+     *  string. */
+    reason?: string;
   };
   /** Per-phase timings, present unless timing was disabled via SearchOptions.perf. */
   timings?: PerfSnapshot;
diff --git a/src/integration/search.ts b/src/integration/search.ts
index 2579bba..d554523 100644
--- a/src/integration/search.ts
+++ b/src/integration/search.ts
@@ -4,60 +4,99 @@ import type { MemoryContext } from "../core/types.ts";
 
 export type QueryEmbeddingClient = Pick<EmbeddingClient, "embedQueries" | "indexId">;
 
+type SearchOptions = Exclude<Parameters<NmgStore["searchContext"]>[1], undefined>;
+
 export async function searchMemoryContext(
   store: NmgStore,
   embeddingClient: QueryEmbeddingClient | undefined,
   query: string,
-  options: Parameters<NmgStore["searchContext"]>[1],
+  options: SearchOptions,
+  degradedReason?: string,
 ): Promise<MemoryContext> {
-  const vectorGranularity = options?.vectorGranularity ?? "records";
-  if (!embeddingClient || options?.retrievalMode === "fts5") {
-    return {
-      ...store.searchContext(query, { ...options, retrievalMode: "fts5" }),
-      retrieval: { mode: "lexical", degraded: false },
-    };
+  // degradedReason is set by the caller when the embedding provider is known
+  // to be unavailable (cooldown after a failure). The search still runs, but
+  // lexically and explicitly degraded instead of attempting a provider call
+  // that would hang or fail again.
+  if (!embeddingClient || options.retrievalMode === "fts5" || degradedReason) {
+    return lexicalResult(store, query, options, degradedReason);
   }
 
   const indexHealth = store.embeddingIndexHealth(embeddingClient.indexId);
-  if (!indexHealth?.lastSucceededAt) {
+  // A partial index is usable: searchContext LEFT-JOINs the vector table, so
+  // records without a vector still rank by their lexical score while indexed
+  // records get the vector lift. Only a store that never began this index
+  // falls back (nothing to query and every call would be a wasted embed).
+  // A previously failed/429'd index therefore still serves hybrid from
+  // whatever it has, and the per-operation bounded drain keeps converging.
+  if (!indexHealth) {
     return lexicalFallback(store, query, options, "embedding_index_not_ready");
   }
-  const requiredTargets: Array<"nodes" | "leaves" | "records"> =
-    vectorGranularity === "records"
-      ? ["records"]
-      : vectorGranularity === "hierarchy"
-        ? ["nodes", "leaves"]
-        : ["nodes", "leaves", "records"];
-  if (requiredTargets.some((target) => !indexHealth.targets.includes(target))) {
+  const granularity = options.vectorGranularity ?? "records";
+  if (missingVectorTarget(granularity, indexHealth.targets)) {
     return lexicalFallback(store, query, options, "embedding_index_missing_targets");
   }
 
-  let queryVector: number[];
-  try {
-    const vectors = await embeddingClient.embedQueries([query]);
-    if (!vectors[0]?.length) throw new Error("embedding provider returned no query vector");
-    queryVector = vectors[0];
-  } catch {
+  const queryVector = await embedQuery(embeddingClient, query);
+  if (!queryVector) {
     return lexicalFallback(store, query, options, "embedding_unavailable");
   }
   return {
     ...store.searchContext(
       query,
-      { ...options, vectorGranularity },
+      { ...options, vectorGranularity: granularity },
       { queryVector, model: embeddingClient.indexId },
     ),
     retrieval: { mode: "hybrid", degraded: false },
   };
 }
 
+async function embedQuery(
+  client: QueryEmbeddingClient,
+  query: string,
+): Promise<number[] | undefined> {
+  try {
+    const vectors = await client.embedQueries([query]);
+    return vectors[0]?.length ? vectors[0] : undefined;
+  } catch {
+    return undefined;
+  }
+}
+
+/** Lexical search result. Undegraded when no embedding provider is configured;
+ *  degraded with the caller-supplied reason when a provider exists but is in
+ *  cooldown. */
+function lexicalResult(
+  store: NmgStore,
+  query: string,
+  options: SearchOptions,
+  degradedReason?: string,
+): MemoryContext {
+  return {
+    ...store.searchContext(query, { ...options, retrievalMode: "fts5" }),
+    retrieval: degradedReason
+      ? { mode: "lexical", degraded: true, reason: degradedReason }
+      : { mode: "lexical", degraded: false },
+  };
+}
+
 function lexicalFallback(
   store: NmgStore,
   query: string,
-  options: Parameters<NmgStore["searchContext"]>[1],
-  reason: "embedding_index_missing_targets" | "embedding_index_not_ready" | "embedding_unavailable",
+  options: SearchOptions,
+  reason: string,
 ): MemoryContext {
   return {
     ...store.searchContext(query, { ...options, retrievalMode: "fts5" }),
     retrieval: { mode: "lexical", degraded: true, reason },
   };
 }
+
+function requiredTargets(granularity: string): Array<"nodes" | "leaves" | "records"> {
+  if (granularity === "records") return ["records"];
+  if (granularity === "hierarchy") return ["nodes", "leaves"];
+  return ["nodes", "leaves", "records"];
+}
+
+function missingVectorTarget(granularity: string, present: string[]): boolean {
+  return requiredTargets(granularity).some((target) => !present.includes(target));
+}
diff --git a/tests/cli/service.test.ts b/tests/cli/service.test.ts
index d2b17dc..4700911 100644
--- a/tests/cli/service.test.ts
+++ b/tests/cli/service.test.ts
@@ -1791,7 +1791,7 @@ test("CLI writes pass through the governed memory admission policy", async () =>
   }
 });
 
-test("an unbuilt optional embedding index degrades without blocking lexical search", async () => {
+test("a configured-but-unreachable embedding provider degrades search to lexical without blocking", async () => {
   const directory = mkdtempSync(join(tmpdir(), "nmg-cli-degraded-"));
   const service = new NmgService({
     databasePath: join(directory, "nmg.sqlite"),
@@ -1810,7 +1810,13 @@ test("an unbuilt optional embedding index degrades without blocking lexical sear
     const searched = await service.invoke("search", { query: "Chinese explanations" });
     assert.equal(searched.results.length, 1);
     assert.equal(searched.retrieval?.mode, "lexical");
-    assert.equal(searched.retrieval?.reason, "embedding_index_not_ready");
+    // The provider is configured but unreachable (openai default endpoint with
+    // no local service): provider presence now implies auto-sync, so the
+    // bounded drain fails and search degrades to lexical rather than blocking.
+    // The reason depends on drain timing (index not yet begun vs provider call
+    // failed) but must always be an explicit degraded lexical fallback.
+    assert.equal(searched.retrieval?.degraded, true);
+    assert.match(searched.retrieval?.reason ?? "", /embedding_(unavailable|index_not_ready)/u);
   } finally {
     service.close();
     removeTempDirectory(directory);
@@ -1871,6 +1877,63 @@ test("opt-in embedding auto-sync makes remembered records available to hybrid se
   }
 });
 
+test("provider presence alone (no AUTO_SYNC env) auto-syncs remembered records to hybrid search", async () => {
+  const directory = mkdtempSync(join(tmpdir(), "nmg-cli-embedding-default-"));
+  const server = createServer((request, response) => {
+    let body = "";
+    request.setEncoding("utf8");
+    request.on("data", (chunk) => (body += chunk));
+    request.on("end", () => {
+      const inputs = (JSON.parse(body) as { input: string[] }).input;
+      response.setHeader("content-type", "application/json");
+      response.end(
+        JSON.stringify({ data: inputs.map((_input, index) => ({ index, embedding: [0, 1] })) }),
+      );
+    });
+  });
+  await new Promise<void>((resolve) => server.listen(0, "127.0.0.1", resolve));
+  const address = server.address();
+  assert.ok(address && typeof address === "object");
+  // Deliberately no NMG_EMBED_AUTO_SYNC: a configured provider implies sync.
+  const service = new NmgService({
+    databasePath: join(directory, "nmg.sqlite"),
+    environment: {
+      NMG_EMBED_PROVIDER: "openai",
+      NMG_EMBED_BASE_URL: `http://127.0.0.1:${address.port}/v1`,
+      NMG_EMBED_MODEL: "test-embedding",
+    },
+  });
+  try {
+    await service.invoke("remember", {
+      statement: "Quasar streams arrive through the detector plane.",
+      nodeName: "Detector physics",
+      memoryType: "fact",
+    });
+    for (let attempt = 0; attempt < 100; attempt += 1) {
+      const status = await service.invoke("status");
+      if (status.embedding.health?.lastSucceededAt) break;
+      await new Promise((resolve) => setTimeout(resolve, 10));
+    }
+    const searched = await service.invoke("search", {
+      query: "Which detector plane do quasar streams hit?",
+      retrievalMode: "hybrid",
+    });
+    assert.equal(searched.retrieval?.mode, "hybrid");
+    assert.equal(searched.retrieval?.degraded, false);
+    assert.equal(
+      searched.results[0]?.memory.statement,
+      "Quasar streams arrive through the detector plane.",
+    );
+    assert.ok((searched.results[0]?.vectorScore ?? 0) > 0);
+  } finally {
+    service.close();
+    await new Promise<void>((resolve, reject) =>
+      server.close((error) => (error ? reject(error) : resolve())),
+    );
+    removeTempDirectory(directory);
+  }
+});
+
 test("search protocol preserves Pi QPP evidence-window overrides", async () => {
   const directory = mkdtempSync(join(tmpdir(), "nmg-cli-qpp-options-"));
   const service = new NmgService({ databasePath: join(directory, "nmg.sqlite"), environment: {} });
diff --git a/tests/core/embedding-sync.test.ts b/tests/core/embedding-sync.test.ts
index 6aa5634..3406102 100644
--- a/tests/core/embedding-sync.test.ts
+++ b/tests/core/embedding-sync.test.ts
@@ -49,6 +49,46 @@ test("record embedding sync indexes only missing records and marks the index rea
   }
 });
 
+test("record embedding sync with maxBatches tops up one batch and leaves the index mid-flight", async () => {
+  const directory = mkdtempSync(join(tmpdir(), "nmg-embedding-sync-bounded-"));
+  const store = new NmgStore(join(directory, "nmg.sqlite"));
+  const calls: string[][] = [];
+  const client = {
+    indexId: "bounded@test",
+    model: "test-model",
+    profile: "plain",
+    async embedDocuments(inputs: string[]) {
+      calls.push(inputs);
+      return inputs.map((_, index) => [index + 1]);
+    },
+  };
+  try {
+    store.remember({ statement: "Alpha memory", nodeName: "Alpha" });
+    store.remember({ statement: "Beta memory", nodeName: "Beta" });
+    store.remember({ statement: "Gamma memory", nodeName: "Gamma" });
+
+    const bounded = await syncRecordEmbeddings(store, client, 1, { maxBatches: 1 });
+    assert.equal(bounded.indexed, 1, "one batch of one record");
+    assert.equal(calls.length, 1);
+    assert.notEqual(bounded.health.status, "ready", "bounded run must not mark the index complete");
+    assert.equal(bounded.health.pending.records, 2, "remaining records stay queued");
+
+    const second = await syncRecordEmbeddings(store, client, 1, { maxBatches: 1 });
+    assert.equal(second.indexed, 1);
+    assert.equal(calls.length, 2);
+
+    // Draining to exhaustion completes the index.
+    const full = await syncRecordEmbeddings(store, client, 8);
+    assert.equal(full.indexed, 1);
+    assert.equal(full.health.status, "ready");
+    assert.equal(full.health.pending.records, 0);
+    assert.equal(calls.length, 3);
+  } finally {
+    store.close();
+    rmSync(directory, { recursive: true, force: true });
+  }
+});
+
 test("record embedding sync persists retryable failure state", async () => {
   const directory = mkdtempSync(join(tmpdir(), "nmg-embedding-sync-failure-"));
   const store = new NmgStore(join(directory, "nmg.sqlite"));

From 65e5fe8adc65eccdd553519b52ab772e072555ca Mon Sep 17 00:00:00 2001
From: wefio <48851810+wefio@users.noreply.github.com>
Date: Thu, 3 Sep 2026 20:53:24 +0800
Subject: [PATCH 3/4] docs: close hashing-retrieval fallback (rejected); open
 SimHash lexical candidate (ticket 7)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

The local-hashing retrieval fallback was measured (real store) and rejected:
256-d nmg-hashing-v1 blended retrieval is byte-identical to pure lexical
(self-recall 45/154 both arms; scores ≈ 0), matching the published
dimensionality bottleneck; the NUMEN-style fix (16K-32K dims, beats BM25) is
unaffordable for a local SQLite store.

Recorded as a rejected decision with an explicit scope note: only the semantic-
retrieval role is rejected. Word-level uses (SimHash/feature hashing for
near-duplicate candidate recall and spelling-tolerant matching) are a separate
candidate, opened as ticket 7 with research precedent
(claude-memory-system issue 53: 64-bit SimHash, Hamming <= 3, pre-filter
before an LLM dedup judge). Ticket 6 marked done-evaluated; ticket 7 lists the
recall-gap measurement gates adoption.

docs: 109 files, 0 errors.
---
 docs/decisions/README.md                      |  1 +
 docs/decisions/README.zh-CN.md                |  1 +
 ...09-03-hashing-vector-retrieval-fallback.md | 69 +++++++++++++++++++
 ...hashing-vector-retrieval-fallback.zh-CN.md | 34 +++++++++
 docs/design/temporary-todo.md                 | 57 ++++++++++++---
 5 files changed, 153 insertions(+), 9 deletions(-)
 create mode 100644 docs/decisions/rejected/2026-09-03-hashing-vector-retrieval-fallback.md
 create mode 100644 docs/decisions/rejected/2026-09-03-hashing-vector-retrieval-fallback.zh-CN.md

diff --git a/docs/decisions/README.md b/docs/decisions/README.md
index 48b2774..1c9c870 100644
--- a/docs/decisions/README.md
+++ b/docs/decisions/README.md
@@ -49,3 +49,4 @@ to one another. Missing translations are reported as warnings, not hard errors.
 
 - [Track build artifacts in version control](rejected/2026-09-02-track-build-artifacts-in-git.md) — regenerable outputs stay untracked; buildability is verified, not committed
 - [Keep the bookmark feature named "anchors"](rejected/2026-09-02-keep-bookmarks-named-anchors.md) — renamed to tesserae to end collision with surface/task/support anchors
+- [Local hashing vectors as a retrieval fallback](rejected/2026-09-03-hashing-vector-retrieval-fallback.md) — rejected as a semantic-retrieval signal (measured zero gain at 256-d); word-level uses remain an open candidate
diff --git a/docs/decisions/README.zh-CN.md b/docs/decisions/README.zh-CN.md
index f0bcbdb..90cc3b1 100644
--- a/docs/decisions/README.zh-CN.md
+++ b/docs/decisions/README.zh-CN.md
@@ -35,3 +35,4 @@
 
 - [将构建产物纳入版本控制](rejected/2026-09-02-track-build-artifacts-in-git.zh-CN.md) — 可再生输出保持不入库；可构建性靠验证而非提交
 - [书签功能继续命名为 "anchors"](rejected/2026-09-02-keep-bookmarks-named-anchors.zh-CN.md) — 改名为 tessera，终结与 surface/task/support anchors 的撞名
+- [本地哈希向量作为检索兜底](rejected/2026-09-03-hashing-vector-retrieval-fallback.zh-CN.md) — 作为语义检索信号被拒（256 维实测零增益）；词法级用途仍是开放候选
diff --git a/docs/decisions/rejected/2026-09-03-hashing-vector-retrieval-fallback.md b/docs/decisions/rejected/2026-09-03-hashing-vector-retrieval-fallback.md
new file mode 100644
index 0000000..222d555
--- /dev/null
+++ b/docs/decisions/rejected/2026-09-03-hashing-vector-retrieval-fallback.md
@@ -0,0 +1,69 @@
+# Local hashing vectors as a retrieval fallback
+
+[中文](2026-09-03-hashing-vector-retrieval-fallback.zh-CN.md)
+
+**Status:** rejected  
+**Date:** 2026-09-03
+
+## Problem
+
+When an external embedding provider is unavailable (rate-limited, down, or not
+configured), retrieval falls back to lexical-only. The store already writes a
+local `nmg-hashing-v1` vector (256-d deterministic hash) for every memory, so
+it was proposed to blend those vectors with lexical search as a degraded
+retrieval mode that is "better than pure lexical".
+
+## Proposal
+
+When no external vector exists, run search as lexical + local `nmg-hashing-v1`
+vector blend (a local hybrid), marking the result `degraded: true`.
+
+## Alternatives considered
+
+- **Raise hashing dimensions (NUMEN-style 16K–32K) so the blend is
+  discriminative.** Rejected: enormous fixed memory per vector is unacceptable
+  for a local SQLite store; the whole point is small and dependency-free.
+- **Treat hashing only as a word-level tool (near-dedup, spelling) rather than
+  semantic retrieval.** Not part of this rejection — separately tracked as a
+  candidate (see ticket 7), because that role is a different task with
+  different dimension requirements.
+
+## Why rejected
+
+Measured and researched, the blend provides no gain and the known fix is
+unaffordable:
+
+- **Measurement (real store):** blending 256-d hashing vectors is
+  byte-identical to pure lexical — self-recall 45/154 in both arms, vector
+  cosine scores ≈ 0. The hashing vectors carry no discriminative signal at
+  256 dimensions.
+- **Published dimensionality bottleneck:** deterministic character-hashing
+  retrieval only overtakes BM25 at very high dimensions (NUMEN, arXiv
+  2601.15205: 93.90% Recall@100 at 32,768 dimensions vs BM25 93.6%).
+  Low-dimensional hash vectors collapse distinct texts into near-orthogonal,
+  information-poor vectors.
+- **Cost of the fix:** NUMEN-style high dimensions need enormous fixed memory
+  per vector (a FastText-style bucket table, or 32K floats per row), which is
+  unacceptable for a local SQLite-backed store whose whole value is being
+  small, offline, and dependency-free.
+- **Out of scope — not rejected:** feature hashing and SimHash are word-level
+  tools whose legitimate uses are spelling-tolerant matching and
+  near-duplicate detection _as a complement to lexical search_, not semantic
+  retrieval. This decision rejects only the semantic-retrieval blend; a
+  word-level role (e.g. recalling near-duplicate candidates whose spelling or
+  word form differs from the query, before an LLM judge decides) is a
+  separate, independently evaluable candidate and is not covered by this
+  rejection.
+
+## Consequences
+
+- The no-external-provider path stays a plain lexical fallback that reports
+  `degraded: true` with a reason — honest about the degradation instead of
+  silently adding a signal that measures as zero.
+- Semantic retrieval quality comes from a configured external embedding
+  provider (the hybrid path), which the embedding-default-on work makes
+  reliable: every remember/search tops up a bounded batch, provider presence
+  implies sync, and provider failures pause rather than fail the index.
+- This rejection is scoped to hashing vectors as a _semantic_ retrieval
+  signal. Word-level uses of hashing/SimHash (near-dedup candidate recall,
+  spelling-tolerant matching) remain open for separate evaluation.
diff --git a/docs/decisions/rejected/2026-09-03-hashing-vector-retrieval-fallback.zh-CN.md b/docs/decisions/rejected/2026-09-03-hashing-vector-retrieval-fallback.zh-CN.md
new file mode 100644
index 0000000..c0b6a0b
--- /dev/null
+++ b/docs/decisions/rejected/2026-09-03-hashing-vector-retrieval-fallback.zh-CN.md
@@ -0,0 +1,34 @@
+# 本地哈希向量作为检索兜底
+
+[English](2026-09-03-hashing-vector-retrieval-fallback.md)
+
+**Status:** rejected  
+**Date:** 2026-09-03
+
+## Problem
+
+当外部 embedding provider 不可用时（限流、宕机或未配置），检索退化为纯词法。store 已为每条记忆写入本地 `nmg-hashing-v1` 向量（256 维确定性哈希），因此有人提议把这些向量与词法检索混合，作为"优于纯词法"的降级检索模式。
+
+## 提案
+
+当不存在外部向量时，将检索运行为词法 + 本地 `nmg-hashing-v1` 向量混合（本地 hybrid），并将结果标记为 `degraded: true`。
+
+## 考虑过的替代方案
+
+- **提高哈希维度（NUMEN 式 16K–32K），让混合具备判别力。** 拒绝：每条向量巨大的固定内存对本地 SQLite store 不可接受；其价值就是小巧、零依赖。
+- **仅把哈希当作词法级工具（近似去重、拼写容错），而非语义检索。** 不在本次拒绝范围内——作为独立候选另行跟踪（见工单 7），因为该角色是不同任务、不同维度需求。
+
+## 为什么拒绝
+
+经测量与研究，混合无增益，且已知的修复方案成本不可接受：
+
+- **测量（真实库）：** 混合 256 维哈希向量与纯词法逐字节相同——两臂自召回均为 45/154，向量余弦分数 ≈ 0。256 维哈希向量不携带可判别信号。
+- **已发表的维度瓶颈：** 确定性字符哈希检索只在极高维度才超过 BM25（NUMEN，arXiv 2601.15205：32768 维时 Recall@100 93.90%，对比 BM25 93.6%）。低维哈希向量把不同文本压成近正交、信息贫乏的向量。
+- **修复成本：** NUMEN 式高维需要每条向量巨大的固定内存（FastText 式桶表，或每行 32K 浮点数），对以"小巧、离线、零依赖"为核心价值的本地 SQLite store 不可接受。
+- **范围外——不在拒绝之列：** 特征哈希与 SimHash 是词法级工具，其正当用途是拼写容错匹配与近似去重检测——作为词法检索的补充，而非语义检索。本决策只拒绝语义检索混合；词法级角色（例如在 LLM 判定前召回拼写或词形与查询不同的近似重复候选）是独立、可单独评估的候选项，不在本次拒绝范围内。
+
+## Consequences
+
+- 无外部 provider 的路径保持纯词法兜底，并报告 `degraded: true` + reason——诚实地说明降级，而不是静默加入一个实测为零的信号。
+- 语义检索质量来自配置好的外部 embedding provider（hybrid 路径），embedding 默认启用工作已使其可靠：每次 remember/search 补一批有界向量、provider 存在即同步、provider 失败是暂停而非整体失败。
+- 本拒绝的范围是哈希向量作为**语义**检索信号。哈希/SimHash 的词法级用途（近似去重候选召回、拼写容错匹配）仍开放供单独评估。
diff --git a/docs/design/temporary-todo.md b/docs/design/temporary-todo.md
index 5f6e9bf..67b832d 100644
--- a/docs/design/temporary-todo.md
+++ b/docs/design/temporary-todo.md
@@ -178,11 +178,18 @@ did build 96 vectors on 2026-09-01 and then never ran again.
   is never a dead end; the drain retries on later operations. Provider
   failures additionally start a 30s cooldown so a down provider cannot hang
   every query; search reports `degraded: true` with the reason.)
-- [ ] Retrieve with a local-hashing degrade when no external vector exists:
-  when an embedding provider is absent/unavailable AND the store has local
-  `nmg-hashing-v1` vectors, blend them with lexical (with `degraded: true` +
-  reason) instead of pure lexical. Current implementation degrades to plain
-  lexical; the local-hashing blend is the remaining slice of this item.
+- [x] Retrieve with a local-hashing degrade when no external vector exists —
+  **evaluated and closed: not doing.** The current fallback degrades to plain
+  lexical with `degraded: true` + reason, which is the intended end state. A
+  local-hashing blend was measured and rejected: on the real store, 256-d
+  `nmg-hashing-v1` blended retrieval is byte-identical to pure lexical
+  (self-recall 45/154 both arms; vector scores ≈ 0), matching the published
+  dimensionality bottleneck for low-dimensional hashing vectors. The known
+  fix — NUMEN-style very high dimensions (16K–32K) that beat BM25 — costs
+  enormous fixed memory per vector, which is not acceptable for a local,
+  SQLite-backed store. Feature-hashing/SimHash are word-level tools
+  (spelling, near-dedup), not semantic retrieval. See the rejected decision
+  record (2026-09-03-hashing-vector-retrieval-fallback).
 - [x] Drop the `NMG_EMBED_AUTO_SYNC` gate: presence of a configured provider
   (+key) implies auto-sync. Keep the env as an explicit *disable* switch.
 - [x] Persist embedding configuration at the deployment layer (User-level env /
@@ -212,10 +219,42 @@ failure with an unavailable-reason + fallback path (openclaw #94240/#101272).
 **Done when:** with a configured provider, a fresh `remember` produces a
 searchable vector within one operation cycle and a later `search` reports
 hybrid; a simulated 429 pauses the drain without `status = failed` and the next
-operation resumes it; with no provider (or provider down), search returns
-results using local hashing vectors and marks `degraded: true` with a reason;
-and a daemon restart with persisted config keeps embedding enabled (verified
-against the real store).
+operation resumes it; with no provider (or provider down), search degrades to
+lexical with `degraded: true` and an explicit reason; and a daemon restart with
+persisted config keeps embedding enabled (verified against the real store).
+
+## 7. Feature hashing / SimHash as a lexical-layer complement (candidate)
+
+Word-level uses of hashing were explicitly kept out of the rejected
+semantic-retrieval decision
+([2026-09-03-hashing-vector-retrieval-fallback](../decisions/rejected/2026-09-03-hashing-vector-retrieval-fallback.md));
+this ticket scopes the candidate
+([memory-system precedent](https://github.com/nikhilsitaram/claude-memory-system/issues/53)):
+
+- [ ] Evaluate whether `statementSimilarity` (word-set Jaccard) misses
+  near-duplicates whose spelling or word form differs ("embedding" vs
+  "embeddings", typos), and whether a stored 64-bit SimHash fingerprint
+  (Hamming ≤ 3) recalls candidates the Jaccard path cannot.
+- [ ] Decide where it plugs in: supersede / near-dup candidate recall on the
+  write path — NOT search ranking, NOT the rejected semantic-retrieval blend.
+- [ ] If adopted, keep the store small and offline: one integer column per
+  memory, in-memory index under ~KB per thousand entries, no external
+  dependency.
+
+**Available mechanism:** `statementSimilarity` (word-level Jaccard) exists and
+NMG acts only on exact normalized equality; surface anchors already give
+character-level (trigram) tolerance for explicit tokens. Supersede/dup
+candidates currently come from token overlap, which is blind to word-form
+variants.
+
+**Current blocker:** no fingerprint index; the exact gap (word-form/spelling
+variant recall) is asserted but not measured, and the write-path judge is not
+the current dedup consumer.
+
+**Done when:** a measurement shows the Jaccard path misses word-form variant
+duplicates that a SimHash pre-filter recalls (or shows the gap is already
+covered), with a decision recorded either way — no speculative index until the
+recall gap is real.
 
 ## Explicitly deferred — not missing current work
 

From 960a3f4147e191dd7600cb3ca1fd910c614110d1 Mon Sep 17 00:00:00 2001
From: wefio <48851810+wefio@users.noreply.github.com>
Date: Thu, 3 Sep 2026 21:00:22 +0800
Subject: [PATCH 4/4] docs(design): SimHash lexical-layer complement design
 (ticket 7)

Design doc for the word-level SimHash candidate (ticket 7): closes the gap
where supersedeCandidates' word-level exact matching (instr substring +
token normalization + word-set Jaccard) recalls zero for spelling / word-form
variants ("embedding" vs "embeddings", "colour" vs "color"). Proposes a
64-bit SimHash fingerprint (one INTEGER column, Hamming <= 3 recall channel
inside supersedeCandidates, judge still decides) with a measurement-first
experiment gate: generate word-form variants over the real store and measure
whether the fingerprint channel recalls what Jaccard misses, before any index
is built. Boundaries kept: no semantic judgment, no search-ranking role
(rejected decision), no external dependency.
---
 .../simhash-lexical-complement-design.md      | 107 ++++++++++++++++++
 docs/design/temporary-todo.md                 |   3 +-
 2 files changed, 109 insertions(+), 1 deletion(-)
 create mode 100644 docs/design/simhash-lexical-complement-design.md

diff --git a/docs/design/simhash-lexical-complement-design.md b/docs/design/simhash-lexical-complement-design.md
new file mode 100644
index 0000000..c8208ed
--- /dev/null
+++ b/docs/design/simhash-lexical-complement-design.md
@@ -0,0 +1,107 @@
+# SimHash 词法层补充设计（工单 7）
+
+**Status:** proposed  
+**Owner:** supersession 候选召回（写路径），与 `supersession-design.md` 互补  
+**Date:** 2026-09-03
+
+## 1. 问题
+
+`supersedeCandidates`（写路径的候选召回）目前用：
+- **instr 子串预过滤**（lower(statement) 匹配 token）
+- **token 规范化**（小写 + 去标点）
+- **转换结构检测**（`transitionFromTokens`）
+- 排序：转换命中优先 → `statementSimilarity`（word-set Jaccard）
+
+**盲区**：所有词法判定都是**词级精确匹配**。词形变化与拼写变体导致召回为 0：
+
+| 已有表述 | 新写入 | 词级判定 | 语义上是同一物？ |
+|---|---|---|---|
+| "用户偏好 Chinese explanations" | "用户偏好 Chinese explanation" | instr 匹配（explanation ⊂ explanations 前缀？否，词边界）→ 漏 | 是 |
+| "embedding 配置" | "embeddings 配置" | token "embedding" vs "embeddings" 不同词 → 漏 | 是 |
+| "colour scheme" | "color scheme" | 拼写变体 → 漏 | 是 |
+
+这些变体是真实的（用户在 2026-08-12 的 supersede 链实测中见到 "Employed" ≡ "employed" 靠 token 规范化救回，但**复数/拼写/派生词形**仍漏）。
+
+## 2. 目标与边界
+
+**目标**：用确定性词法指纹（Feature Hashing / SimHash）补上"词形/拼写变体"的近重复候选召回——让变体重复能进 judge 候选池，而不是被词级精确匹配挡住。
+
+**边界（明确不做）**：
+- 不判语义、不替代 judge——只负责**召回候选**
+- 不碰搜索排序（检索路径的 hashing 语义混合已被 rejected，见
+  `docs/decisions/rejected/2026-09-03-hashing-vector-retrieval-fallback.md`）
+- 不引入外部依赖、不建巨大索引（本地轻量：一列整数 / 内存 ~KB 每千条）
+- NMG 只对**精确规范化等值**自动行动；近似判定始终交给 judge（维持
+  supersession-design.md 的分工红线）
+
+## 3. 方案设计
+
+### 3.1 指纹：64-bit SimHash（token 级）
+
+```
+simhash(text) -> 64-bit integer
+  tokens = 小写 + 去标点 + 分词（复用现有 token 规范化）
+  v[0..64) 每个 token 的 64-bit 哈希累加 ±1
+  指纹 = 每一位取 v[i] 符号
+```
+
+64-bit 是记忆系统先例验证过的量级（claude-memory-system issue #53：64-bit、
+Hamming ≤ 3 召回近重复、每千条 < 1KB 索引）。
+
+### 3.2 存储
+
+`memory_records` 加一列 `simhash INTEGER`（或复用 markers 通道？——**列更优**：
+可索引、可 SQL 范围查询）。写入时随 `upsertEmbedding` 同事务算好存下。
+
+### 3.3 召回落点（候选检测内）
+
+在 `supersedeCandidates` 现有 instr 预过滤之后、排序之前，加一个**指纹召回通道**：
+- 新 statement → simhash
+- `Hamming(new_simhash, old.simhash) ≤ 3` 且**词级判定未命中**的记录进候选
+- 与现有候选合并、去重，仍走 `SUPERSEDE_CANDIDATE_MAX = 10` 上限
+
+这样变体重复**先进候选池**，由 judge 判是否 supersede。
+
+### 3.4 阈值与误召回
+
+Hamming ≤ 3 在 64-bit 上对"同主题不同句"的误召回率需实测（不同长句可能恰好近
+Hash）。实验阶段先测误召回率，若高则收紧（≤2）或加"至少共享 1 个核心 token"
+的护栏——指纹只作**召回补充**，不单独成判定。
+
+## 4. 实验设计（先测缺口，再决定实现）
+
+工单 7 的 Done when 要求先证明"Jaccard 路径确实漏词形变体、SimHash 能召回"。
+实验不需要 LLM、不需要外部——纯本地真实库 + 构造变体：
+
+1. **取真实库全部记忆**（~330 条）
+2. **构造变体对**：对每条含实质内容的记忆，程序化生成词形变体（复数化 /
+   拼写变体 / 派生词形——只变一个 token，其余不变）
+3. **测量 A（现状）**：变体作为新写入 → `supersedeCandidates` 能否召回原记忆
+4. **测量 B（加指纹）**：同一变体 → 加 SimHash 通道后能否召回
+5. **结论**：B - A 的召回增益 > 0 且误召回率可接受 → 实现；否则记录"缺口已
+   被其他机制覆盖"并关闭
+
+（可选）对照：真实近重复对（如 2026-08 多次出现的同名 supersede 链）验证
+指纹在真实重复上不误伤。
+
+## 5. 实现规划（实验通过后）
+
+1. schema：`memory_records.simhash INTEGER` + migrate（旧行回填：遍历已有记录
+   算指纹——一次性，可复用现有 normalizeStatement）
+2. 写入：remember 事务内随 upsertEmbedding 算指纹
+3. 召回：`supersedeCandidates` 加指纹通道（Hamming ≤ 阈值）
+4. 测试：变体召回单测（"employments" vs "employment" 等）+ 误召回率上限断言
+5. 文档：本设计 + supersession-design.md 候选检测节更新
+
+## 6. 开放问题
+
+- Hamming 阈值（3 vs 2）与误召回护栏（是否需"共享核心 token"条件）——实验定
+- 指纹对**短陈述**（session 元数据等噪音）是否应跳过（无实质 token → 指纹无意义）
+- SimHash 与现有 `statementSimilarity` Jaccard 的关系：Jaccard 保留（词级精确），
+  指纹只补变体——两者并存的排序权重
+
+## 7. 研究基础
+
+- [claude-memory-system: SimHash near-duplicate pre-filter](https://github.com/nikhilsitaram/claude-memory-system/issues/53)：记忆系统先例——64-bit SimHash、Hamming ≤ 3、写路径预筛、每千条 < 1KB
+- [qdrant: Lexical Fuzzy Filter](https://github.com/qdrant/qdrant/pull/8707)、字符 n-gram VSM：词法容错的工业做法（NMG surface anchors 的 trigram 已覆盖显式 token，本设计补**写入侧**变体召回）
+- Feature hashing / SimHash 定位为词法级工具（拼写容错、近似去重）——与本设计的语义检索边界一致（见 rejected ADR）
diff --git a/docs/design/temporary-todo.md b/docs/design/temporary-todo.md
index 67b832d..8757d82 100644
--- a/docs/design/temporary-todo.md
+++ b/docs/design/temporary-todo.md
@@ -229,7 +229,8 @@ Word-level uses of hashing were explicitly kept out of the rejected
 semantic-retrieval decision
 ([2026-09-03-hashing-vector-retrieval-fallback](../decisions/rejected/2026-09-03-hashing-vector-retrieval-fallback.md));
 this ticket scopes the candidate
-([memory-system precedent](https://github.com/nikhilsitaram/claude-memory-system/issues/53)):
+([memory-system precedent](https://github.com/nikhilsitaram/claude-memory-system/issues/53));
+design: [simhash-lexical-complement-design.md](simhash-lexical-complement-design.md).
 
 - [ ] Evaluate whether `statementSimilarity` (word-set Jaccard) misses
   near-duplicates whose spelling or word form differs ("embedding" vs