Repository navigation
Description
Summary
On OpenClaw 2026.6.1 (2e08f0f), the main memory index got into a reindex-thrash loop on a large active index. The live SQLite DB had chunks and cached embeddings, but no identity metadata row, so vector search failed closed with index metadata is missing. Repeated rebuild attempts left many abandoned main.sqlite.tmp-* files, and after a successful CLI rebuild the in-session dynamic memory_search tool still served the stale paused state until manager/gateway reload.
This looks like a combined memory-core failure mode:
- interrupted rebuilds do not clean up temporary SQLite files;
- large/active indexes can keep re-dirtying and spawning full rebuild attempts;
- a successful on-disk rebuild does not invalidate the live memory-search manager's cached disabled/identity state.
Live repro/evidence from local agent
Environment:
- OpenClaw:
2026.6.1 (2e08f0f) - OS: Linux x64
- Agent:
main - Memory config: provider
openai, modeltext-embedding-3-large, sourcesmemory,sessions
Before repair:
- Live DB:
/home/exedev/.openclaw/memory/main.sqlite - Size: about
2.3G PRAGMA integrity_check:okchunks:5462embedding_cache:34255metarows:0openclaw memory status --agent mainreported:Indexed: 412/2360 files · 5462 chunksDirty: yesIndex identity: index metadata is missingVector search: paused until memory is rebuilt
Tmp leak/thrash evidence:
- Around
2026-06-04 21:46 AST, there were 10 abandonedmain.sqlite.tmp-*DB families totaling roughly40GBbefore cleanup. - One Codex-spawned
openclaw-memoryprocess had open FDs tomain.sqlite.tmp-457c93d7-46cc-49e2-87d7-e0812ce8119b{,-wal,-shm}but file mtimes/sizes stayed unchanged during a 30s observation window after about 20 minutes of runtime. It looked stalled, not active. - After deleting only tmp files not held open, terminating the stale process with
SIGTERM, and deleting its leftover tmp family, memory dir usage fell to about3.2Gafter the clean rebuild.
Manual repair that succeeded on disk:
openclaw memory index --agent main --forcePost-repair CLI verification:
- Command printed
Memory index updated (main). - No
main.sqlite.tmp-*files remained. - Live DB had
metarowmemory_index_meta_v1:- provider
openai - model
text-embedding-3-large - vectorDims
3072
- provider
chunks:5522embedding_cache:34318openclaw memory status --agent main --index --jsonreported:dirty: falsevector.enabled: truevector.available: truecustom.indexIdentity.status: valid- embedding probe
ok: true
openclaw memory search --agent main --query 'Morningstar NHYDY fair value' --max-results 3returned results.
Still broken in live manager/tool path after repair:
The dynamic in-session memory_search tool still returned:
{
"disabled": true,
"unavailable": true,
"error": "index metadata is missing",
"action": "Tell the user to run: openclaw memory status --index or openclaw memory index --force."
}That was after the CLI status/index probe already saw the index as valid and active. This suggests the memory-search manager caches the disabled identity state and does not re-read the repaired index metadata.
Expected behavior
- A failed/interrupted rebuild should remove its
main.sqlite.tmp-*files, or mark them recoverable and clean them on next startup/status/index operation. - Only one rebuild should run per agent/index at a time; concurrent or repeated full rebuild attempts should be coalesced/locked.
- After a successful rebuild and atomic swap, the live search manager should invalidate any cached disabled state and immediately observe the new
memory_index_meta_v1identity. - Large session-backed indexes should not repeatedly dirty themselves into full rebuild thrash.
Related issues
This seems related but not identical to:
- Slack socket starved by blocking SQLite VACUUM on bloated main.sqlite; interrupted vacuums leak 53GB of orphaned .tmp files #83712
- Gateway memory_search index stuck dirty: provider.model empty during boot overwrites correct index identity #90042
- Memory index meta never written when gateway auto-sync finds identity missing with existing chunks #90338
- [Bug]:Intermittent memory_search "index metadata is missing" despite valid builtin memory index; likely search/reindex race. Locally hotfixed. #90361
- memory status always reports "Vector search: paused" with blank expected model (2026.6.1) #90413
- agentmemory__memory_search returns "index metadata is missing" persistently (memory-core manager state cache) #90414
The distinct repro here is the combined loop: large active main index + missing meta + repeated abandoned tmp rebuilds + CLI repair succeeds while dynamic memory_search remains paused until manager reload.
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Activity
clawsweeper commentedon Jun 5, 2026
Thanks for the context here. I did a careful shell check against current
main, and this is already implemented.Close as implemented on current main: memory reindex now has a manager-owned lock, shadow build/publication lifecycle, cleanup, and a direct regression test proving a live manager recovers after a separate CLI reindex without restart.
So I’m closing this as already implemented rather than keeping a duplicate issue open.
Review details
Best possible solution:
Keep the manager-owned lock, shadow-build, atomic-publication, and identity-refresh flow; reopen only with a current-main reproduction of remaining legacy artifact cleanup failure.
Do we have a high-confidence way to reproduce the issue?
Yes—current regression coverage defines the reported order: keep a manager live, repair through a separate CLI reindex, then verify that the same manager becomes valid and returns search content. This read-only review did not execute the test.
Is this the best way to solve the issue?
Yes—the current manager-owned publish/reload contract is preferable to the rejected filesystem-polling approach because the reindex owner records and publishes the authoritative state directly.
Security review:
Security review: This issue has no proposed patch to assess.
AGENTS.md: found and applied where relevant.
What I checked:
extensions/memory-core/src/memory/manager.self-heal-missing-identity.test.ts:188, e69c3df2361b)extensions/memory-core/src/memory/manager-sync-ops.ts:498, e69c3df2361b)extensions/memory-core/src/memory/manager.self-heal-missing-identity.test.ts:188, 39898fe91bd0)Likely related people:
extensions/memory-core/src/memory/manager-sync-ops.ts,extensions/memory-core/src/memory/manager-db.ts)extensions/memory-core/src/memory/manager.self-heal-missing-identity.test.ts,extensions/memory-core/src/memory/manager-reindex-lock.ts)extensions/memory-core/src/memory/manager.ts,extensions/memory-core/src/memory/manager-sync-ops.ts)Codex review notes: model internal, reasoning high; reviewed against e69c3df2361b; fix evidence: commit 39898fe91bd0, main fix timestamp 2026-07-01T08:59:20Z.
vincentkoc commentedon Jun 15, 2026
Maintainer closeout: the stale manager/tmp-reindex failure cluster remains valid, but PR #92943 is closed. Its per-observer filesystem identity polling conflicts with the memory hot-path ownership rule, and making publication/recovery coherent expanded into a broad lifecycle rewrite that still introduced a synchronous search-path polling regression under fresh autoreview.
Keeping this issue open for one manager-owned publication/reload contract with live gateway proof; guarded temp cleanup remains separately addressed by #92891.
2 remaining items
vincentkoc commentedon Aug 12, 2026
Closing as completed. The memory reindex lifecycle is now repaired at its owner boundary:
The first three repairs shipped in https://github.com/openclaw/openclaw/releases/tag/v2026.6.10, and the complete chain shipped in stable https://github.com/openclaw/openclaw/releases/tag/v2026.7.1. Focused real
node:sqliteproof keeps a live manager open, performs a separate CLI forced reindex, then verifies valid identity and newly indexed search results without reopening the manager.Please reopen only with a current-release reproduction where the live manager remains paused after CLI repair, or aged unlocked reindex shadow files remain on disk.