Skip to content

memory-core main reindex thrashes, leaks main.sqlite.tmp files, and leaves memory_search paused after repair #90508

Closed

Description

@joeykrug

Summary

On OpenClaw 2026.6.1 (2e08f0f), the main memory index got into a reindex-thrash loop on a large active index. The live SQLite DB had chunks and cached embeddings, but no identity metadata row, so vector search failed closed with index metadata is missing. Repeated rebuild attempts left many abandoned main.sqlite.tmp-* files, and after a successful CLI rebuild the in-session dynamic memory_search tool still served the stale paused state until manager/gateway reload.

This looks like a combined memory-core failure mode:

  1. interrupted rebuilds do not clean up temporary SQLite files;
  2. large/active indexes can keep re-dirtying and spawning full rebuild attempts;
  3. a successful on-disk rebuild does not invalidate the live memory-search manager's cached disabled/identity state.

Live repro/evidence from local agent

Environment:

  • OpenClaw: 2026.6.1 (2e08f0f)
  • OS: Linux x64
  • Agent: main
  • Memory config: provider openai, model text-embedding-3-large, sources memory,sessions

Before repair:

  • Live DB: /home/exedev/.openclaw/memory/main.sqlite
  • Size: about 2.3G
  • PRAGMA integrity_check: ok
  • chunks: 5462
  • embedding_cache: 34255
  • meta rows: 0
  • openclaw memory status --agent main reported:
    • Indexed: 412/2360 files · 5462 chunks
    • Dirty: yes
    • Index identity: index metadata is missing
    • Vector search: paused until memory is rebuilt

Tmp leak/thrash evidence:

  • Around 2026-06-04 21:46 AST, there were 10 abandoned main.sqlite.tmp-* DB families totaling roughly 40GB before cleanup.
  • One Codex-spawned openclaw-memory process had open FDs to main.sqlite.tmp-457c93d7-46cc-49e2-87d7-e0812ce8119b{,-wal,-shm} but file mtimes/sizes stayed unchanged during a 30s observation window after about 20 minutes of runtime. It looked stalled, not active.
  • After deleting only tmp files not held open, terminating the stale process with SIGTERM, and deleting its leftover tmp family, memory dir usage fell to about 3.2G after the clean rebuild.

Manual repair that succeeded on disk:

openclaw memory index --agent main --force

Post-repair CLI verification:

  • Command printed Memory index updated (main).
  • No main.sqlite.tmp-* files remained.
  • Live DB had meta row memory_index_meta_v1:
    • provider openai
    • model text-embedding-3-large
    • vectorDims 3072
  • chunks: 5522
  • embedding_cache: 34318
  • openclaw memory status --agent main --index --json reported:
    • dirty: false
    • vector.enabled: true
    • vector.available: true
    • custom.indexIdentity.status: valid
    • embedding probe ok: true
  • openclaw memory search --agent main --query 'Morningstar NHYDY fair value' --max-results 3 returned results.

Still broken in live manager/tool path after repair:

The dynamic in-session memory_search tool still returned:

{
  "disabled": true,
  "unavailable": true,
  "error": "index metadata is missing",
  "action": "Tell the user to run: openclaw memory status --index or openclaw memory index --force."
}

That was after the CLI status/index probe already saw the index as valid and active. This suggests the memory-search manager caches the disabled identity state and does not re-read the repaired index metadata.

Expected behavior

  • A failed/interrupted rebuild should remove its main.sqlite.tmp-* files, or mark them recoverable and clean them on next startup/status/index operation.
  • Only one rebuild should run per agent/index at a time; concurrent or repeated full rebuild attempts should be coalesced/locked.
  • After a successful rebuild and atomic swap, the live search manager should invalidate any cached disabled state and immediately observe the new memory_index_meta_v1 identity.
  • Large session-backed indexes should not repeatedly dirty themselves into full rebuild thrash.

Related issues

This seems related but not identical to:

The distinct repro here is the combined loop: large active main index + missing meta + repeated abandoned tmp rebuilds + CLI repair succeeds while dynamic memory_search remains paused until manager reload.

Activity

  1. clawsweeper commented on Jun 5, 2026

    @clawsweeper
    Contributor

    Thanks for the context here. I did a careful shell check against current main, and this is already implemented.

    Close as implemented on current main: memory reindex now has a manager-owned lock, shadow build/publication lifecycle, cleanup, and a direct regression test proving a live manager recovers after a separate CLI reindex without restart.

    So I’m closing this as already implemented rather than keeping a duplicate issue open.

    Review details

    Best possible solution:

    Keep the manager-owned lock, shadow-build, atomic-publication, and identity-refresh flow; reopen only with a current-main reproduction of remaining legacy artifact cleanup failure.

    Do we have a high-confidence way to reproduce the issue?

    Yes—current regression coverage defines the reported order: keep a manager live, repair through a separate CLI reindex, then verify that the same manager becomes valid and returns search content. This read-only review did not execute the test.

    Is this the best way to solve the issue?

    Yes—the current manager-owned publish/reload contract is preferable to the rejected filesystem-polling approach because the reindex owner records and publishes the authoritative state directly.

    Security review:

    Security review: This issue has no proposed patch to assess.

    AGENTS.md: found and applied where relevant.

    What I checked:

    • Live-manager recovery on current main: The manager refreshes index identity before search decisions, and the regression test performs a separate CLI-purpose forced reindex against an already-live manager, then requires valid identity and successful repaired search without reopening it. (extensions/memory-core/src/memory/manager.self-heal-missing-identity.test.ts:188, e69c3df2361b)
    • Manager-owned reindex lifecycle: Current reindex builds a shadow database under an exclusive cross-process lock, publishes atomically, removes the shadow family in finally, and has lock-guarded aged-orphan cleanup coverage. (extensions/memory-core/src/memory/manager-sync-ops.ts:498, e69c3df2361b)
    • Fix provenance: This commit added the live-manager-after-CLI-reindex regression coverage and changed the reindex/database/lock implementation paths; it is the best identifiable provenance for the current behavior. (extensions/memory-core/src/memory/manager.self-heal-missing-identity.test.ts:188, 39898fe91bd0)
    • Release containment check: No local version tag contains the provenance commit, and ancestry against the supplied latest release tag is unavailable in this partial/grafted checkout; the fix is therefore recorded as current-main behavior rather than claiming a shipped release. (39898fe91bd0)
    • Maintainer direction now satisfied: The member closeout requested one manager-owned publication/reload contract and rejected per-observer filesystem polling; current main implements the manager-owned lifecycle and direct recovery coverage.

    Likely related people:

    • vincentkoc: History includes stale-temp cleanup and stale-publication work in the same reindex lifecycle. (role: recent memory lifecycle contributor; confidence: high; commits: a42bda5b37e9, 25e68e9; files: extensions/memory-core/src/memory/manager-sync-ops.ts, extensions/memory-core/src/memory/manager-db.ts)
    • 849261680: Introduced the direct live-manager-after-CLI-reindex regression coverage and related lock/publication changes. (role: recovery regression contributor; confidence: high; commits: 39898fe91bd0; files: extensions/memory-core/src/memory/manager.self-heal-missing-identity.test.ts, extensions/memory-core/src/memory/manager-reindex-lock.ts)
    • steipete: Current-file history attributes the latest carried-forward memory implementation lines to a broad recent refactor, though it is not the semantic fix provenance. (role: recent adjacent refactor contributor; confidence: medium; commits: 4511c9b71f78; files: extensions/memory-core/src/memory/manager.ts, extensions/memory-core/src/memory/manager-sync-ops.ts)

    Codex review notes: model internal, reasoning high; reviewed against e69c3df2361b; fix evidence: commit 39898fe91bd0, main fix timestamp 2026-07-01T08:59:20Z.

  2. vincentkoc commented on Jun 15, 2026

    @vincentkoc
    Member

    Maintainer closeout: the stale manager/tmp-reindex failure cluster remains valid, but PR #92943 is closed. Its per-observer filesystem identity polling conflicts with the memory hot-path ownership rule, and making publication/recovery coherent expanded into a broad lifecycle rewrite that still introduced a synchronous search-path polling regression under fresh autoreview.

    Keeping this issue open for one manager-owned publication/reload contract with live gateway proof; guarded temp cleanup remains separately addressed by #92891.

  3. added
    P1High-priority user-facing bug, regression, or broken workflow.
    impact:crash-loopCrash, hang, restart loop, or process-level availability failure.
    impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.
    issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.
    clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.
    clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.
    clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.
    clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.
    clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.
    issue-rating: 🐚 platinum hermitGood issue quality with a plausible reproduction path needing some confirmation.
    clawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.
    and removed
    clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.
    issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.
    on Jun 19, 2026
  4. 2 remaining items

  5. added
    issue-rating: 🦐 gold shrimpDecent issue quality, but reproduction details are still incomplete.
    clawsweeper:needs-infoClawSweeper needs more reporter information before it can verify this issue.
    issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.
    clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.
    and removed
    clawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.
    issue-rating: 🐚 platinum hermitGood issue quality with a plausible reproduction path needing some confirmation.
    clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.
    clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.
    clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.
    clawsweeper:needs-infoClawSweeper needs more reporter information before it can verify this issue.
    issue-rating: 🦐 gold shrimpDecent issue quality, but reproduction details are still incomplete.
    on Jul 31, 2026
  6. self-assigned this
    on Aug 12, 2026
  7. vincentkoc commented on Aug 12, 2026

    @vincentkoc
    Member

    Closing as completed. The memory reindex lifecycle is now repaired at its owner boundary:

    The first three repairs shipped in https://github.com/openclaw/openclaw/releases/tag/v2026.6.10, and the complete chain shipped in stable https://github.com/openclaw/openclaw/releases/tag/v2026.7.1. Focused real node:sqlite proof keeps a live manager open, performs a separate CLI forced reindex, then verifies valid identity and newly indexed search results without reopening the manager.

    Please reopen only with a current-release reproduction where the live manager remains paused after CLI repair, or aged unlocked reindex shadow files remain on disk.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P1High-priority user-facing bug, regression, or broken workflow.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.impact:crash-loopCrash, hang, restart loop, or process-level availability failure.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.

Type

No type

Fields

Priority

None yet

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions