fix(memory): loud signal when semantic recall degrades to non-semantic - #2015
Merged
Conversation
…c (stop it hiding) multi_layer_recall silently falls back to non-semantic order when the query embedding is absent/lexical or no memory carries a vector — the semantic layer just returns [] and the importance/recency/keyword layers survive, ignoring the query text. That LOOKS like working recall (returns memories) but the ranking is meaningless — the exact trap the agent-memory bridge hit tonight (off-topic results for every query). Now: when a query was given but the semantic layer couldn't contribute, log LOUD — embedder id, memories-with-vectors count, whether the query embedded — so this class of degradation is visible, not silent ([[reliability-is-it-works-not-that-it-reports-failure-well]]). Diagnostic only (no behavior change). The actual fix — wire the corpus manager to the neural embedder (currently hardcoded lexical, main.rs:343) + populate memory vectors — is the follow-up; the qwen3-embedding-0.6b GGUF is already on disk so it's tractable. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
joelteply
enabled auto-merge (squash)
July 26, 2026 02:40
joelteply
added a commit
that referenced
this pull request
Jul 26, 2026
…embedder + governed VRAM + persistent cache + coma fix) (#2016) * fix(memory): global recall embedder resolves neural lazily + on-demand candidate-vector backfill — semantic recall for the agent-memory bridge The agent-memory bridge (memory/remember, memory/recall-hook) recalls through the GLOBAL PersonaMemoryManager, which main.rs constructed with the hardcoded LEXICAL bootstrap embedder — deliberately, because NOTHING may gate the IPC socket bind on a GPU/gateway probe (concurrency guide). The per-persona path resolved neural on spawn; the global/bridge path never did. Result, dogfooded live: recall returned the SAME off-topic memories for every query — non-semantic order that ignores the query text. Two gaps, both closed here, reusing the existing resolver + storage (no new machinery): 1. LazyRecallEmbedder (cognition/embedding.rs) — resolves the dedicated in-process Qwen3-Embedding-0.6B on the FIRST recall via new resolve_recall_embedder_local() (paths: in-process GGUF → lexical floor; no chat adapter, which the global manager has none of), caches it process-stably, delegates. Zero boot-path cost — the probe + calibration is paid once on first real recall, never at socket-bind. This is the "separate addressing follow-up" main.rs's old comment promised. 2. ensure_memory_embeddings (memory/mod.rs) — agent memories are written with embedding: None, so corpus.memories_with_embeddings() was empty and SemanticRecallLayer no-op'd even with a neural query vector. multi_layer_recall now backfills missing candidate vectors on demand (async, cached, outside any lock) before the sync recall, so the semantic layer has something to rank. First recall per persona pays it; content is immutable so vectors never go stale; idempotent once populated. Tests: ensure_memory_embeddings_backfills_missing_vectors (backfill + idempotence), lazy_recall_embedder_is_boot_safe_before_resolution (no probe until first embed). Existing memory module tests unperturbed (36/36). Complements the loud-degrade signal (b413aaa): that made the degrade visible; this removes it on the bridge path when the embedder serves. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo * fix(memory): bound + fail-fast the candidate-embedding backfill so a down embedder can't hang the SessionStart recall The on-demand backfill (prior commit) embedded EVERY missing candidate synchronously inside multi_layer_recall. Two hazards that surfaced dogfooding live against a 375-memory corpus with the in-process Qwen3 embedder returning degenerate (zero) vectors under GPU-OOM: 1. Down embedder → the backfill grinds all N candidates through a failing GPU forward pass, hanging the recall (the recall-hook command timed out). Now: FAIL_FAST — if the first few embeds all come back empty, the embedder is down; bail immediately. The caller's loud-degrade signal still reports the resulting non-semantic recall. 2. Working embedder but large cold corpus → one recall blocked on hundreds of embeds. Now: MAX_PER_CALL=64 caps per-recall work; a big corpus embeds incrementally across several recalls (amortized). Content is immutable, so partial progress never goes stale. Test: ensure_memory_embeddings_fails_fast_when_embedder_down (counts embed calls, asserts the backfill bails after ~3 empties instead of attempting all 20). Existing backfill + idempotence test still green. Note (separate, not this commit): the underlying degenerate embeddings are the in-process Qwen3-Embedding-0.6B GGUF returning all-zero vectors on real content — the embed backend (backends/llamacpp.rs) is correct (fresh embedding-mode context, KV cleared per text, PoolingType::Last, 2048 clamp), so the zeros come from the Metal forward pass failing under VRAM pressure (the boot-time GPU-OOM, #175 family). Recall goes fully semantic once the embedder yields real vectors; the wiring + backfill are proven live via the `recall.embedder.resolved source=in-process-llamacpp-global` probe. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo * fix(inference): govern the embedding VRAM lane — lease before allocating, fail loud instead of decoding to degenerate zeros The embedding forward pass allocated a ~1.5 GiB Metal context (2048-ctx compute + KV) UNGOVERNED — grabbing VRAM behind the ResourceGovernor's back and competing with the serving lane. Under pressure the Metal command buffer OOM'd and the decode returned an ALL-ZERO vector (the "degenerate embedding" that silently broke semantic recall for the agent-memory bridge). It's the last blocker on the recall-wiring fix. Make the embed lane a governed GPU consumer, mirroring cognition/eval.rs::acquire_eval_lane_slot exactly (no parallel allocator — concurrency guide #6): create_embedding now acquires a Pinned VRAM LeaseRequest from ResourceDaemon::global() before the spawn_blocking forward pass, holding the RAII LeaseGuard for the duration (freed on drop). Granted ⇒ the bytes are reserved so the decode has room to succeed; InsufficientCapacity ⇒ fail LOUD with a named error (probe embed.vram.refused) instead of allocating a doomed context that emits garbage. Embeddings are GPU-only ([[gpu-is-non-negotiable...no-cpu-fallback]]) so there is NO CPU spill like eval — the honest degrade is a refusal the caller surfaces as "no signal" (and the recall backfill's fail-fast handles cleanly), never a zero vector. Ungoverned node (no daemon) = behavior unchanged. Probes: embed.vram.leased / embed.vram.refused. Const, in-code policy (not env). Scope = per-call CONTEXT admission control (the OOM fix); registering the embed model's resident WEIGHTS as a ResourceConsumer (footprint/reclaim, mirroring ServingConsumer) is the residency-accounting follow-up. This is step 1 of the GPU-accelerated / grid-command / cache-layer embedding arc. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo * feat(memory): persistent embedding cache — embed once EVER, warmed on boot for every persona + agent The process-global embedding cache (global_embedding_cache) was in-memory only, so it died on every restart and all content re-embedded — which, with the governed embed lane, means re-fighting the serving lane for VRAM (155 lease refusals observed on a warm 375-memory corpus). Persist it. EmbeddingCache gains snapshot_to/load_from: a compact, dep-free, atomic binary snapshot ([u64 count] then per entry [u64 key][u32 dim][dim × f32], all LE; write-temp-then-rename). spawn_embedding_cache_persistence warms the cache from the snapshot at boot, then snapshots every 60s on its OWN tokio task (RTOS shape: own task + tokio::time::interval, file write on spawn_blocking, off every hot path — concurrency guide). Best-effort throughout: a lost snapshot just re-embeds; probes embedding.cache.loaded / embedding.cache.flushed. Wired once at boot (main.rs) against ~/.continuum/cache/embedding-cache.bin. Because this is the ONE cache every persona AND agent shares — their recall embedders all wrap global_embedding_cache via CachingEmbeddingProvider — a single warm-load brings back the whole citizenry's vectors at once: each unique content embeds once, EVER. Steady state → the VRAM refusals collapse to near-zero and cold-start recall is instant. Step 2 of the GPU/grid/cache embedding arc (after the governed lane). Next: grid-shared backing + ai/embedding/generate command. Test: embedding_cache_snapshot_round_trips (byte-identical restore, missing file = Ok(0)). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo * fix(cognition): dream inference must acquire a serving lane + rest between review passes — stop it starving reactive responding (personas looked comatose) SEVERE live failure (2026-07-26): 4 resident personas dreamed near-CONTINUOUSLY (dream_consolidation ran 00:00→05:03) and did NOT respond to direct chat — alive but comatose. Root cause (mapped, not guessed): 1. The dream's model call (distill_reviewing) called `self.adapter.generate_text` DIRECTLY, bypassing `acquire_serving_lane`. The turn path acquires a lane and the #139 reservation always holds one lane for a DIRECTED chat turn — but the dream slipped past it entirely, grabbing a physical llama decode lane uncounted. The semaphore then handed the chat turn a permit while the physical lane was busy dreaming, so the directed reply queued INSIDE llama behind the dream. Delivery was fine (Benchy DID receive the message + ran recall of 8 memories); the turn's model call was starved. FIX: wrap the dream's generate_text in `acquire_serving_lane(false)` (non-directed) — caps all dream inference at MAX_LANES-1 and guarantees a waiting directed turn wins the lane. The load-bearing responsiveness fix. 2. The quiet-day review-only pass had NO cooldown: it drains REVIEW_ONLY_BATCH (6) beliefs, emits CadenceHint::Sleep, but the governor re-ticks every ~30s and immediately reviews the next 6 — forever, for a large belief store (Atlas: 4,336 engrams), compounded by the in-memory `reviewed` set resetting on each dev restart. FIX: REVIEW_ONLY_COOLDOWN_MS (5min) per-persona rest gate in try_review_only — belief hygiene trickles instead of grinding. Consolidation of FRESH experience stays ungated (real learning is never throttled). Both preserve the organism (the fade/consolidation still runs) while freeing the lane so the being can respond and act. 14 dream_consolidation tests stay green (quiet-day contract holds: immediate re-tick still sleeps). Relates to #139, #221, the acting-organism arc. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
multi_layer_recall silently degrades to non-semantic order when there's no neural query embedding or no memory vectors — looks like working recall but ranking is meaningless (the trap the agent-memory bridge hit). Now logs loud (embedder id + vector count) so it can't hide. Diagnostic only; the neural-embedder wiring is the follow-up (model GGUF already on disk).
🤖 Generated with Claude Code