fix(memory): semantic recall for the agent-memory bridge — lazy neural embedder + governed embed VRAM lane - #2034
Merged
Merged
Conversation
…d candidate-vector backfill — semantic recall for the agent-memory bridge The agent-memory bridge (memory/remember, memory/recall-hook) recalls through the GLOBAL PersonaMemoryManager, which main.rs constructed with the hardcoded LEXICAL bootstrap embedder — deliberately, because NOTHING may gate the IPC socket bind on a GPU/gateway probe (concurrency guide). The per-persona path resolved neural on spawn; the global/bridge path never did. Result, dogfooded live: recall returned the SAME off-topic memories for every query — non-semantic order that ignores the query text. Two gaps, both closed here, reusing the existing resolver + storage (no new machinery): 1. LazyRecallEmbedder (cognition/embedding.rs) — resolves the dedicated in-process Qwen3-Embedding-0.6B on the FIRST recall via new resolve_recall_embedder_local() (paths: in-process GGUF → lexical floor; no chat adapter, which the global manager has none of), caches it process-stably, delegates. Zero boot-path cost — the probe + calibration is paid once on first real recall, never at socket-bind. This is the "separate addressing follow-up" main.rs's old comment promised. 2. ensure_memory_embeddings (memory/mod.rs) — agent memories are written with embedding: None, so corpus.memories_with_embeddings() was empty and SemanticRecallLayer no-op'd even with a neural query vector. multi_layer_recall now backfills missing candidate vectors on demand (async, cached, outside any lock) before the sync recall, so the semantic layer has something to rank. First recall per persona pays it; content is immutable so vectors never go stale; idempotent once populated. Tests: ensure_memory_embeddings_backfills_missing_vectors (backfill + idempotence), lazy_recall_embedder_is_boot_safe_before_resolution (no probe until first embed). Existing memory module tests unperturbed (36/36). Complements the loud-degrade signal (b413aaa): that made the degrade visible; this removes it on the bridge path when the embedder serves. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…down embedder can't hang the SessionStart recall The on-demand backfill (prior commit) embedded EVERY missing candidate synchronously inside multi_layer_recall. Two hazards that surfaced dogfooding live against a 375-memory corpus with the in-process Qwen3 embedder returning degenerate (zero) vectors under GPU-OOM: 1. Down embedder → the backfill grinds all N candidates through a failing GPU forward pass, hanging the recall (the recall-hook command timed out). Now: FAIL_FAST — if the first few embeds all come back empty, the embedder is down; bail immediately. The caller's loud-degrade signal still reports the resulting non-semantic recall. 2. Working embedder but large cold corpus → one recall blocked on hundreds of embeds. Now: MAX_PER_CALL=64 caps per-recall work; a big corpus embeds incrementally across several recalls (amortized). Content is immutable, so partial progress never goes stale. Test: ensure_memory_embeddings_fails_fast_when_embedder_down (counts embed calls, asserts the backfill bails after ~3 empties instead of attempting all 20). Existing backfill + idempotence test still green. Note (separate, not this commit): the underlying degenerate embeddings are the in-process Qwen3-Embedding-0.6B GGUF returning all-zero vectors on real content — the embed backend (backends/llamacpp.rs) is correct (fresh embedding-mode context, KV cleared per text, PoolingType::Last, 2048 clamp), so the zeros come from the Metal forward pass failing under VRAM pressure (the boot-time GPU-OOM, #175 family). Recall goes fully semantic once the embedder yields real vectors; the wiring + backfill are proven live via the `recall.embedder.resolved source=in-process-llamacpp-global` probe. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…ing, fail loud instead of decoding to degenerate zeros The embedding forward pass allocated a ~1.5 GiB Metal context (2048-ctx compute + KV) UNGOVERNED — grabbing VRAM behind the ResourceGovernor's back and competing with the serving lane. Under pressure the Metal command buffer OOM'd and the decode returned an ALL-ZERO vector (the "degenerate embedding" that silently broke semantic recall for the agent-memory bridge). It's the last blocker on the recall-wiring fix. Make the embed lane a governed GPU consumer, mirroring cognition/eval.rs::acquire_eval_lane_slot exactly (no parallel allocator — concurrency guide #6): create_embedding now acquires a Pinned VRAM LeaseRequest from ResourceDaemon::global() before the spawn_blocking forward pass, holding the RAII LeaseGuard for the duration (freed on drop). Granted ⇒ the bytes are reserved so the decode has room to succeed; InsufficientCapacity ⇒ fail LOUD with a named error (probe embed.vram.refused) instead of allocating a doomed context that emits garbage. Embeddings are GPU-only ([[gpu-is-non-negotiable...no-cpu-fallback]]) so there is NO CPU spill like eval — the honest degrade is a refusal the caller surfaces as "no signal" (and the recall backfill's fail-fast handles cleanly), never a zero vector. Ungoverned node (no daemon) = behavior unchanged. Probes: embed.vram.leased / embed.vram.refused. Const, in-code policy (not env). Scope = per-call CONTEXT admission control (the OOM fix); registering the embed model's resident WEIGHTS as a ResourceConsumer (footprint/reclaim, mirroring ServingConsumer) is the residency-accounting follow-up. This is step 1 of the GPU-accelerated / grid-command / cache-layer embedding arc. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
… boot for every persona + agent The process-global embedding cache (global_embedding_cache) was in-memory only, so it died on every restart and all content re-embedded — which, with the governed embed lane, means re-fighting the serving lane for VRAM (155 lease refusals observed on a warm 375-memory corpus). Persist it. EmbeddingCache gains snapshot_to/load_from: a compact, dep-free, atomic binary snapshot ([u64 count] then per entry [u64 key][u32 dim][dim × f32], all LE; write-temp-then-rename). spawn_embedding_cache_persistence warms the cache from the snapshot at boot, then snapshots every 60s on its OWN tokio task (RTOS shape: own task + tokio::time::interval, file write on spawn_blocking, off every hot path — concurrency guide). Best-effort throughout: a lost snapshot just re-embeds; probes embedding.cache.loaded / embedding.cache.flushed. Wired once at boot (main.rs) against ~/.continuum/cache/embedding-cache.bin. Because this is the ONE cache every persona AND agent shares — their recall embedders all wrap global_embedding_cache via CachingEmbeddingProvider — a single warm-load brings back the whole citizenry's vectors at once: each unique content embeds once, EVER. Steady state → the VRAM refusals collapse to near-zero and cold-start recall is instant. Step 2 of the GPU/grid/cache embedding arc (after the governed lane). Next: grid-shared backing + ai/embedding/generate command. Test: embedding_cache_snapshot_round_trips (byte-identical restore, missing file = Ok(0)). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…tween review passes — stop it starving reactive responding (personas looked comatose) SEVERE live failure (2026-07-26): 4 resident personas dreamed near-CONTINUOUSLY (dream_consolidation ran 00:00→05:03) and did NOT respond to direct chat — alive but comatose. Root cause (mapped, not guessed): 1. The dream's model call (distill_reviewing) called `self.adapter.generate_text` DIRECTLY, bypassing `acquire_serving_lane`. The turn path acquires a lane and the #139 reservation always holds one lane for a DIRECTED chat turn — but the dream slipped past it entirely, grabbing a physical llama decode lane uncounted. The semaphore then handed the chat turn a permit while the physical lane was busy dreaming, so the directed reply queued INSIDE llama behind the dream. Delivery was fine (Benchy DID receive the message + ran recall of 8 memories); the turn's model call was starved. FIX: wrap the dream's generate_text in `acquire_serving_lane(false)` (non-directed) — caps all dream inference at MAX_LANES-1 and guarantees a waiting directed turn wins the lane. The load-bearing responsiveness fix. 2. The quiet-day review-only pass had NO cooldown: it drains REVIEW_ONLY_BATCH (6) beliefs, emits CadenceHint::Sleep, but the governor re-ticks every ~30s and immediately reviews the next 6 — forever, for a large belief store (Atlas: 4,336 engrams), compounded by the in-memory `reviewed` set resetting on each dev restart. FIX: REVIEW_ONLY_COOLDOWN_MS (5min) per-persona rest gate in try_review_only — belief hygiene trickles instead of grinding. Consolidation of FRESH experience stays ungated (real learning is never throttled). Both preserve the organism (the fade/consolidation still runs) while freeing the lane so the being can respond and act. 14 dream_consolidation tests stay green (quiet-day contract holds: immediate re-tick still sleeps). Relates to #139, #221, the acting-organism arc. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Give the agent-memory bridge real semantic recall
Dogfooding surfaced that agent-memory recall was non-semantic — different queries returned the same off-topic memories. Root cause was three-layered and this branch fixes all of it, ending with the real culprit: ungoverned embed VRAM, not missing wiring.
The chain (5 commits)
multi_layer_recalllogs LOUD when it falls back to non-semantic (embedder id + memories-with-vectors count). Stops the failure hiding.ensure_memory_embeddingsbackfills missing candidate vectors on demand (agent memories are writtenembedding: None).MAX_PER_CALL+ 3-empties-bail so a down embedder can't hang the SessionStart recall.create_embeddingnow acquires a Pinned VRAM lease fromResourceDaemon::global()before the forward pass (mirrors the eval-lane slot). This fixed the degenerate all-zero vectors: they were Metal VRAM contention with the serving lane (Serialization bug in decision pipeline: agentCoordination null values #175 family), not a bad embed path. Refused ⇒ FAIL LOUD, never a doomed zero-vector context.Proof (live)
Query "GPU Metal VRAM out of memory serving crash" now returns on-topic serving/GPU memories, and different queries return different results — no longer the same 3 for everything.
embed.vram.leased=391,embed.vram.refused=155 (honest refusals under pressure),degenerate-after-deploy=0.This is the substrate every persona's recall rides on, and mine (the agent-memory bridge) — semantic recall for the whole being, governed so it can't decode to garbage under VRAM pressure.
Follow-ups (own arc): register embed-model weights as a ResourceConsumer (#225, shrinks the 155 refusals);
ai/embedding/generategrid command.🤖 Generated with Claude Code