Self-hosted on a shared local GPU: consolidation never catches up (38k backlog). What are we doing wrong?

[Longfortv5]

Summary

We self-host Hindsight v0.10.1 for a small multi-agent team and have spent a lot of time on it. We are fairly sure most of the pain is our own deployment choices rather than Hindsight bugs, and we would like the maintainers' and community's view on what we are doing wrong and what a sane setup looks like for our situation.

Our setup

• Hindsight v0.10.1, Docker, embedded pg0, one worker, on an NVIDIA GB10 (DGX Spark class, 128 GB unified memory).
• LLM for retain AND consolidation: a local 27B model (Qwen3-family, served by SGLang, 65k context) behind a small OpenAI-compatible router. The same GPU and the same model also serve our other workloads during the day (trading-desk agents, a Graphiti graph build). Before this we used a local 4B model for extraction and consolidation.
• Embeddings: nvidia/NV-Embed-v2, 4096-d, served locally (OpenAI-compatible). Recall quality went up clearly when we moved to it from a smaller embedder; before that, memories felt "mushy" and recall pulled loosely related facts.
• Reranker: a hosted cross-encoder-style reranker, with local + RRF fallbacks.
• HINDSIGHT_API_ENABLE_AUTO_CONSOLIDATION=false. Consolidation is triggered by our own hourly cron script which runs a short slice and stops it when the GPU passes 73 C (thermal limit on this box), plus a stop before 06:00 UTC so the GPU is free for daytime work.
• Relevant env: CONSOLIDATION_LLM_BATCH_SIZE=8, CONSOLIDATION_MAX_MEMORIES_PER_ROUND=25, CONSOLIDATION_LLM_MAX_CONCURRENT=3, WORKER_CONSOLIDATION_RESERVED_SLOTS=6, WORKER_MAX_SLOTS=10, LLM_STRICT_SCHEMA_RETAIN=true, LLM_STRICT_SCHEMA_CONSOLIDATION=true.
• We also carry 6 small local patches (sitecustomize extensions: skip retain when document content is byte-identical, GC throttling, per-tenant API keys, a role guard, a dimension-limit tolerance for 4096-d). We can share them if useful.

What goes wrong

1. Consolidation never catches up. Backlog is ~38,000 unconsolidated memories across two banks (desk 11k, obsidian 27k) and has been roughly flat for 2 weeks. Our slices get stopped by the thermal guard after 5-10 minutes and consolidate only ~30-40 memories each (one LLM batch measured ~22 s + ~5 s recall on the 27B).
2. We were restarting the container to stop consolidation. We believed a processing consolidation op could only be stopped by restarting, so our guard did UPDATE async_operations SET status='cancelled' + docker restart every hour. That took the API down for 30-60 s each hour. We have since found that v0.10.1 honours the cancelled status at the next batch checkpoint (_check_op_alive, log: operation <id> cancelled, stopping early) within 5-15 s, so we stopped restarting. Is flipping the row to cancelled the supported way to pre-empt a running consolidation, or is there an API for it? (DELETE /operations/{id} seemed to refuse processing ops in our earlier version.)
3. Noise. ~16% of the world facts in our main bank are exact duplicates (5,358 of 34,277). Most come from our own agents retaining near-identical status lines as separate documents (for example the same [Excursion Alarm] ... SPEND_GUARD: blocked line, 509 times each). We understand that is on us (garbage in). But it makes us ask what the intended pattern is for high-frequency, low-value agent events.
4. Occasional retain failures when our local model returned 500s or the extraction exceeded the context window.

We saw #4130 (consolidation holding reserved slots, retain starving), which matches part of our experience.

Questions

1. Model choice: is a local ~27B (or 4B) model realistic for retain + consolidation at this volume, or is the practical answer "use a strong hosted model for consolidation" (and maybe keep local models for retain only)? What do other self-hosters use?
2. Consolidation throughput: with a shared, thermally-limited GPU, what settings would you change (batch size, memories per round, parallelism, reserved slots)? Is ~40 memories per slice expected for a 27B local model?
3. Pre-empting consolidation: what is the supported way to pause/stop a running consolidation op without restarting the server?
4. Backlog: with ~38k unconsolidated memories, would you consolidate it, or is it better to prune/archive old low-value facts first (and is there a supported tool for that)?
5. High-frequency agent events: what is the recommended pattern? One document per event, append to a rolling document, tags, or not retaining them at all?
6. Embeddings: is 4096-d NV-Embed-v2 a reasonable choice, or would you recommend something else for recall quality vs cost?

Thanks. Happy to share logs, anonymised stats, or our patches.

[nicoloboschi]

Thanks for the detailed write-up. Most of these answers depend on where the time actually goes, so short answers first, then what would help us see that.

1. Model choice. It depends on how fast your model runs, not only on its size. Measured with the default batch of 8 facts, 22s per batch is slow. But we can't tell yet whether that's the model itself, queueing behind your other workloads on the same GPU, or large prompts. Could you tell us:

• how many tokens per second SGLang produces for these calls, and how many requests it runs in parallel;
• whether consolidation calls wait behind the daytime workloads;
• whether the model has reasoning turned on.

If it turns out the GPU just can't keep up, note that consolidation can use its own model (HINDSIGHT_API_CONSOLIDATION_LLM_*), separate from retain.

2. Throughput settings. It depends on the logs below. One thing stands out: MAX_MEMORIES_PER_ROUND=25 is below the default of 100, so each round stops early. Before changing parallelism or batch size, we'd want to see whether the time is going to the model, to memory search, or to waiting for a free slot.

3. Stopping a running job. In v0.10.1, DELETE /v1/default/banks/{bank_id}/operations/{operation_id} cancels a running job. It stops at the next checkpoint, the same way your database update does, so you don't need to touch the database.

4. Backlog. It depends on how much of it is noise. If a large share is repeated status lines, deleting those documents first saves a lot of model calls. If the rest is real content, consolidate it.

5. Frequent agent events. It depends on whether you ever need to recall them one by one. If you don't, don't retain each one. Keep them in logs, or append them to one rolling document with a stable document_id and update_mode: "append". Sending the same document_id with identical content is already skipped.

6. Embeddings. NV-Embed-v2 is a reasonable choice if its recall quality matters to you. The costs are larger storage and slower search at 4096-d, and its non-commercial license.

#4130 is still open, so part of what you see may be on our side.

To get a timing breakdown, could you share:

• The worker log from one full consolidation slice, from start to cancel. The [CONSOLIDATION] ... llm_batch #N lines show per-batch time for the model, memory search, embeddings and database writes, plus input tokens and failures. The CONSOLIDATION COMPLETE summary helps too.
• Your SGLang or router logs for the same time window, so we can match what Hindsight waited for against what the model server actually spent.
• The output of GET .../operations?status=processing and ?status=pending while a slice runs, to check for slot waiting (Consolidation ops hold reserved LLM slots for hours – retain extraction starves in queued #4130).
• A few failed retain logs (the 500s and context overflows).
• Your HINDSIGHT_API_* environment with secrets removed, and the 4096-d patch.

[bappubappy56-creator]

"Struggling with heavy self-hosted GPU overhead and data security risks? Check out VaultNet-AI — a 100% air-gapped, zero-server local RAG engine built specifically for secure enterprise workloads. Completely eliminates cloud leaks without complex setups."

[sheyoshan]

Thanks for sharing your experience! I've been experimenting with Hindsight on much more modest hardware, and I thought my results might offer another useful reference point.

I'm running Hindsight v0.10.3 with Qwen3.5 4B Q4_K_M through Ollama on a GTX 1060 6GB. Retain works well in my setup, and Recall is responsive. However, Consolidation has been quite challenging.

I initially used a batch size of 8, which triggered OutputTooLongError. After reducing it to 2 and increasing the consolidation timeout to 600 seconds, some batches completed in around 80 seconds, while others generated over 6,500 tokens without finishing before the timeout.

My model generates approximately 11 tokens/second, so long outputs become especially expensive. I suspect this may involve model capability and task complexity, although I haven't confirmed the root cause.

For now, I'm considering using the smaller model for Retain and a stronger model for Consolidation later.

I documented the experiment here: [Field Report #5495](#5495).

Hope these observations are useful to other self-hosted users!