Field report: Hindsight v0.10.3 with local Qwen3.5 4B Q4_K_M — Retain/Recall work well, but Consolidation is inconsistent and can time out

[sheyoshan]

Type: Community field report / model-and-hardware compatibility discussion, not a confirmed Hindsight bug report.
Test date: October 11, 2026 (deployment baseline from October 10).
Privacy: Prompt bodies, extracted personal facts, observation text, user identifiers, and private network addresses have been omitted.

TL;DR

I spent roughly 6–7 hours investigating Hindsight with a local Qwen3.5 4B Q4_K_M model served by Ollama on modest hardware. Retain works well in my usage, and Recall is responsive. The main difficulty has been Consolidation: some 2-fact batches complete in ~80 seconds and generate a concise, valid JSON response, while other requests generate thousands of tokens without finishing before a 600-second timeout. This can trigger automatic retries and very long total batch times.

My working hypothesis is a task/model/hardware mismatch: a 4B-class quantized model sometimes struggles to terminate complex, long-context, schema-constrained consolidation reliably; at ~11 generated tokens/second, an overlong completion readily becomes a timeout. This is not a proven root cause. In particular, failed requests had no returned output, so I could not examine the unfinished JSON or establish that repetition was occurring.

Importantly, this does not mean Hindsight is unusable. I plan to continue accumulating Facts and graph links through Retain/Recall and revisit Consolidation/Reflect/Mental Models when a stronger model or faster hardware is available.

Environment

Component	Configuration
Hindsight	v0.10.3, full Docker image, embedded PostgreSQL (pg0)
Hindsight host	ASUS VivoBook A515EA, Windows, 16 GB RAM, Docker Desktop / WSL 2
Inference host (separate machine)	Windows 10, Intel Core i7-8700, 8 GB system RAM, NVIDIA GTX 1060 6 GB VRAM
Inference runtime	Ollama (reported 0.34.4)
LLM	qwen3.5:4b-q4_K_M (approximately 4.7B parameters; model download ~3.39 GB)
Model context	~32,768 tokens configured/observed
Connectivity	Hindsight -> remote Ollama over a private Tailscale network
Chat integration	Open WebUI plus a third-party Hindsight Memory for Open WebUI Filter
Memory storage	Persistent named Docker volume; embeddings/reranking on the Hindsight host
Measured generation	Approximately 10.8–11.5 tokens/s in relevant Ollama logs

The Hindsight API and Control Plane are bound to loopback on the VivoBook (127.0.0.1:8888 and 127.0.0.1:9999). Ollama is not publicly exposed for this experiment.

What succeeded

• Hindsight deployment, API health, embedded database, named-volume persistence setup, and connectivity to Ollama.
• A basic document Retain test and subsequent Recall test.
• Open WebUI -> Hindsight automatic Retain, with per-user memory banks. I left the Filter's Recall/Reflect features disabled while collecting data.
• On this workload, Hindsight's Fact storage and graph/link construction remained useful even without fully completed Consolidation.
• Some individual Consolidation requests succeeded and emitted concise structured JSON with the expected creates / updates / deletes arrays.

These are observations from my specific workload, not a benchmark of accuracy against a labeled test set. I have not independently benchmarked full Reflect or Mental Model quality.

Consolidation issue and configuration changes

Initial configuration (relevant fields):

HINDSIGHT_API_LLM_PROVIDER=ollama
HINDSIGHT_API_LLM_MODEL=qwen3.5:4b-q4_K_M
HINDSIGHT_API_LLM_MAX_CONCURRENT=1
HINDSIGHT_API_LLM_TIMEOUT=300
HINDSIGHT_API_ENABLE_OBSERVATIONS=true
Consolidation LLM Batch Size (bank setting): 8 (initial/default)

During consolidation, the worker reported OutputTooLongError for an 8-fact batch and bisected it into 4+4. Other attempts repeatedly encountered Ollama connection error and timeout/retry behavior. I confirmed that the worker was still processing (not simply an idle database or a dead HTTP server).

I then:

1. Changed the bank-level Consolidation LLM Batch Size from 8 to 2.
2. Set HINDSIGHT_API_CONSOLIDATION_LLM_TIMEOUT=600, while keeping the general HINDSIGHT_API_LLM_TIMEOUT=300.
3. Before recreating the Docker container, made a Hindsight admin backup and verified the archive inventory (23 tables / 15,348 records). I retained the persistent named volume and previous container as a fallback. Restore from that archive was not tested.
4. Recreated the service with the same persistent volume, ports and network. The initial health check briefly returned an empty reply during startup; subsequent checks confirmed service recovery.
5. Cancelled the old consolidation operation and started a new one from the Control Plane so the new bank-level batch-size setting could take effect. A cancelled operation can leave already committed batches intact.

Important: Changing Batch Size while a job was already running did not retroactively change that job's effective batching. Some logged 2-fact units in the old operation came from adaptive bisection; they are not proof that the whole old job was configured as Batch Size = 2.

Selected measurements from the logs

These are different requests / task stages, not a controlled same-prompt A/B comparison:

Observed case	Result
Initial 8-fact batch	OutputTooLongError; split into smaller sub-batches
Older batch after adaptive splitting	2 memories; 3 LLM calls; ~7,324.6 s total LLM time; ultimately updated 2 observations
New operation, batch #1	2 memories; 1 LLM call; ~79.7 s LLM time; updated 2 observations
Successful Ollama-side request around that time	~20,340 prompt tokens; ~299 evaluated output tokens; ~78.3 s end-to-end; HTTP 200
Problematic Ollama-side generation	Approximately 6,500+ generated tokens at ~10.9 tokens/s; failed around the 600 s request limit; no completed response body to inspect

The Hindsight per-batch approximate input-token counter and Ollama's prompt-evaluation count are not necessarily identical; they should not be treated as interchangeable measurements. Similarly, one successfully handled 2-fact batch does not prove the remaining batches would be stable.

Why the generation length matters on this hardware

At ~11 generated tokens/s, a 300-token reply takes roughly half a minute to generate (plus prompt processing), whereas 6,500 tokens require close to ten minutes of generation alone. The slow hardware magnifies an overlong output into a timeout; it does not by itself explain why the model generated so much.

I also checked the Hindsight v0.10.3 Ollama native provider implementation: its /api/chat payload sets "stream": false and defaults to "think": false unless overridden. Therefore, I would not assume those 6,500 tokens were a hidden thinking trace. The most plausible unverified explanations include excessively long structured fields, too many update decisions, or generation degeneration. We cannot distinguish these without the failed completion text.

Bank snapshot after the tests

The Control Plane showed:

Total memories: 488
Documents:      25
World facts:   439
Experience:      2
Observations:   47
Links:       ~13,000
  Temporal:   ~5,200
  Semantic:   ~7,600
  Entity:        202
Consolidation dashboard: Done 278 / Pending 210 / Failed 0

Dashboard interpretation: In v0.10.3, the UI computes Done = total_nodes - pending - failed, so the 278 “Done” figure includes the 47 Observations. That corresponds to 231 consolidated source Facts out of 441 World + Experience Facts (~52.4%). A displayed Failed = 0 means no Facts are currently permanently marked failed; it does not mean that no earlier LLM attempts timed out.

The link count confirms that graph-building took place, not that every link is relevant or Recall quality is proven.

Working hypothesis, and limitations

Hypothesis: Qwen3.5 4B Q4_K_M can carry out Hindsight's smaller Retain tasks and support a useful Fact/Recall workflow, but is not consistently reliable for long-context Consolidation on this setup. Hindsight's consolidation prompt asks it to compare newly extracted facts against existing observations, preserve source UUIDs, choose CREATE vs UPDATE vs DELETE, and produce schema-constrained JSON. My successful runs show it can do that; the extended generation and timeout runs show that it does not always complete within the available inference budget.

What I cannot conclude from this experiment:

• That Hindsight itself is broken, or that every Qwen3.5 4B deployment behaves the same way.
• That a specific output field was looping: failed calls returned no output to inspect.
• That the 4B parameter count is the only cause: sampling defaults, prompt content, model/server version, and Ollama/structured-output behavior may all matter.
• That upgrading the GPU alone fixes model-output stability (it may only reduce elapsed time).
• That Reflect or Mental Models fail in the same way: those functions were not independently characterized to the same depth.

Suggestions for other self-hosted users

• Separate functional success from end-to-end reliability. A few successful JSON calls do not establish stable consolidation of an entire bank.
• Track per-request latency and output-token counts, not only HTTP status; keep the Control Plane's LLM Requests trace enabled when privacy requirements allow. Failed non-streaming requests may have no Output field.
• Try a smaller Consolidation LLM Batch Size (e.g. 2) and consider a dedicated consolidation timeout, but don't mistake these for a fix to model output instability. A live operation may retain its old batch parameters until restarted.
• Back up before changing containers or consolidation settings; persist the pg0 volume; distinguish Docker container lifecycle from data lifecycle. Do not publish .env contents or actual memory-bank data.
• Evaluate stronger models separately for Consolidation/Reflect, while keeping economical models for Retain when they work well.
• Keep Facts. Pending consolidation is not the same as lost data. Note that previously consolidated Facts may need an explicit reprocessing strategy if a better model is used later.

Questions for Hindsight maintainers and other users

1. Do you have recommended minimal model capability tests specifically for Consolidation (beyond a basic JSON-schema smoke test)?
2. Have others seen very long, non-terminating structured completions from Qwen3.5 4B/9B or other local models on consolidation prompts?
3. Is there a supported way to capture partial output safely for timed-out native Ollama calls, or to detect pathological length/repetition before the full timeout?
4. What is the recommended low-resource architecture for retaining Facts now and consolidating them later with a separate, more capable model?

Related discussions/reports: Hindsight #2544 — degenerate consolidation output, Hindsight #2150 — long/stuck consolidation LLM calls, and Ollama #16685 — Qwen3.5 9B non-terminating long output. These are analogous reports, not proof of the same root cause.

Bottom line: This experiment was still worthwhile. Retain + Facts + Recall already provide value on inexpensive hardware. I view Consolidation/Reflect/Mental Models as separately upgradable layers—not a reason to discard the memory bank or the Hindsight project.

[IlyaKhalafi]

For the low-resource split, v0.10.3 supports separate consolidation settings. Keep Retain on the working model and set HINDSIGHT_API_CONSOLIDATION_LLM_PROVIDER, HINDSIGHT_API_CONSOLIDATION_LLM_MODEL, HINDSIGHT_API_CONSOLIDATION_LLM_BASE_URL and HINDSIGHT_API_CONSOLIDATION_LLM_API_KEY together. These select the server-level endpoint, which receives bank content. With HINDSIGHT_API_ENABLE_AUTO_CONSOLIDATION=false, trigger consolidation manually after selecting that model. Check the bank's auto-consolidation override. Stop the old operation and restart the server after changing environment settings before evaluating the new configuration. See the versioned configuration guide.

HINDSIGHT_API_CONSOLIDATION_MAX_COMPLETION_TOKENS bounds generated tokens. CONSOLIDATION_MAX_TOKENS controls recall context instead. The native reader maps the completion cap to num_predict and raises OutputTooLongError when the cap is reached. The consolidator splits failed batches down to individual facts, marking remaining single-fact failures. A tight cap can therefore leave facts marked failed.

Changing models does not reprocess consolidated or failed facts. The pending query excludes both. I have not verified a safe reset/reprocessing workflow.

For partial-output diagnosis, the native reader expects one complete JSON response, with no partial-output capture found in that path. Overriding stream=true alone would break that parser. After stopping competing consolidation work, replay the traced request and resolved settings against Ollama's streaming API, changing only streaming. Preserve its schema, context window, options and thinking setting. Capture chunks without applying its decisions to your bank. Use synthetic or redacted facts, noting that redaction can change the failure.

For a capability gate, repeat fixed new-fact, update, duplicate and contradiction cases. Check source UUIDs and unsupported deletions alongside JSON validity, completion rate and latency.