Chat with Sources with Ollama local model too slow to work

[chris-opendata]

I configured open notebook with Ollama local models Gemma4:12b and Qwen3.8 on my Mac M5 Pro 24GB laptop. I tested "Chat with Sources" to find generative AI usage in a 45-page research article. The 'thinking" icon animates to no end for several hours. It happened to both models. Are there any optimization tips to make it work? Thanks.

[Michael-WhiteCapData]

A multi-hour spinner is not normal generation latency; first update Open Notebook before tuning the models. Current main contains several fixes directly related to this case:

• Ollama's default num_ctx was reduced from 128k to 8192 to avoid consumer-GPU OOM/slowdown.
• num_ctx can now be set per Ollama credential.
• thinking-model output handling was fixed.
• context-length failures stop retrying instead of looping through repeated background attempts.
• the frontend request timeout is configurable.

After updating, I would test in this order:

1. In Settings → API Keys, set the Ollama credential's num_ctx to 8192 initially.
2. Use one model only and confirm it is fully resident with ollama ps. If it repeatedly loads/unloads or uses CPU heavily, the 12B model plus context is exceeding comfortable memory.
3. Exclude the full 45-page source and include its generated insights first. If that works, add full content and raise num_ctx gradually.
4. Try a non-thinking or smaller model for chat. Reasoning models can spend a large part of the output budget on thinking; they are usually a poor first diagnostic for source chat.
5. Check the API/container log while sending one request. A context-length error, repeated retry, or Ollama timeout is much more actionable than the UI spinner.

The relevant changes and configuration names are recorded in the project's CHANGELOG, including the 8192 Ollama default and per-credential num_ctx.

On 24 GB unified memory, start with the smaller of the two models, 8k context, insights-only context, and no other loaded Ollama model. Increase one dimension at a time after a successful response.

[lfnovo]

@Michael-WhiteCapData thank you, this is the answer I'd have written. I checked each point against the changelog and it holds: the 8192 default and per-credential num_ctx, the thinking-model fix, the configurable frontend timeout, and context-length errors no longer retrying (that last one is on main, not yet in a release).

@chris-opendata one more cause that fits "spins for hours" specifically: #1327. Source chat's stream sent nothing while a slow model was generating, so the connection went idle and got dropped. The answer was actually produced, but only showed up after leaving and reopening the source. If that's what you see (the reply appears after navigating away), it's that bug, and #1332 fixes it.

Combined with Michael's checklist (8k context, insights before full content, a smaller non-thinking model first), that should get you to a working baseline. I'm closing this as answered; reopen it with the API log from one request if it still hangs.

Status: answer. Tuning per the checklist above; idle-connection drop tracked in #1327 / #1332.