Please add OpenAI compatible support

[arsaboo]

...so it can be used with other local providers such as llama.cpp.

[aladin7]

I'd like to work on this. I'll keep Ollama as the default and add an opt-in OpenAI-compatible endpoint path suitable for local providers such as llama.cpp, with focused mocked tests for request and tool-call handling. I'll first verify the existing settings flow and keep the patch narrowly scoped.

[benmaster82]

Thanks for jumping on this so quickly, @aladin7!
The approach sounds spot on-keeping
Ollama as the default while adding a clean, opt-in OpenAl-compatible endpoint is exactly the way to go. Love the focus on keeping it narrowly scoped and including mocked tests.
I see you've already linked a PR. I'll head over there to review the code. Thanks again for the awesome contribution!

[arsaboo]

Trying to configure llama.cpp but the models are not detected:

[benmaster82]

Hi @arsaboo, good timing! Your screenshot shows the v1.3.2 UI, which predates this feature: the "Ollama URL" field there only speaks the Ollama API (/api/tags), which is why a llama.cpp URL shows no models. v1.4.0 shipped about 20 minutes after your comment and includes the full OpenAI-compatible provider support: https://github.com/benmaster82/writher/releases/latest

After updating: Settings → Assistant provider → "OpenAI-compatible", then set the Assistant URL to http://192.168.2.162:8085/v1, models are discovered via /v1/models and the first one is selected automatically. One note for the voice assistant's function calling with llama.cpp: start the server with the --jinja flag and a chat template that supports tools, otherwise transcription-to-action won't trigger. Let me know how it goes!

[arsaboo]

OK, that worked. But it stays stuck at "Loading speech model..."

Here's what I see in the logs:

2026-07-19 11:33:18 [INFO] Model 'small' cached: True
2026-07-19 11:33:18 [INFO] Loading Whisper model 'small'...
2026-07-19 11:33:33 [INFO] Ollama URL set to: http://192.168.2.162:8085/v1
2026-07-19 11:34:05 [INFO] Quitting...
2026-07-19 11:34:05 [INFO] Shutdown complete.
2026-07-19 11:34:05 [INFO] Process exit.
2026-07-19 11:34:58 [INFO] Model 'small' cached: True
2026-07-19 11:34:58 [INFO] Loading Whisper model 'small'...
2026-07-19 12:03:12 [INFO] Microphone list refreshed.
2026-07-19 12:03:27 [INFO] Whisper model set to: base
2026-07-19 12:03:30 [INFO] Whisper model set to: small
2026-07-19 12:03:36 [INFO] Quitting...
2026-07-19 12:03:36 [INFO] Shutdown complete.
2026-07-19 12:03:36 [INFO] Process exit.
2026-07-19 12:10:08 [INFO] Model 'small' cached: True
2026-07-19 12:10:08 [INFO] Loading Whisper model 'small'...
2026-07-19 12:10:22 [INFO] Assistant provider set to: openai
2026-07-19 12:10:26 [WARNING] Could not fetch assistant models: HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /v1/models (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x0000018FFF785BD0>: Failed to establish a new connection: [WinError 10061] No connection could be made because the target machine actively refused it'))
2026-07-19 12:10:37 [INFO] Assistant URL set to: http://192.168.2.162:8085/v1
2026-07-19 12:10:44 [INFO] Assistant model set to: unsloth/Qwen3.6-35B-A3B-GGUF:NOTHINK
2026-07-19 12:12:24 [INFO] Assistant model set to: unsloth/Qwen3.6-27B-GGUF:Q6_K_XL
2026-07-19 12:12:28 [INFO] Recording mode set to toggle
2026-07-19 12:12:29 [INFO] Recording mode set to hold

[benmaster82]

Thanks for the follow-up, this one led us to two real bugs, both fixed in v1.4.1, just released: https://github.com/benmaster82/writher/releases/latest

What was happening: if the very first model download gets interrupted (closing the app, connection drop), it leaves a partial cache ,WritHer then said "Loading speech model…" while it was actually re-downloading in the background, which looks stuck. On top of that, even with a complete model on disk, the loader contacted huggingface.co to revalidate it, so a slow or blocked connection could hang startup indefinitely.

In v1.4.1:

• the cache is properly validated (partial downloads are detected and reported as "Downloading…")
• a fully cached model now loads with zero network access startup is also much faster (~1s model load)

Please update to v1.4.1 and relaunch. If it still hangs after upgrading, delete this folder and relaunch to force a clean download:
%USERPROFILE%\.cache\huggingface\hub\models--Systran--faster-whisper-<size>

Let me know if v1.4.1 sorts it out and I'll close this one. Thanks again for the reports, both of today's fixes exist because of them! 🙏

[arsaboo]

Now it stays stuck at Downloading speech model...

I deleted the folders from %USERPROFILE%\.cache\huggingface\hub\models--Systran--faster-whisper-<size> and it tries to download them on load, but it doesn't look like the model files are downloading.

[benmaster82]

Quick follow-up that may be relevant to your "stuck at Loading speech model" report:
today we caught Hugging Face's CDN throttling a download from ~9 MB/s to ~6 KB/s on the
same connection within minutes - which makes the first-launch download look frozen. So on
top of the partial-cache bug fixed in v1.4.1, you may simply have hit a bad CDN moment.

If it ever looks stuck again: quit WritHer from the tray and relaunch - the download
resumes on a fresh connection at full speed. There is also a verified manual download
procedure in the README Troubleshooting section now, which works even where
huggingface.co is blocked or slow.

If the model download stays slow or stuck, you can install the model manually no Python needed, just PowerShell (built into Windows):

1. Quit WritHer from the tray
2. Open PowerShell and paste this (change $size if you use a different model):

$size = "small"   # tiny | base | small | medium | large-v3
$dir = "$env:USERPROFILE\.cache\huggingface\hub\models--Systran--faster-whisper-$size"
New-Item -ItemType Directory -Force "$dir\snapshots\manual" | Out-Null
New-Item -ItemType Directory -Force "$dir\refs" | Out-Null
Set-Content -Path "$dir\refs\main" -Value "manual" -NoNewline -Encoding ascii
foreach ($f in "model.bin","config.json","tokenizer.json","vocabulary.txt") {
  Write-Host "downloading $f..."
  curl.exe -L -o "$dir\snapshots\manual\$f" "https://huggingface.co/Systran/faster-whisper-$size/resolve/main/$f"
}

3. Relaunch WritHer it will say "Loading speech model…" and start in a couple of seconds, with zero network access.

The procedure is also documented in the README Troubleshooting section. If even these downloads crawl, it's the Hugging Face CDN having a moment the same command will fly an hour later, and curl shows you the live speed so you can tell.

[arsaboo]

Those PowerShell commands worked. I think there is some issue with the built-in download mechanism. Thanks for clarifying this. I will test it out and report any further issues I encounter.