[FEATURE] Windows: inference device setting (Auto/CPU) with automatic CPU fallback on CUDA failure

[nconder]

Problem:
Users currently cannot run FluidVoice Windows on older NVIDIA GPUs. The bundled parakeet.cpp/ggml CUDA backend initializes unconditionally, and when kernel launch fails (e.g. "the provided PTX was compiled with an unsupported toolchain" on a Maxwell Quadro M2000M with driver 555.85), ggml aborts the entire process (0xc0000409). The app crashes on every launch with no way to select CPU inference instead.

There is also no way to deliberately choose CPU inference — useful on VRAM-limited cards (4 GB) where users want to keep the GPU free for other local AI workloads.

Full logs and root cause analysis in #585

Proposed solution:
Two parts, one setting:

1. Manual device selection — a setting in the app (Auto / GPU / CPU) that controls whether the ggml backend initializes CUDA at all. Persisted, applied at startup, and survives in-app updates and relaunches.

2. Automatic CPU fallback — when set to Auto (default), catch CUDA initialization and kernel-launch errors instead of aborting, log a warning, and reinitialize the backend on CPU. Users on unsupported GPUs get a working app out of the box and never see a crash.

Parakeet models are small enough that CPU inference is a usable experience, so fallback is a real solution rather than a degraded one. This would resolve the launch crash in #585 (crash 1) at the root.

Alternatives considered:
Current workaround (confirmed working, documented in #585): launch via a wrapper script that sets CUDA_VISIBLE_DEVICES=-1. This is fragile — it requires users to diagnose the crash via stderr capture first, doesn't survive launches that bypass the wrapper (in-app updater relaunch, autostart), and hides the GPU from the process rather than letting the app manage backend selection properly.

Updating the NVIDIA driver may resolve PTX JIT compatibility for some users, but newer CUDA toolkits have dropped Maxwell support entirely, so a driver update cannot be a general fix for older GPUs.

Note: the nvdxgdmal64.dll teardown crash (crash 2 in #585) is out of scope here it occurs with CUDA already disabled and needs a separate fix in the GPU/renderer teardown path.

How I verified CPU inference works:

After root-causing the launch crash (#585) to ggml's CUDA backend aborting on an unsupported PTX error, we tested whether the app functions with the GPU excluded from inference. Launching fluidvoice.exe with the environment variable CUDA_VISIBLE_DEVICES=-1 hides all CUDA devices from the process, which forces ggml to initialize its CPU backend instead of aborting.

Result: the app launched successfully for the first time on this machine, and dictation worked end-to-end — audio captured, transcribed by the parakeet model on CPU, and text delivered. This was on modest hardware (a laptop with a 2016-era
Quadro M2000M, i.e. no GPU assist at all), and transcription performance was usable for normal dictation.

This confirms two things: (1) nothing in the pipeline actually requires CUDA the CPU path is fully functional and only unreachable because the CUDA error aborts the process before fallback can occur, and (2) a manual CPU option / automatic fallback is implementable with the backends already shipped in the app. The workaround has been running as a daily driver via a wrapper script
since, with no inference-related issues.

Test hardware:
Intel Core i7-6820HQ (4C/8T, Skylake, 2015) — CPU inference only, GPU fully excluded via CUDA_VISIBLE_DEVICES=-1. Dictation was responsive and accurate on this old quad-core mobile CPU, so the CPU fallback path should be viable across virtually all hardware.

[nconder]

This is how I forced FluidVoice to use the CPU on Windows.

A note on how the workaround is scoped, since it matters for anyone testing it:

I deliberately did NOT set CUDA_VISIBLE_DEVICES globally (via setx or system environment variables), because that would hide the GPU from every CUDA application under the user account — including other local AI tooling that should keep GPU acceleration.

Instead, the app is launched through a small wrapper script:

@echo off
set CUDA_VISIBLE_DEVICES=-1
set WEBVIEW2_ADDITIONAL_BROWSER_ARGUMENTS=--disable-gpu
start "" "%LOCALAPPDATA%\FluidVoice\fluidvoice.exe"

On Windows, a child process inherits a copy of its parent's environment at launch. The set statements modify only the wrapper's own environment block; fluidvoice.exe inherits that copy, and when the app (or the wrapper) exits, nothing persists — no registry changes, no global state, no effect on any other process. The GPU remains fully available to everything else on the system, including concurrently running CUDA workloads.

The flip side is that the workaround only applies to launches that go through the wrapper. Anything that starts fluidvoice.exe directly, the original installer shortcut, an in-app updater relaunch, a future AutoStart entry gets a clean environment and crashes again. This fragility is exactly why an in-app inference device setting would be the proper fix: a persisted setting travels with the app across every launch path, which no environment-variable workaround can do.

(CUDA_VISIBLE_DEVICES=-1 alone fixed the launch abort; --disable-gpu was additionally required to stop a separate access violation in nvdxgdmal64.dll_unloaded after dictation — details in #585.)

[nconder]

@altic-dev if the Windows code gets published I could codify this feature.

[altic-dev]

Hey! It’s still barebones and not worth makin it public yet. Hopefully soon in the future! But can take advise on it if you want to help :)

…

On Sun, Jul 12, 2026 at 3:10 PM nconder ***@***.***> wrote: @altic-dev <https://github.com/altic-dev> if the Windows code gets published I could codify this feature. — Reply to this email directly, view it on GitHub <#586?email_source=notifications&email_token=BVSOW2SMU22SG2H7HPO4VKT5EQEETA5CNFSNUABIM5UWIORPF5TWS5BNNB2WEL2ENFZWG5LTONUW63SDN5WW2ZLOOQXTCNZWGE2TKMBYUZZGKYLTN5XKO3LFNZ2GS33OUVSXMZLOOSWGM33PORSXEX3DNRUWG2Y#discussioncomment-17615508>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BVSOW2QT67PZ5A33Y556HFT5EQEETAVCNFSNUABJKJSXA33TNF2G64TZHMYTANRRGMZDOMZRGE5UI2LTMN2XG43JN5XDWMJQGQYTCMJTG2QXMAQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>