GPU and CUDA

GPU & CUDA

TypeWhisper ships optional NVIDIA CUDA acceleration for its bundled local transcription engines. The GPU runtime is fetched on demand the first time you select CUDA — nothing GPU-related ships inside the packages — and cached locally. This page covers what it requires, which engines it applies to, and how to switch backends.

Requirements

To use the CUDA backend you need just one thing:

• An NVIDIA GPU and driver — this is the only hard requirement.

The CUDA 12 runtime libraries the engines need (libcudart.so.12, libcublas.so.12, cuDNN, and friends) are downloaded on demand the first time you select NVIDIA CUDA, verified by SHA-256, and cached under ~/.local/share/TypeWhisper/Runtimes/. Only the libraries you are actually missing are downloaded, so a host with a partial CUDA install fetches just the gaps and a host that already has a full CUDA toolkit on its library path downloads nothing. You do not need to install a CUDA toolkit yourself, and nothing GPU-related is bundled into the packages.

If an NVIDIA GPU is present but the runtime isn't cached yet, the Dictation section shows a Download CUDA runtime button (see below). On a host with no NVIDIA GPU, CUDA is unavailable and the engine stays on CPU — see Troubleshooting.

Choosing the backend

The Dictation section has an acceleration selector with three choices:

Option	Behavior
Auto	Use CUDA when available, otherwise CPU.
CPU	Force CPU.
NVIDIA CUDA	Force the CUDA backend. Selectable whenever an NVIDIA GPU is present; if the runtime libraries aren't cached yet, TypeWhisper offers to download them.

Restart to switch

The process pins the native runtime it first loads. TypeWhisper distinguishes that pinned runtime from the compute backend actually in use, so the status text reads "Restart TypeWhisper to apply" only when a switch genuinely needs a fresh process — for example a CPU-only process moving to CUDA, or recovering from a failed native-library load. Toggling compute on a process that is already CUDA-capable (such as falling back to CPU after a GPU-context error) no longer asks for a restart it doesn't need.

Download CUDA runtime

When an NVIDIA GPU is present but the CUDA runtime isn't cached yet, the Dictation page shows a Download CUDA runtime button. Clicking it fetches only the libraries TypeWhisper is missing — with a download-progress indicator — into ~/.local/share/TypeWhisper/Runtimes/, then asks you to restart TypeWhisper to load CUDA. The download only starts on a host that actually has an NVIDIA GPU, so a GPU-less machine never pulls the large runtime.

These downloads are resumable: an interrupted fetch streams into a .partial staging file and picks up where it left off on the next attempt rather than starting over, and a stalled or half-open connection aborts within seconds instead of hanging. When two TypeWhisper processes share the cache, a cross-process lock keeps their downloads from colliding.

Fix CUDA path

When the CUDA 12 libraries already exist on disk (for example a system toolkit) but aren't visible to the running process, TypeWhisper shows a Fix CUDA path button instead. Clicking it preloads libcudart.so.12 and libcublas.so.12 into the process so the engine can find them without LD_LIBRARY_PATH. Restart TypeWhisper afterward to load CUDA.

Clear GPU runtime

Once a runtime is cached, the Dictation page shows a Clear GPU runtime button (in place of Download CUDA runtime). A cached library is never re-verified or re-fetched once present, so if a cached runtime becomes corrupt the engine silently falls back to CPU with no automatic repair. Clearing the runtime deletes the cached libraries — both the engine's own GPU build and the shared CUDA math-library cache — so the next launch re-provisions them from scratch. A restart is required for the fresh download to take effect, because libraries already loaded this session are held until the process exits; after a successful clear, TypeWhisper waits until the next launch before offering the download again.

Scope

CUDA applies to the bundled whisper.cpp and sherpa-onnx (Parakeet, Canary) engines. Other bundled local engines stay on CPU.

Related pages

• Transcription engines — the engines available, including bundled whisper.cpp.
• Dictation — the acceleration selector and model settings.
• Requirements — full list of Linux requirements and optional helpers.

---

Changelog

Date	Change
2026-06-17	Initial version.
2026-06-17	Added acceleration selector, restart-to-switch behavior, Fix CUDA path button, and clarified only two libs are required.
2026-06-23	v0.10.0: CUDA runtime is now fetched on demand (no longer bundled) and cached under Runtimes/; added the Download CUDA runtime action; extended CUDA to the sherpa-onnx engine (Parakeet, Canary).
2026-06-24	v0.11.0: added the Clear GPU runtime action for corrupt-cache recovery; runtime downloads are now resumable and watchdog-guarded; refined the restart-to-switch wording now that the pinned runtime is tracked separately from the compute backend.