Support for NVIDIA GPU on parakeet models

[hubertkirch]

First of all, the application is truly brilliant, and thank you for creating, maintaining and improving. Excellent job!

In terms of a new feature: support parakeet with GPU acceleration. The application works wonderfully on my Macbook Air M4, however, my development desktop with a 5090 still runs parakeet on the CPU, which (of course) is notably slower.

If you point me in the right direction on how to do the implementation, I would be happy to create the PR myself.

[cjpais]

Hi @hubertkirch check out the library cjpais/transcribe-rs, if we get it working there as a first step it'll be helpful to bundle here

[hubertkirch]

I am working on incorporating GPU support for parakeet, but the ort library is giving issues. Running the model through ONNX runtime on python does inference perfectly fine. Will circle back when I have more information.

[timmmmmmmmm]

Just wanted to say that I came here specifically to see whether more people are experiencing this. I have a MacBook Air M1 for personal use and a fairly beefy HP EliteBook for work, with a 12th gen Intel i7-12850HX and an RTX A3000.

On my MacBook Air it is almost instant, and I genuinely enjoy showing Handy to people because of how fast it is, it almost feels like a party trick. But when I want to use it for actual work on my HP EliteBook, it is so slow that it really takes me out of my flow.

I still really prefer Handy because I like that it provides these open-source models for ASR, and it is such a nice and simple open-source free package, so thank you for that. In the README and elsewhere in the repo it is mentioned that running on the CPU should be enough, which is why this difference is so surprising.

On my laptop, the transcription speed is maybe around 2x real time, while on my MacBook it is almost instant.

[hubertkirch]

For my 5090, I ended up building a custom tool in Python that does around 40-100x real-time inference.

[cjpais]

Hey @timmmmmmmmm if you're willing to test builds of Handy, I am happy to try and distribute ones which include CUDA acceleration specifically. I think more or less the under the hood library should support it easily, but maybe distribution may be a little challenging. I dont have a NVIDIA machine right now so I won't be able to test things myself but happy to put out stuff to test if it improves UX

[timmmmmmmmm]

That would be great, I'll be happy to test and give feedback from my machine.

[hubertkirch]

I would be happy to test as well.

[cjpais]

Thank you both! Hopefully I'll review a PR in transcribe-rs and try to add some support for CUDA. I'm not sure how bundling will go but will update this thread when I have a build

[ballyhoo38923]

Building Handy with CUDA Support: 3x Faster Transcription (Non-Developer Guide)

A Little Context

I'm not a developer. My background is in IT and software testing — mostly Microsoft technologies — and I'd never touched C++, Rust, React, Tauri, etc before this. I now work at a medical practice where fast, accurate transcription matters, so I spent time evaluating about half a dozen open-source speech-to-text programs. Handy stood out immediately for its clean, simple UI and the fact that it just works. It quickly became my daily driver. Thank you cjpais for building such a nice piece of software!

The one thing I wanted was better GPU utilization. My RTX 4060 Ti 16GB was barely being tapped during transcription. So with the help of Claude, I worked through building Handy from source with CUDA support over the course of about 3.5 hours. We hit six separate build errors along the way — each one a different layer of the toolchain — and solved them all. I have zero C++ or Rust experience, so this was entirely an exercise in persistence and good AI-assisted troubleshooting. In full disclosure, Claude even generated this preface and build guide. Credit where credit is due.

The results speak for themselves. I'm hoping the developer might consider making CUDA a supported build option so other NVIDIA GPU users can benefit without going through this process. I also plan to upgrade to an RTX 5070 Ti soon and will post updated benchmarks when that happens.

Benchmark Results

Test setup: A 3-minute, 10-second audio clip of a physician reading a surgical operative report — roughly 570 words of dense medical terminology. Audio was played back through a speaker to simulate real-world dictation conditions. The same clip was used across all tests to eliminate variables.

Build	Transcription Time	GPU Utilization	Speedup
Official Handy (Vulkan)	1 min 1 sec	~55%	Baseline
CUDA Debug build	23 sec	90%+	2.7x
CUDA Release build (Test 1)	19.27 sec	97%	3.2x
CUDA Release build (Test 2)	18.98 sec	97%	3.2x

From over a minute down to under 20 seconds — a 3.2x speedup with the GPU fully engaged.

My Setup

Component	Details
GPU	NVIDIA RTX 4060 Ti 16GB (Ada Lovelace, sm_89)
OS	Windows 11
Visual Studio	2026 (MSVC 19.50.35725.0)
CUDA Toolkit	13.1.115
CMake	4.0.3-rc1
Rust	Latest stable
Handy	v0.7.8

Note: This is a bleeding-edge toolchain. Some of the issues below (particularly VS 2026 and CUDA 13.1 compatibility) may not apply if you're on VS 2022 + CUDA 12.x.

What I Changed

1. Swap Vulkan for CUDA in transcribe-rs

Clone transcribe-rs locally and change the whisper feature flag in transcribe-rs/Cargo.toml:

# Before
whisper-rs = { version = "0.13", optional = true, features = ["vulkan"] }

# After
whisper-rs = { version = "0.13", optional = true, features = ["cuda"] }

Then point Handy's src-tauri/Cargo.toml to your local copy:

transcribe-rs = { path = "../../transcribe-rs", features = ["whisper", "parakeet", "moonshine", "sense_voice"] }

2. Set Environment Variables

These must be set before every build (in every new terminal session):

$env:CMAKE_GENERATOR = "Ninja"
$env:CUDAFLAGS = "-allow-unsupported-compiler --std=c++17"
$env:CUDAARCHS = "89"

Adjust CUDAARCHS to match your GPU:

GPU Family	Architecture	CUDAARCHS Value
RTX 20-series (Turing)	sm_75	75
RTX 30-series (Ampere)	sm_86	86
RTX 40-series (Ada Lovelace)	sm_89	89
RTX 50-series (Blackwell)	sm_120	120

3. Build

cd C:\path\to\Handy

# Dev build (faster to compile, unoptimized, requires Vite dev server running)
bun run tauri dev

# Release build (slower to compile, fully optimized, standalone exe)
bun run tauri build

First build takes 10–15 minutes (79 CUDA kernels). Subsequent builds use the cache and are much faster. The release build produces a standalone exe at src-tauri\target\release\handy.exe.

Note: The release build may fail at the code-signing/bundling step with "program not found" — this is normal. The signing certificate is specific to the official releases. Your exe is already built and fully functional at that point.

---

Build Errors and Solutions

I hit six issues during the build. Here's each one and how we solved it.

1. libclang.dll Not Found

Error: Unable to find libclang

Why: The bindgen crate (used by whisper-rs-sys) needs libclang, which isn't on PATH by default.

Fix: Set LIBCLANG_PATH to your LLVM installation. Visual Studio bundles one:

$env:LIBCLANG_PATH = "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\Llvm\x64\bin"

2. CMake Generator Incompatibility (VS 2026)

Error: CMAKE_GENERATOR_INSTANCE should not be set for Ninja generator

Why: The Rust cmake crate auto-detects Visual Studio and passes instance metadata to CMake. This flag is invalid for the Ninja generator and causes a hard failure.

Fix: Force Ninja as the generator:

$env:CMAKE_GENERATOR = "Ninja"

Ninja must be installed and on PATH. It's often bundled with CMake, or install via winget install Ninja-build.Ninja.

3. VS 2026 Unrecognized by nvcc

Error: nvcc fatal: Host compiler older than 5.0 is not supported

Why: CUDA 13.1's nvcc doesn't recognize the VS 2026 compiler version and misidentifies it as ancient.

Fix:

$env:CUDAFLAGS = "-allow-unsupported-compiler"

4. Deprecated CUDA Architectures

Error: nvcc fatal: Unsupported gpu architecture 'compute_52'

Why: CUDA 13.1 dropped support for Maxwell (sm_52) and Pascal (sm_61), but whisper.cpp defaults to those older architectures.

Fix: Set your specific GPU architecture:

$env:CUDAARCHS = "89"

This also cuts compile time ~50% by only targeting your GPU.

Forward compatibility note: The binary embeds both native code (SASS) and intermediate code (PTX). Future GPUs can JIT-compile the PTX at first launch, so the binary is forward-compatible with a small one-time startup delay.

5. C++17 Required by CCCL/CUB

Error: CUB requires at least C++17

Why: CUDA 13.1's CCCL libraries enforce C++17 as a minimum standard, but whisper.cpp doesn't set CMAKE_CUDA_STANDARD. The build compiles ~42 of 79 CUDA kernels before failing on sum.cu.

Fix:

$env:CUDAFLAGS = "-allow-unsupported-compiler --std=c++17"

6. Missing Feature Flags

Error: unresolved imports: transcribe_rs::engines::moonshine, transcribe_rs::engines::sense_voice

Why: When switching to the local path dependency, it's easy to accidentally drop feature flags from the dependency line.

Fix: Make sure all original features are listed in Handy/src-tauri/Cargo.toml:

transcribe-rs = { path = "../../transcribe-rs", features = ["whisper", "parakeet", "moonshine", "sense_voice"] }

---

Complete Build Steps (Copy-Paste Ready)

# Prerequisites: CUDA Toolkit, Ninja, Rust, Bun, Visual Studio with C++ workload

# 1. Clone repos side by side
git clone https://github.com/cjpais/Handy.git
git clone https://github.com/cjpais/transcribe-rs.git

# 2. In transcribe-rs/Cargo.toml, change whisper-rs features from "vulkan" to "cuda"

# 3. In Handy/src-tauri/Cargo.toml, change the transcribe-rs line to:
#    transcribe-rs = { path = "../../transcribe-rs", features = ["whisper", "parakeet", "moonshine", "sense_voice"] }

# 4. Set environment variables (required in every new terminal)
$env:CMAKE_GENERATOR = "Ninja"
$env:CUDAFLAGS = "-allow-unsupported-compiler --std=c++17"
$env:CUDAARCHS = "89"

# 5. Build
cd Handy
bun run tauri build

# 6. Run (release exe is self-contained, no dev server needed)
.\src-tauri\target\release\handy.exe

Suggestions for the Project

1. CUDA as an official build option. Even as an experimental flag, this would let NVIDIA GPU users get dramatically better performance without modifying the source.
2. CUDA build documentation. A section in BUILD.md covering the required environment variables and common pitfalls would go a long way.
3. Graceful architecture handling. whisper.cpp's hardcoded architecture list (52, 61, 70, 75) breaks on newer CUDA versions that dropped legacy support. Respecting CUDAARCHS or auto-detecting the installed GPU would prevent this.

---

Built on March 1, 2026. I'll update this post with RTX 5070 Ti benchmarks when I have the card.

[cjpais]

It's being worked on officially, the main issue is distribution and ci/cd pipelines