Skip to content

Getting Good Results

Jump to bottom
LucQuebec edited this page Jul 5, 2026 · 2 revisions

Getting Good Results — a practical guide

Whisper is powerful, but the quality and speed you get depend heavily on which model you pick, the audio you feed it, and how you call the API. This page walks through the problems developers hit most often, with the symptom → cause → fix.

If you read only one thing: use base (or base.en), feed clean 16 kHz-ish audio, run off the main thread, and load the model once. The rest is tuning.

1. "It's too slow"

On-device transcription is CPU-bound. Speed is dominated by the model size and the length of the audio.

Symptom Cause Fix
Every call takes many seconds Model is too large for the device Drop from small → base → tiny. base.en is a great default on phones.
First call is much slower than the rest Model load + one-time warm-up Load the model once with loadModel(...), reuse it for every transcription, and only releaseModel(...) when you're truly done.
Long recordings take forever Time scales with audio duration Transcribe in shorter chunks, and/or skip silence (trim it, or use the Pro VAD).
Slower than expected on a fast phone Too few or too many threads Set WhisperConfig(threads = N) — 4 is a good default; match it to the device's performance cores. More isn't always faster.
UI freezes during transcription You're calling it on the main thread transcribe / loadModel are suspend functions — call them from a coroutine (Dispatchers.Default/IO).

Rule of thumb on a mid-range phone: tiny/base run at or faster than real-time; small can be several times slower than real-time. Test on your actual target hardware, not just an emulator.

Battery & heat: long back-to-back jobs on a big model will warm the device and drain battery. For long audio, chunk it and consider a foreground service.

2. "The transcription is wrong / it doesn't understand"

Accuracy problems are usually audio quality or model/language choice, not a bug.

  • Use a bigger model. tiny trades accuracy for speed. If words are wrong, try base → small. Accuracy improves noticeably at each step.
  • Set the language explicitly when you know it. WhisperConfig(language = "fr") is more reliable than "auto", especially on short or noisy clips where auto-detection can guess the wrong language and produce garbage.
  • Use the .en model for English. base.en is more accurate for English than the multilingual base of the same size.
  • Check the audio is actually speech-grade. Whisper expects clean voice. Very quiet audio, heavy compression artifacts, or clipping (too loud, distorted) all hurt. Aim for a clear, well-leveled recording.
  • Mono, ~16 kHz is ideal. The library resamples for you (WAV/MP3/FLAC, any rate), but garbage in = garbage out: a 8 kHz phone-call recording simply has less information than a clean 16 kHz capture.

3. Ambient noise

Whisper is fairly robust to mild background noise, but loud or constant noise (traffic, music, fans, crowd) degrades accuracy fast.

When recording yourself:

  • Get close to the microphone. Signal-to-noise ratio matters more than anything.
  • Use the speech-tuned mic source. On Android, record with MediaRecorder.AudioSource.VOICE_RECOGNITION (or VOICE_COMMUNICATION) rather than plain MIC — the OS applies noise suppression / AGC tuned for speech on most devices.
  • Record 16 kHz mono 16-bit PCM to skip any resampling loss: 
    val rec = AudioRecord(
    MediaRecorder.AudioSource.VOICE_RECOGNITION,
    16000, AudioFormat.CHANNEL_IN_MONO, AudioFormat.ENCODING_PCM_16BIT, bufSize,
)

When you can't control the noise:

  • Prefer a bigger model — small handles noise better than tiny.
  • Trim non-speech regions before transcribing (see hallucinations, below). The Pro tier's WhisperVAD / streaming does this automatically.

4. "It invents words during silence or music" (hallucinations)

This is the single most surprising behaviour for newcomers. On non-speech audio (silence, music, noise), Whisper can hallucinate plausible-sounding text — often a repeated phrase like "Thanks for watching" or a looping sentence.

Why: Whisper always tries to produce text for the audio it's given. Feed it 10 seconds of silence and it will still "transcribe" something.

Fixes:

  • Don't feed it non-speech. Trim leading/trailing silence and long gaps.
  • Use Voice Activity Detection to segment on speech only. The Pro tier ships WhisperVAD (offline) and VAD-segmented streaming for exactly this. In the Free tier, do a simple energy check yourself and skip quiet chunks.
  • Chunk long audio on natural pauses rather than one giant buffer — reduces runaway repetition loops.

5. Language, accents and translation

  • 99 languages are supported by the multilingual models. Set language to an ISO code ("en", "fr", "es", "de", …) or "auto" to detect.
  • Translate to English with WhisperConfig(translate = true) — it transcribes and translates in one pass.
  • Strong accents / code-switching (mixing languages mid-sentence) are hard. Pin the primary language and use a larger model.
  • Proper nouns, names, jargon, acronyms are frequently mis-spelled. This is a known Whisper limitation; a larger model helps somewhat. (Biasing with an initial prompt/vocabulary is on the roadmap — not exposed in the current API.)

6. Memory & lifecycle (avoid leaks and crashes)

  • Load once, reuse, release once. Reloading the model for every transcription is slow and wasteful. Keep one WhisperModel for the session. 
    val model = Whisper.loadModel(context, path)   // once
// ... many transcribe(...) calls ...
Whisper.releaseModel(model)                    // when done
  • Always releaseModel (e.g. in onCleared / onDestroy) — the native context holds hundreds of MB for larger models. Forgetting it leaks that memory.
  • Big models can OOM low-RAM devices. If you crash loading small, fall back to base/tiny.
  • One transcription at a time per model. Don't call transcribe concurrently on the same WhisperModel from multiple threads.

7. Quick decision guide

Your goal Model Notes
Fastest, English, low-end device tiny.en Lowest accuracy, real-time-ish
Best all-round default base / base.en Recommended starting point
Highest accuracy, short clips small Slow on phones; fine for offline batch
Multilingual base / small Set language explicitly when known

Pre-flight checklist before you ship:

  • Model loaded once and reused, released on teardown
  • transcribe called off the main thread (coroutine)
  • Language set explicitly if known (not "auto")
  • Recording close to the mic, VOICE_RECOGNITION source, 16 kHz mono
  • Silence/non-speech trimmed (or Pro VAD) to avoid hallucinations
  • Tested on a real target device, not just an emulator

Still stuck? See Troubleshooting for specific error messages, or open an issue on the repo. Real-time streaming, VAD and quantized models are in the Pro build — see jokobee.com.

Clone this wiki locally