Documentation-correction release. No behaviour changes — but two
explanations published in 0.7.1 turned out to be wrong, both in the
direction of sounding more certain than the evidence supported.
"The workstation runs the CUDA build forced onto CPU" — refuted
0.7.1 offered this as the leading explanation for a 16-core workstation
trailing a 6-core laptop, and told readers not to treat -Device cpu on
a GPU box as representative of a CPU-only install. A second, genuinely
CPU-only runtime was provisioned alongside the GPU one to test it:
- There is no separate CPU build. PyPI ships one
ctranslate2wheel
with CUDA compiled in, and all four native binaries are
byte-identical across the two installs (sha256). - A CPU run never loads the NVIDIA libraries.
cuda_pathsputs them
on the DLL search path even for-Device cpu, but after a real int8
decodecublas64_12,cublasLt64_12,cudnn_ops64_9,
cudnn_graph64_9andnvrtc64_120_0are all unloaded. - The two stacks measure the same. Pinned to identical package
versions so they differed by exactly the threenvidia-*wheels, four
of five models produced identical segment counts and identical WER to
the decimal.
-Device cpu on a GPU machine is a faithful CPU-only measurement, so
contributors do not need GPU-free hardware to submit CPU numbers.
large-v3's poor WER had the wrong cause
0.7.1 explained it as the model "transcribing more literally — ~5% more
words than the captions contain". Those surplus words are repeated text
from decodes that got stuck: the runs that produced them tripped
watch-local's own repetition detector 1–5 times each.
On the reference fixture large-v3 degenerated in 8 of 8 GPU runs and
2 of 3 CPU runs. No other model has tripped the detector once. The
single clean run emitted 5,713 words against a 5,778-word reference —
fewer than the captions — and scored 5.9% rather than 13–17%.
Root cause is upstream and verified in faster_whisper/transcribe.py:
the default temperature ladder retries a segment judged "too repetitive"
(compression_ratio_threshold=2.4), and above temperature 0 decoding
switches from beam search to stochastic sampling — so identical audio
legitimately yields different transcripts, segment counts and runtimes.
large-v3 figures are now published as ranges with run counts attached.
tiny through medium reproduce to within contention noise and are
unchanged.
Also fixed while re-checking every number against the raw data
- "the laptop is roughly 2x faster than the i9" was stale — that
ratio came from the 16-thread sweep 0.7.1 withdrew. The default-thread
tables give 1.30–1.58x, and the page contradicted itself twice. - "
large-v3on GPU varied 16% between two identical runs" predated the
six-run repeat table added here. Eight GPU runs now span 255–414 s —
the slowest 62% slower than the fastest. - The order-reversed venv pairing now reports all three pairs,
including the contaminatedsmallpair that moved against the
conclusion. Omitting it would have been the same failure this release
is retracting. /watch's runtime CPU warning still carried the old 57–75 min range;
it now says 49–75 min and attributes the spread to run-to-run decode
instability rather than to the two machines measured.- The
large-v3row of the thread-count table now carries a warning: its
+28% is two single degenerate draws, and two runs at a fixed thread
count already span 51%.
Full detail: docs/benchmarks.md
🤖 Generated with Claude Code