Skip to content

v0.7.2 -- documentation corrections

Latest

Choose a tag to compare

@PBNZ PBNZ released this 27 Jul 05:21
· 4 commits to main since this release

Documentation-correction release. No behaviour changes — but two
explanations published in 0.7.1 turned out to be wrong, both in the
direction of sounding more certain than the evidence supported.

"The workstation runs the CUDA build forced onto CPU" — refuted

0.7.1 offered this as the leading explanation for a 16-core workstation
trailing a 6-core laptop, and told readers not to treat -Device cpu on
a GPU box as representative of a CPU-only install. A second, genuinely
CPU-only runtime was provisioned alongside the GPU one to test it:

  • There is no separate CPU build. PyPI ships one ctranslate2 wheel
    with CUDA compiled in, and all four native binaries are
    byte-identical across the two installs (sha256).
  • A CPU run never loads the NVIDIA libraries. cuda_paths puts them
    on the DLL search path even for -Device cpu, but after a real int8
    decode cublas64_12, cublasLt64_12, cudnn_ops64_9,
    cudnn_graph64_9 and nvrtc64_120_0 are all unloaded.
  • The two stacks measure the same. Pinned to identical package
    versions so they differed by exactly the three nvidia-* wheels, four
    of five models produced identical segment counts and identical WER to
    the decimal.

-Device cpu on a GPU machine is a faithful CPU-only measurement, so
contributors do not need GPU-free hardware to submit CPU numbers.

large-v3's poor WER had the wrong cause

0.7.1 explained it as the model "transcribing more literally — ~5% more
words than the captions contain". Those surplus words are repeated text
from decodes that got stuck
: the runs that produced them tripped
watch-local's own repetition detector 1–5 times each.

On the reference fixture large-v3 degenerated in 8 of 8 GPU runs and
2 of 3 CPU runs
. No other model has tripped the detector once. The
single clean run emitted 5,713 words against a 5,778-word reference —
fewer than the captions — and scored 5.9% rather than 13–17%.

Root cause is upstream and verified in faster_whisper/transcribe.py:
the default temperature ladder retries a segment judged "too repetitive"
(compression_ratio_threshold=2.4), and above temperature 0 decoding
switches from beam search to stochastic sampling — so identical audio
legitimately yields different transcripts, segment counts and runtimes.

large-v3 figures are now published as ranges with run counts attached.
tiny through medium reproduce to within contention noise and are
unchanged.

Also fixed while re-checking every number against the raw data

  • "the laptop is roughly 2x faster than the i9" was stale — that
    ratio came from the 16-thread sweep 0.7.1 withdrew. The default-thread
    tables give 1.30–1.58x, and the page contradicted itself twice.
  • "large-v3 on GPU varied 16% between two identical runs" predated the
    six-run repeat table added here. Eight GPU runs now span 255–414 s —
    the slowest 62% slower than the fastest.
  • The order-reversed venv pairing now reports all three pairs,
    including the contaminated small pair that moved against the
    conclusion. Omitting it would have been the same failure this release
    is retracting.
  • /watch's runtime CPU warning still carried the old 57–75 min range;
    it now says 49–75 min and attributes the spread to run-to-run decode
    instability rather than to the two machines measured.
  • The large-v3 row of the thread-count table now carries a warning: its
    +28% is two single degenerate draws, and two runs at a fixed thread
    count already span 51%.

Full detail: docs/benchmarks.md

🤖 Generated with Claude Code

https://claude.ai/code/session_01TtKPaytUeHUKADa7UosGzd