Benchmark faster whisper turbo v3

[asr-lord]

#WIP

Benchmark with faster-whisper-large-v3-turbo-ct2

For reference, here's the time and memory usage that are required to transcribe 13 minutes of audio using different implementations:

• openai/whisper@25639fc
• faster-whisper@d57c5b4

Large-v3 model on GPU

Implementation	Precision	Beam size	Time	Max. GPU memory	Max. CPU memory	WER %
openai/whisper-large-v3	fp16	5	2m23s	MB	MB
openai/whisper-turbo	fp16	5	39s	MB	MB
faster-whisper	fp16	5	52.023s	4521MB	901MB	2.883
faster-whisper	int8	5	52.639s	2953MB	2261MB	4.594
faster-distil-large-v3	fp16	5	26.126s	2409MB	900MB	2.392
faster-distil-large-v3	int8	5	22.537s	1481MB	1468MB	2.392
faster-large-v3-turbo	fp16	5	19.155s	2537MB	899MB	1.919
faster-large-v3-turbo	int8	5	19.591s	1545MB	1526MB	1.919

WER on librispeech clean val split.

[mvoodarla]

We now support the new whisper-large-v3-turbo on Sieve!

Use it via sieve/speech_transcriber: https://www.sievedata.com/functions/sieve/speech_transcriber
Use sieve/whisper directly: https://www.sievedata.com/functions/sieve/whisper

Just set speed_boost to True. API guide is under "Usage Guide" tab.

[George0828Zhang]

Would be great if medium (or more sizes) are added for comparison! In OpenAI's implementation, turbo is 8x faster than v3 (medium is 2x and base is 7x), while offering similar WER as large-v2, which sounds surreal. Wonder how it translate to FW version.

[zx3777]

Would be great if medium (or more sizes) are added for comparison! In OpenAI's implementation, turbo is 8x faster than v3 (medium is 2x and base is 7x), while offering similar WER as large-v2, which sounds surreal. Wonder how it translate to FW version.

In my test, with the same 10-minute audio, Medium took 52 seconds, and Turbo took 39 seconds.

[createOne999]

You may find this discussion helpful:
openai/whisper#2363 (comment)

[zx3777]

Compared to Medium, Turbo and large-v3 has a more advanced timeline. Subtitles generated by Turbo appear earlier, but end precisely on time. Medium subtitles also appear early, but to a much lesser extent than Turbo. However, Medium subtitles are delayed in disappearing.

I find that subtitles disappearing late is a better experience than them appearing early. I should still use medium.

[MahmoudAshraf97]

Compared to Medium, Turbo and large-v3 has a more advanced timeline. Subtitles generated by Turbo appear earlier, but end precisely on time. Medium subtitles also appear early, but to a much lesser extent than Turbo. However, Medium subtitles are delayed in disappearing.

I find that subtitles disappearing late is a better experience than them appearing early. I should still use medium.

or you could use a forced alignment model after the transcription, much better timings than whisper

[jhj0517]

Great, thanks for your efforts! I hope turbo will be officially added here soon!

faster-whisper/faster_whisper/utils.py

Lines 12 to 29 in d57c5b4

	_MODELS = {
	"tiny.en": "Systran/faster-whisper-tiny.en",
	"tiny": "Systran/faster-whisper-tiny",
	"base.en": "Systran/faster-whisper-base.en",
	"base": "Systran/faster-whisper-base",
	"small.en": "Systran/faster-whisper-small.en",
	"small": "Systran/faster-whisper-small",
	"medium.en": "Systran/faster-whisper-medium.en",
	"medium": "Systran/faster-whisper-medium",
	"large-v1": "Systran/faster-whisper-large-v1",
	"large-v2": "Systran/faster-whisper-large-v2",
	"large-v3": "Systran/faster-whisper-large-v3",
	"large": "Systran/faster-whisper-large-v3",
	"distil-large-v2": "Systran/faster-distil-whisper-large-v2",
	"distil-medium.en": "Systran/faster-distil-whisper-medium.en",
	"distil-small.en": "Systran/faster-distil-whisper-small.en",
	"distil-large-v3": "Systran/faster-distil-whisper-large-v3",
	}

[NilaierMusic]

I benchmarked the models on my laptop using the same audio file in both sequential and batched processing. Seems that large-v3-turbo generally performs exceptionally well, offering greater accuracy than the base model while maintaining efficient processing times.

System Specifications

• CPU: Intel Core i7-12650H
• GPU: NVIDIA GeForce RTX 3060 Laptop (6 GB VRAM)
• RAM: SODIMM Samsung DDR4 8x2 GB 3200 MHz

Benchmark Details

• All models were tested with int8 precision.
• WER (Word Error Rate) was calculated by comparing the original French subtitles of a video with the transcriptions generated by the models.
• The language was explicitly set to French to prevent any translation errors or incorrect transcriptions.

---

Sequential Processing Benchmark

Model	WER (%)	Total Time (s)	Transcribe Time (s)	Model Load Time (s)
tiny	24.1%	28.95	28.44	0.51
base	16.0%	33.42	32.72	0.70
small	10.5%	55.62	53.21	2.41
medium	10.7%	113.25	106.30	6.95
large	17.6%	240.52	227.31	13.20
large-v1	8.7%	168.58	155.14	13.44
large-v2	8.5%	178.28	164.74	13.53
large-v3	17.6%	230.77	217.43	13.34
large-v3-turbo	9.5%	46.14	38.99	7.15

Observations:

• The large-v3-turbo model achieves a WER of 9.5%, which is significantly better than the base model and comparable to large-v2.
• In terms of speed, large-v3-turbo completes transcription in 38.99 seconds, much faster than other large models.

---

Batched Processing Benchmark

For batched processing, I used 10 batches for each model. I tried to use 16 batches, but some models thrown out-of-memory (OOM) errors due to the 6 GB VRAM limit.

Model	WER (%)	Total Time (s)	Transcribe Time (s)	Model Load Time (s)
tiny	23.6%	5.48	4.56	0.92
base	16.5%	6.92	5.70	1.22
small	9.8%	12.45	9.98	2.47
medium	8.9%	26.33	19.47	6.86
large	7.9%	35.97	29.66	6.31
large-v1	12.1%	42.90	29.64	13.26
large-v2	8.8%	43.17	29.71	13.46
large-v3	7.9%	42.97	29.69	13.28
large-v3-turbo	7.7%	18.68	11.47	7.20

Observations:

• With batched processing, large-v3-turbo achieves the best WER of 7.7%, outperforming all other models in both accuracy and speed.
• The transcribe time for large-v3-turbo is 11.47 seconds, making it suitable for real-time applications even on a laptop GPU.

---

Conclusions

• The large-v3-turbo model offers an excellent balance between accuracy and processing speed, especially evident in batched processing scenarios.
• It outperforms the base model in terms of WER while maintaining significantly lower processing times compared to other large models.

[tjongsma]

Just chiming in, I've tried using v3-turbo for streaming and found that it hallucinates more/misses audio more than other faster-whisper models. For example for this 10 second audio clip of an obama speech:
temp_audio_wav.zip
Using v3-turbo:

from faster_whisper import WhisperModel
model = WhisperModel(model_size_or_path="deepdml/faster-whisper-large-v3-turbo-ct2", device="cuda", compute_type="float16")

segments, info = model.transcribe("temp_audio.wav", initial_prompt="",
                                                                max_new_tokens=224,
                                                                beam_size=5,
                                                                temperature=0,
                                                                language="en",
                                                                word_timestamps=True,
                                                                vad_filter=False)
for seg in segments:
    print("[%.2fs -> %.2fs] %s" % (seg.start, seg.end, seg.text))

produces

[0.00s -> 9.66s] ...to give the president a chance.

Whereas using medium:

from faster_whisper import WhisperModel
model = WhisperModel(model_size_or_path="medium", device="cuda", compute_type="float16")

segments, info = model.transcribe("temp_audio.wav", initial_prompt="",
                                                                max_new_tokens=224,
                                                                beam_size=5,
                                                                temperature=0,
                                                                language="en",
                                                                word_timestamps=True,
                                                                vad_filter=False)
for seg in segments:
    print("[%.2fs -> %.2fs] %s" % (seg.start, seg.end, seg.text))

produces

[0.00s -> 1.70s] So give the president a chance.
[2.00s -> 4.96s] Governor Romney, I'm glad that you recognize that al-Qaeda is a threat.
[5.38s -> 9.66s] Because a few months ago, when you were asked what's the biggest geopolitical threat facing America, you said...

Not sure if any of you are experiencing anything similar? Or maybe an official faster-whisper turbo-v3 release would perform better.

[NilaierMusic]

Just chiming in, I've tried using v3-turbo for streaming and found that it hallucinates more/misses audio more than other faster-whisper models. For example for this 10 second audio clip of an obama speech: temp_audio_wav.zip Using v3-turbo:

from faster_whisper import WhisperModel
model = WhisperModel(model_size_or_path="deepdml/faster-whisper-large-v3-turbo-ct2", device="cuda", compute_type="float16")

segments, info = model.transcribe("temp_audio.wav", initial_prompt="",
                                                                max_new_tokens=224,
                                                                beam_size=5,
                                                                temperature=0,
                                                                language="en",
                                                                word_timestamps=True,
                                                                vad_filter=False)
for seg in segments:
    print("[%.2fs -> %.2fs] %s" % (seg.start, seg.end, seg.text))

produces

[0.00s -> 9.66s] ...to give the president a chance.

Whereas using medium:

from faster_whisper import WhisperModel
model = WhisperModel(model_size_or_path="medium", device="cuda", compute_type="float16")

segments, info = model.transcribe("temp_audio.wav", initial_prompt="",
                                                                max_new_tokens=224,
                                                                beam_size=5,
                                                                temperature=0,
                                                                language="en",
                                                                word_timestamps=True,
                                                                vad_filter=False)
for seg in segments:
    print("[%.2fs -> %.2fs] %s" % (seg.start, seg.end, seg.text))

produces

[0.00s -> 1.70s] So give the president a chance.
[2.00s -> 4.96s] Governor Romney, I'm glad that you recognize that al-Qaeda is a threat.
[5.38s -> 9.66s] Because a few months ago, when you were asked what's the biggest geopolitical threat facing America, you said...

Not sure if any of you are experiencing anything similar? Or maybe an official faster-whisper turbo-v3 release would perform better.

Haven't encountered that. I've tried the same audio and both models return the same transcription. I did notice that turbo model hallucinates more on noisy data than v3, but that's to be expected considering what we saw with common voice 15 benchmark.

[Jiltseb]

Sharing the benchmarking results with Turbo model compared to other large whisper models on one of the biggest open source long-form ASR evaluation dataset.
Our tests were conducted on a subset of YouTube-commons: yotutube-commons-asr-eval ).

Model	WER	Speed
large v3-turbo	13.40%	129.5x
large v3	13.20%	55.3x
large v2	14.10%	54.6x
distill-large-v3 (en)	15.00%	142.9x

The whisper-turbo model achieves a Word Error Rate (WER) similar to large models and excels in speed among multilingual models.

[Sharrnah]

Also it does not seem to support translation task, even though it is mentioned.
also tried it with Transformer large-v3 turbo, same behaviour.

[George0828Zhang]

Also it does not seem to support translation task, even though it is mentioned. also tried it with Transformer large-v3 turbo, same behaviour.

They specifically mentioned the translation task being excluded... openai/whisper#2363

excluding translation data, on which we don’t expect turbo to perform well.