Parakeet MLX 0.2.7 Benchmark Result

[senstella]

All runs were conducted with my Macbook Air M2 16GB 2022 with the following chunking parameters:

• Chunk Duration: 120 seconds
• Overlap Duration: 15 seconds

with model of mlx-community/parakeet-tdt-0.6b-v2 as bf16 precision which are default settings in CLI.

Please note that Macbook Air has no cooling fans so it throttled GPU a lot while benchmarking.
(I ran them continously in one script)

Audio File	Audio Dur. (s)	Avg. Proc. Time (s)	RTF	RTFx (1/RTF)	Runs
beautyofbezier.wav	1465.09	38.42	0.0262	38.13	3
youtuberewind2018.wav	493.01	12.51	0.0254	39.42	3
mathbreakthrough.wav	4428.65	126.90	0.0287	34.90	3
fireship.wav	239.21	7.32	0.0306	32.69	3
mechinterp.wav	13356.63	397.15	0.0297	33.63	3

Note that RTFx equals how many seconds of audio it can process in one second. So, in average, with Macbook Air M2, it can process 35 seconds of audio for each seconds.

[tennyson-mccalla]

Awesome! What do I need to do to reproduce this on my machines?

[senstella]

https://gist.github.com/senstella/76bc185802e10e8b4bd4f4fda7295531

You can run this script with audio folders specified. And it will generate benchmark_results folder where it contains all the information you need!

[tennyson-mccalla]

MacBook Air M3 24 GB 2024

Chunk duration 120.0s
Using fixed overlap duration of 15.0 seconds

❯ ./parakeet-mlx-benchmark.py --audio_dir .
Found 5 audio files
Loading model: mlx-community/parakeet-tdt-0.6b-v2
config.json: 100%|████████████████████████████████████████████████████████████████████████████| 36.2k/36.2k [00:00<00:00, 95.9MB/s]
Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: pip install huggingface_hub[hf_xet] or pip install hf_xet
model.safetensors: 100%|██████████████████████████████████████████████████████████████████████| 2.47G/2.47G [00:55<00:00, 44.3MB/s]
Using fixed overlap duration of 15.0 seconds
Processing audio files: 0%| | 0/5 [00:00<?, ?it/s]
./fireship.wav 0 120.0
./fireship.wav 1 120.0
Processing audio files: 20%|██████████████▍ | 1/5 [00:23<01:34, 23.57s/it]
./mechinterp.wav 0 120.0
./mechinterp.wav 1 120.0
Processing audio files: 40%|████████████████████████████▍ | 2/5 [15:40<27:28, 549.37s/it]
./mathbreakthrough.wav 0 120.0
./mathbreakthrough.wav 1 120.0
Processing audio files: 60%|██████████████████████████████████████████▌ | 3/5 [23:16<16:53, 506.72s/it]
./youtuberewind2018.wav 0 120.0
./youtuberewind2018.wav 1 120.0
Processing audio files: 80%|████████████████████████████████████████████████████████▊ | 4/5 [24:09<05:27, 327.22s/it]
./beautyofbezier.wav 0 120.0
./beautyofbezier.wav 1 120.0
Processing audio files: 100%|███████████████████████████████████████████████████████████████████████| 5/5 [26:39<00:00, 319.89s/it]
Results saved to benchmark_results/benchmark_results.csv

Benchmark Summary (15s fixed overlap):
Chunk duration 120.0s:
Avg processing time: 155.73s
Avg RTF: 0.04
Avg RTFx: 25.33

Best configurations by processing time:

Chunk=120.0s, Overlap=15s: 6.95s
Chunk=120.0s, Overlap=15s: 21.95s

audio_file	audio_duration	chunk_duration	overlap_duration	overlap_ratio	processing_time_avg	processing_time_std	real_time_factor	rtfx	num_successful_runs
fireship.wav	239.210666667	120.0	15.0	0.125	6.947362065	0.445257068	0.029	34.43	1869883524222
mechinterp.wav	13356.628771	120.0	15.0	0.125	454.6502470	127.8734261	0.034	29.38	7810436965062
mathbreakthrough.wav	4428.641	120.0	15.0	0.125	223.9153113	11.6232016	0.051	19.78	1919135681432
youtuberewind2018.wav	493.006083333	120.0	15.0	0.125	21.9509096	1.3129964	0.045	22.46	4594830915916
beautyofbezier.wav	1465.08627083	120.0	15.0	0.125	71.1691575	1.8237994	0.049	20.59	8597181974052

[tennyson-mccalla]

Why was it slower for me?

[tennyson-mccalla]

For completion I ran it with the environment variables as suggested by that post over on the mlx repo:

parakeet-mlx on  master [!?] is 📦 v0.2.7 via 🐍 v3.11.12 (parakeet-mlx) on ☁️ (us-east-1)
❯ MLX_MAX_OPS_PER_BUFFER=8 MLX_MAX_MB_PER_BUFFER=1000000 uv run --python "python" --with mlx==0.23.1 parakeet-mlx-benchmark.py --audio_dir .
Installed 1 package in 8ms
Found 5 audio files
Loading model: mlx-community/parakeet-tdt-0.6b-v2
Using fixed overlap duration of 15.0 seconds
Processing audio files: 0%| | 0/5 [00:00<?, ?it/s]
./fireship.wav 0 120.0
./fireship.wav 1 120.0
Processing audio files: 20%|██████████████▍ | 1/5 [00:22<01:28, 22.12s/it]
./mechinterp.wav 0 120.0
./mechinterp.wav 1 120.0
Processing audio files: 40%|████████████████████████████▍ | 2/5 [15:30<27:10, 543.46s/it]
./mathbreakthrough.wav 0 120.0
./mathbreakthrough.wav 1 120.0
Processing audio files: 60%|██████████████████████████████████████████▌ | 3/5 [22:40<16:23, 491.64s/it]
./youtuberewind2018.wav 0 120.0
./youtuberewind2018.wav 1 120.0
Processing audio files: 80%|████████████████████████████████████████████████████████▊ | 4/5 [23:23<05:14, 314.36s/it]
./beautyofbezier.wav 0 120.0
./beautyofbezier.wav 1 120.0
Processing audio files: 100%|███████████████████████████████████████████████████████████████████████| 5/5 [25:10<00:00, 302.09s/it]
Results saved to benchmark_results/benchmark_results.csv
Benchmark Summary (15s fixed overlap):
Chunk duration 120.0s:
Avg processing time: 146.99s
Avg RTF: 0.04
Avg RTFx: 28.64
Best configurations by processing time:
Chunk=120.0s, Overlap=15s: 6.94s
Chunk=120.0s, Overlap=15s: 17.25s
Chunk=120.0s, Overlap=15s: 49.65s

audio_file	audio_duration	chunk_duration	overlap_duration	overlap_ratio	processing_time_avg	processing_time_std	real_time_factor	rtfx	num_successful_runs
fireship.wav	239.21066666666667	120.0	15.0	0.125	6.943742990	0.469544888	0.029	34.449815754	2
mechinterp.wav	13356.628770833333	120.0	15.0	0.125	450.1619383	70.6657115	0.034	29.670719877	2
mathbreakthrough.wav	4428.64	120.0	15.0	0.125	210.9342752	1.5209169	0.048	20.995355055	2
youtuberewind2018.wav	493.0060833333333	120.0	15.0	0.125	17.248407483	2.8231665	0.035	28.582701552	2
beautyofbezier.wav	1465.0862708333334	120.0	15.0	0.125	49.647166491	1.6582456	0.034	29.509967525	2

[tennyson-mccalla]

Ran it again with the MacBook Air elevated just for consistency (the previous run was not elevated). While I can say that the numbers are better with the laptop elevated it's not a huge jump and at no point have I even matched your best number from your benchmark run despite this significantly better specced machine.

[senstella]

One reason I can think of is that using MLX 0.23.1 loses the benefit of the fused attention kernel speed up introduced in 0.25.0. The Parakeet implementation leverages an additive mask! I wonder if the second result is from a run using 0.25.2 with those flags or if it's simply an re-run with MacBook Air elevated?

[tennyson-mccalla]

One reason I can think of is that using MLX 0.23.1 loses the benefit of the fused attention kernel speed up introduced in 0.25.0. The Parakeet implementation leverages an additive mask! I wonder if the second result is from a run using 0.25.3 with those flags or if it's simply an re-run with MacBook Air elevated?

Haven't updated to 0.25.3 yet but perhaps I'll try that as well.

[senstella]

Oops, typo! 0.25.2 I meant!

[tennyson-mccalla]

I wanted to throw a comparison in here. I'm also using whisper.cpp for my project and this is the result from the cli tool there on the fireship.wave sample:

❯ ./build/bin/whisper-cli -m models/ggml-base.bin -f ../../parakeet-mlx/fireship.wav
whisper_init_from_file_with_params_no_state: loading model from 'models/ggml-base.bin'
whisper_init_with_params_no_state: use gpu = 1
whisper_init_with_params_no_state: flash attn = 0
whisper_init_with_params_no_state: gpu_device = 0
whisper_init_with_params_no_state: dtw = 0
whisper_init_with_params_no_state: devices = 3
whisper_init_with_params_no_state: backends = 3
whisper_model_load: loading model
whisper_model_load: n_vocab = 51865
whisper_model_load: n_audio_ctx = 1500
whisper_model_load: n_audio_state = 512
whisper_model_load: n_audio_head = 8
whisper_model_load: n_audio_layer = 6
whisper_model_load: n_text_ctx = 448
whisper_model_load: n_text_state = 512
whisper_model_load: n_text_head = 8
whisper_model_load: n_text_layer = 6
whisper_model_load: n_mels = 80
whisper_model_load: ftype = 1
whisper_model_load: qntvr = 0
whisper_model_load: type = 2 (base)
whisper_model_load: adding 1608 extra tokens
whisper_model_load: n_langs = 99
whisper_model_load: Metal total size = 147.37 MB
whisper_model_load: model size = 147.37 MB
whisper_backend_init_gpu: using Metal backend
ggml_metal_init: allocating
ggml_metal_init: found device: Apple M3
ggml_metal_init: picking default device: Apple M3
ggml_metal_load_library: using embedded metal library
ggml_metal_init: GPU name: Apple M3
ggml_metal_init: GPU family: MTLGPUFamilyApple9 (1009)
ggml_metal_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_init: GPU family: MTLGPUFamilyMetal3 (5001)
ggml_metal_init: simdgroup reduction = true
ggml_metal_init: simdgroup matrix mul. = true
ggml_metal_init: has residency sets = true
ggml_metal_init: has bfloat = true
ggml_metal_init: use bfloat = false
ggml_metal_init: hasUnifiedMemory = true
ggml_metal_init: recommendedMaxWorkingSetSize = 17179.89 MB
ggml_metal_init: skipping kernel_get_rows_bf16 (not supported)
ggml_metal_init: skipping kernel_mul_mv_bf16_f32 (not supported)
ggml_metal_init: skipping kernel_mul_mv_bf16_f32_1row (not supported)
ggml_metal_init: skipping kernel_mul_mv_bf16_f32_l4 (not supported)
ggml_metal_init: skipping kernel_mul_mv_bf16_bf16 (not supported)
ggml_metal_init: skipping kernel_mul_mv_id_bf16_f32 (not supported)
ggml_metal_init: skipping kernel_mul_mm_bf16_f32 (not supported)
ggml_metal_init: skipping kernel_mul_mm_id_bf16_f16 (not supported)
ggml_metal_init: skipping kernel_flash_attn_ext_bf16_h64 (not supported)
ggml_metal_init: skipping kernel_flash_attn_ext_bf16_h80 (not supported)
ggml_metal_init: skipping kernel_flash_attn_ext_bf16_h96 (not supported)
ggml_metal_init: skipping kernel_flash_attn_ext_bf16_h112 (not supported)
ggml_metal_init: skipping kernel_flash_attn_ext_bf16_h128 (not supported)
ggml_metal_init: skipping kernel_flash_attn_ext_bf16_h192 (not supported)
ggml_metal_init: skipping kernel_flash_attn_ext_bf16_hk192_hv128 (not supported)
ggml_metal_init: skipping kernel_flash_attn_ext_bf16_h256 (not supported)
ggml_metal_init: skipping kernel_flash_attn_ext_bf16_hk576_hv512 (not supported)
ggml_metal_init: skipping kernel_flash_attn_ext_vec_bf16_h96 (not supported)
ggml_metal_init: skipping kernel_flash_attn_ext_vec_bf16_h128 (not supported)
ggml_metal_init: skipping kernel_flash_attn_ext_vec_bf16_h192 (not supported)
ggml_metal_init: skipping kernel_flash_attn_ext_vec_bf16_hk192_hv128 (not supported)
ggml_metal_init: skipping kernel_flash_attn_ext_vec_bf16_h256 (not supported)
ggml_metal_init: skipping kernel_flash_attn_ext_vec_bf16_hk576_hv512 (not supported)
ggml_metal_init: skipping kernel_cpy_f32_bf16 (not supported)
ggml_metal_init: skipping kernel_cpy_bf16_f32 (not supported)
ggml_metal_init: skipping kernel_cpy_bf16_bf16 (not supported)
whisper_backend_init: using BLAS backend
whisper_init_state: kv self size = 6.29 MB
whisper_init_state: kv cross size = 18.87 MB
whisper_init_state: kv pad size = 3.15 MB
whisper_init_state: loading Core ML model from 'models/ggml-base-encoder.mlmodelc'
whisper_init_state: first run on a device may take a while ...
whisper_init_state: Core ML model loaded
whisper_init_state: compute buffer (conv) = 5.61 MB
whisper_init_state: compute buffer (cross) = 7.72 MB
whisper_init_state: compute buffer (decode) = 97.27 MB

system_info: n_threads = 4 / 8 | WHISPER : COREML = 1 | OPENVINO = 0 | Metal : EMBED_LIBRARY = 1 | CPU : ARM_FMA = 1 | FP16_VA = 1 | MATMUL_INT8 = 1 | DOTPROD = 1 | ACCELERATE = 1 | AARCH64_REPACK = 1 |

main: processing '../../parakeet-mlx/fireship.wav' (3827371 samples, 239.2 sec), 4 threads, 1 processors, 5 beams + best of 5, lang = en, task = transcribe, timestamps = 1 ...

[00:00:00.000 --> 00:00:05.200] Yesterday, Google released a new version of Gemini 2.5 Pro, and the thing that everybody
[00:00:05.200 --> 00:00:08.720] always thought would happen, but never happened, finally just happened.
[00:00:08.720 --> 00:00:13.440] They built arguably the best coding AI model of all time, currently ranking number 1 in
[00:00:13.440 --> 00:00:14.800] all alumarinas.
[00:00:14.800 --> 00:00:18.560] The timing for this release is surprising, because their annual developer conference
[00:00:18.560 --> 00:00:22.680] Google I/O is just two weeks away, and that's where they normally hype up their best new
[00:00:22.680 --> 00:00:23.680] tech.
[00:00:23.680 --> 00:00:31.620] Which could only mean that they must have something even bigger cooking, like maybe Gemini
[00:00:31.620 --> 00:00:34.020] 3 or Gemini 2.5 Ultra.
[00:00:34.020 --> 00:00:38.000] Only time will tell, but one thing we do know for sure, thanks to an accidentally published
[00:00:38.000 --> 00:00:42.060] blog post, is that the Android 16 UI is getting a major overhaul.
[00:00:42.060 --> 00:00:46.680] To be more emotional and expressive, in today's video we'll try out Google's latest model,
[00:00:46.680 --> 00:00:50.440] and look at other recent developments in the AI space that every vibe coder needs to know
[00:00:50.440 --> 00:00:51.440] about.
[00:00:51.440 --> 00:00:58.840] But before we talk about Gemini, another huge development is that OpenAI is abandoning
[00:00:58.840 --> 00:01:01.560] its plans to convert to a for-profit company.
[00:01:01.560 --> 00:01:06.580] Many people, and especially Elon Musk, were outraged at the idea of OpenAI making money.
[00:01:06.580 --> 00:01:09.220] And this headline sounds like a huge win for humanity.
[00:01:09.220 --> 00:01:12.320] But with any announcement like this, the devil is in the details.
[00:01:12.320 --> 00:01:18.060] And what we find is that the current, capped-profit LLC will become an uncapped-profit, public-benefit
[00:01:18.060 --> 00:01:19.060] corporation.
[00:01:19.060 --> 00:01:21.400] The key word there being uncapped-profit.
[00:01:21.400 --> 00:01:25.480] The non-profit will continue to oversee and be a shareholder in the PBC.
[00:01:25.480 --> 00:01:29.440] And this approach has much better optics for OpenAI, but it paves the way for them to make
[00:01:29.440 --> 00:01:31.200] tons of money without much friction.
[00:01:31.200 --> 00:01:34.760] All they have to do is pretend that AI is super scary, and convince politicians that they're
[00:01:34.760 --> 00:01:37.640] trying really hard to keep it in a cage to benefit the public.
[00:01:37.640 --> 00:01:42.240] It's worth noting that Anthropic and XAI are also public-benefit corporations.
[00:01:42.240 --> 00:01:46.760] Now, despite having these super-advanced AI models, OpenAI just agreed to acquire Windsor
[00:01:46.760 --> 00:01:50.100] AVS Code 4 for an incredible $3 billion.
[00:01:50.100 --> 00:01:54.880] Meanwhile, OpenAI is claiming that their AI is one of the top 50 programmers in the world.
[00:01:54.880 --> 00:01:59.480] Impressive, but that begs the question, why pay $3 billion for AVS Code 4 when you could
[00:01:59.480 --> 00:02:01.380] just build one with your own AI?
[00:02:01.380 --> 00:02:04.880] But OpenAI is feeling less and less dominant in the AI space every day.
[00:02:04.880 --> 00:02:10.440] Gemini 2.5 is completely dominating LM Arena, including the coding and webdeverenas, which
[00:02:10.440 --> 00:02:15.280] is like a blind taste test where people choose the best LLM based on actual responses.
[00:02:15.280 --> 00:02:19.440] However, if we look at Live Bench, which takes a more scientific approach and tries to provide
[00:02:19.440 --> 00:02:24.080] LLMs with contamination-free questions, you'll find that OpenAI is still on top and Gemini
[00:02:24.080 --> 00:02:25.080] falls behind.
[00:02:25.080 --> 00:02:29.240] Furthermore, it appears Gemini falls behind on its own benchmarks compared to the last
[00:02:29.240 --> 00:02:31.880] version they released, except when it comes to coding.
[00:02:31.880 --> 00:02:35.960] Like I've said many times before though, never trust benchmarks and just try it for yourself.
[00:02:35.960 --> 00:02:39.880] The first test I usually do is to see if it can generate Svelte 5 room code.
[00:02:39.880 --> 00:02:42.760] And surprisingly, Gemini 2.5 almost did it right.
[00:02:42.760 --> 00:02:47.200] I prompted it for a simple to-do app and the actual syntax looks correct, although it wasn't
[00:02:47.200 --> 00:02:50.200] doing things correctly and the app itself didn't actually run.
[00:02:50.200 --> 00:02:53.320] That's disappointing, but at least it's a step in the right direction.
[00:02:53.320 --> 00:02:57.040] The next thing I had to do was vibe code a game from scratch with 3JS.
[00:02:57.040 --> 00:03:00.440] And the end result was pretty solid, but not significantly better than any of the other
[00:03:00.440 --> 00:03:01.680] models I've tried.
[00:03:01.680 --> 00:03:05.120] But one thing about Gemini is that it's especially good with vision prompts.
[00:03:05.120 --> 00:03:08.320] So what I did is sketch out my application on a piece of toilet paper.
[00:03:08.320 --> 00:03:12.360] Then I asked it to build a full stack application with a Postgres database.
[00:03:12.360 --> 00:03:15.160] And the end result actually had some pretty solid vibes.
[00:03:15.160 --> 00:03:17.680] But now the more important decision is where do we deploy it?
[00:03:17.680 --> 00:03:21.280] Well, that's why you need to know about Svelte, the sponsor of today's video.
[00:03:21.280 --> 00:03:25.680] If you're old enough to remember Faroku, Svelte is like a superior modern successor, where
[00:03:25.680 --> 00:03:30.660] you can deploy entire full stack applications, databases, and static websites, all backed
[00:03:30.660 --> 00:03:33.120] by Google Kubernetes engine and Cloudflare.
[00:03:33.120 --> 00:03:36.840] Without the pain of all these yaml configs, we can easily ship our app by first connecting
[00:03:36.840 --> 00:03:41.240] to get repo or Docker image to Svelte, provision some resources, and then click the
[00:03:41.240 --> 00:03:42.240] deploy button.
[00:03:42.240 --> 00:03:46.040] What's especially awesome is that they just released a database studio that allows us
[00:03:46.040 --> 00:03:48.760] to manage our database directly from their web UI.
[00:03:48.760 --> 00:03:51.800] All infrastructure and data can be easily managed in one place.
[00:03:51.800 --> 00:03:55.760] It gives Svelte a try for free with $50 and free credits using the link below.
[00:03:55.760 --> 00:03:59.080] This has been the code report, thanks for watching, and I will see you in the next one.

whisper_print_timings: load time = 130.09 ms
whisper_print_timings: fallbacks = 0 p / 0 h
whisper_print_timings: mel time = 70.35 ms
whisper_print_timings: sample time = 1038.72 ms / 5927 runs ( 0.18 ms per run)
whisper_print_timings: encode time = 292.54 ms / 9 runs ( 32.50 ms per run)
whisper_print_timings: decode time = 59.85 ms / 21 runs ( 2.85 ms per run)
whisper_print_timings: batchd time = 3857.72 ms / 5864 runs ( 0.66 ms per run)
whisper_print_timings: prompt time = 148.24 ms / 1704 runs ( 0.09 ms per run)
whisper_print_timings: total time = 10367.78 ms
ggml_metal_free: deallocating

[lifeiteng]

do you have pytorch + mps result?