Repository navigation
Performance of SGLang-Omni on Apple Silicon Mac #2158
Replies: 15 comments 2 replies
|
Tested MOSS-Transcribe-Diarize on a 24 GB Apple M5 MacBook Pro with the implementation in PR #1989.
Configuration: BF16, concurrency 1, greedy decoding, Long-form quality and execution
The full Meanwhile split measures transcription quality and performance. The 90-minute/30-minute runs cap decoding at one token and therefore validate only encoder/prefill execution and memory behavior. The 128K and 32K labels are maximum token context lengths, not memory sizes or audio-duration limits. The implementation uses encoder-window micro-batching and embedding-aware chunked prefill to prevent long-audio OOMs; Torch MPS additionally uses a static KV cache. No OOM occurred and macOS memory pressure remained healthy. RSS does not fully capture Metal unified-memory allocations; MLX worker footprint was about 2.6 GB ready / 4.2 GB warm, while Torch MPS was about 7.6 GB ready / 7.7 GB warm and retained about 13 GB after the longest run. Reproduction scripts and commands: public benchmark Gist. |
|
Tested Qwen3-ASR-0.6B with MLX 4-bit on an Apple M4 Pro MacBook Pro.
Ran Across repeats, throughput ranged from 11.979–12.211 samples/s, and WER remained 1.91%. Pooled p95 latency was 100.3 ms. Headline performance figures are averages across the three repeats.
|
|
M1 Pro 32 GB host results (matched official checkpoints where noted). Short-clip latency rows are not SeedTTS EN full-corpus numbers — those are called out separately for Fun-ASR.
Env (shared): Source write-ups
Happy to re-run any row on a fresher |
|
@yeahdongcn thanks for opening this discussion — useful to collect data. I have a little concern that our scripts may use different settings. If we cannot align with each other, the data might not be comparable. Should we standardize a default Apple Silicon recipe for this thread — e.g. pin: |
|
Tested Qwen3-ASR-0.6B with native MLX 4-bit on an Apple M4 Pro with 24 GB unified memory. Benchmark Results
Ran with client concurrency 1, Throughput ranged from 11.588–11.620 samples/s, and WER remained 1.9143% across all three runs. Pooled p95 latency was 104.72 ms. Headline performance figures are averages across the three measured runs.
|
|
Tested MOSS-TTS Local v1.5 on a 24 GB Apple M5 MacBook Pro with the implementation in PR #1991. Benchmark Results
MLX delivered 78.81% higher request throughput and 44.14% lower mean latency than Torch MPS. For this TTS benchmark, RTF is inference time / generated-audio duration, so lower is better. Quality and memory
All 2,176/2,176 WAVs were readable. WER, speaker similarity, and UTMOS each evaluated 1,088/1,088 samples per backend with zero skipped samples. Both output sets used the same pinned Qwen3-ASR-1.7B MLX evaluator, so the TTS backend comparison is not confounded by different ASR backends. RSS alone misses Metal allocations; the ~13/~17 GB figures are steady Reproduction details
The complete benchmark summary and implementation details are also recorded in PR #1991. |
|
Tested on a 24 GB Apple M5 Max MacBook Pro. Benchmark Results
Result Summary: MLX comes out ~3.9x faster here, in the same direction as the M1/M2/M4 numbers earlier in this thread. Note the two rows differ in quantization (4-bit vs BF16), not just backend, matching each path's realistic default config rather than an apples-to-apples precision comparison. Test Env: SGLang v0.5.19, MLX 0.32.2, mlx-lm 0.31.3, Torch 2.13.0. Matched-checkpoint Qwen3-ASR-0.6B on both backends, single request (--asr.engine.max_running_requests 1), 2 warmups + 10 measured requests, tests/data/query_to_cars.wav |
|
Tested Qwen3-ASR-0.6B on an Apple M6 Mac mini (24 GB unified memory).
Protocol for both runs: concurrency 1, The MLX run used Environment: Python 3.12.15, SGLang 0.5.21, MLX 0.32.3, mlx-lm 0.32.0, PyTorch 2.13.0, Transformers 5.12.1. The dataset was |
|
Tested Qwen3-ASR-0.6B through SGLang-Omni on an Apple M4 Pro MacBook Pro with 14 CPU cores, 20 GPU cores and 24 GB unified memory, running macOS 27.0.1 (26A434).
Ran Headline performance figures are averages across the three measured passes. Pooled p95 latency was 104.40 ms for MLX and 394.17 ms for MPS; WER was unchanged across repeats within each backend. These configurations differ in weight precision and runtime defaults. Background applications remained open, with existing swap and some swap-in activity; neither measured interval showed an increase in swap-outs. Reproduction scripts, pinned versions, per-pass results, resource measurements and functional checks: GitHub Gist. |
|
Tested Qwen3-ASR-0.6B on an Apple M5 Max Mac Studio (18-core CPU, 32-core GPU, 36 GB unified memory).
Protocol for both: Both documented paths needed one change at this commit:
Comparability: the rows differ in weight precision as well as backend. Compared with earlier rows, this run uses a newer stack and, for MPS, the extra flag above. The Claude desktop app stayed open during both runs, and no thermal warnings were recorded. Environment: Python 3.12.16, SGLang 0.5.22, MLX 0.32.3 (native runtime), PyTorch 2.14.1, Transformers 5.19.0.
CommandsSGLANG_OMNI_VENV="$HOME/voxt-dev-setup/venv-apple" ./install.sh --non-interactive
source ~/voxt-dev-setup/venv-apple/bin/activate
python -m benchmarks.dataset.prepare --dataset seedtts
# MLX
PYTHONPATH="$PWD" python resample_seedtts_16k.py ~/voxt-dev-setup/bench/seedtts-en-16k
Voxt/build/omni-runtime/bin/qwen3_asr_server \
--model-kind qwen3_asr \
--model-directory ~/voxt-dev-setup/models/Qwen3-ASR-0.6B-4bit \
--port 8000
python -m benchmarks.eval.benchmark_asr_seedtts \
--port 8000 \
--meta ~/voxt-dev-setup/bench/seedtts-en-16k/meta.lst \
--concurrencies 1 --repeats 3 --warmup \
--model-path mlx-community/Qwen3-ASR-0.6B-4bit \
--model-revision 313d850181767edf09f00a9c289becca70e58cd0 \
--quantization 4bit \
--max-running-requests 1 \
--disable-resource-monitor \
--output ~/voxt-dev-setup/bench/mlx.json
# Torch/MPS
export DYLD_LIBRARY_PATH="$(brew --prefix ffmpeg@7)/lib${DYLD_LIBRARY_PATH:+:$DYLD_LIBRARY_PATH}"
sgl-omni serve \
--model-path "$(hf download Qwen/Qwen3-ASR-0.6B --revision 5eb144179a02acc5e5ba31e748d22b0cf3e303b0 --quiet)" \
--model-name Qwen/Qwen3-ASR-0.6B \
--asr.engine.max_running_requests 1 \
--asr.engine.enable_multimodal false \
--port 8000
python -m benchmarks.eval.benchmark_asr_seedtts \
--port 8000 \
--concurrencies 1 --repeats 3 --warmup \
--model-path Qwen/Qwen3-ASR-0.6B \
--model-revision 5eb144179a02acc5e5ba31e748d22b0cf3e303b0 \
--dtype bfloat16 \
--max-running-requests 1 \
--disable-resource-monitor \
--output ~/voxt-dev-setup/bench/mps.json
|
|
Native TTS on an Apple M4 MacBook (10-core GPU, 16 GB unified memory), from the #2888 Layer 2 validation. Everything ran on AC power with no other GPU jobs. The SeedTTS runs use the first 100 samples per split, scored by native Qwen3-ASR-1.7B-8bit, at concurrency 1.
Spawn to first audio, cold / warm (medians of 5), with idle footprint: CosyVoice3 3.73 / 3.16 s at 2.82 GB, and Ming 15.5 / 14.5 s at 9.75 GB. The method and the details are in #2888 (comment). |
|
Tested Qwen3-ASR-0.6B on an Apple M4 Pro MacBook Pro (14-core CPU, 20-core GPU, 48 GB unified memory), at the same commit and with the same protocol and commands as the M5 Max Mac Studio post above.
|
|
Native Qwen3-TTS CustomVoice (#3139) on an Apple M4 MacBook Air (8-core GPU, 16 GB unified memory). The SeedTTS runs use the first 100 samples per split, no reference audio (EN: Ryan, ZH: Vivian, as in the cookbook; no
Commands: Other checks at the same commit:
Caveat: this Mac had other apps open and about 10 to 11 GB of swap in use after the runs, so the speed numbers are likely pessimistic. WER/CER are not affected by that. |
|
Native MiniMax Music 3 on an Apple M5 Max Mac Studio (18-core CPU, 40-core GPU, 128 GB unified memory), with stage offload (#2008) on and off, from the #2888 Layer 2 validation. On AC power, MLX 0.32.3, checkpoint
Seeded outputs are byte-identical between the two modes at every length. Offload adds 0.15–0.45 s of weight restore per request with the checkpoint in the page cache. Method: #3411's |
|
Ming-omni-tts 4-bit, native against mlx-audio, on an Apple M5 Max Mac Studio (40-core GPU, 128 GB unified memory). This is a #2888 Layer 2 validation item. Both frameworks run on MLX 0.32.4 with the same checkpoint (
Spawn to first audio, cold / warm (medians of 5), with idle footprint: native 1.71 / 1.17 s at 10.02 GB, and mlx-audio 2.42 / 1.84 s at 10.97 GB. An earlier, noisier EN run gave native RTF 0.230 against mlx-audio 0.255. The method, the request settings and the commands are in #2888 (comment). |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
If you have an Apple Silicon Mac and would like to share your SGLang-Omni performance, please post your results below. To make the results reproducible and easier to compare, please include your Mac chip / memory configuration and the exact SGLang-Omni commit used for the test.
Benchmark Results
Feel free to include the benchmark command or other relevant details below the table.
All reactions