feat: Implement native Apple Silicon transcription with MLX-Whisper or whisper.cpp

[davidamacey]

Summary

OpenTranscribe currently uses WhisperX for transcription, which doesn't fully utilize Apple Silicon's GPU capabilities (falls back to CPU on MPS devices). Implementing a native Apple Silicon solution using MLX-Whisper or whisper.cpp would provide significant performance improvements for macOS users with M1/M2/M3 chips.

Current State

• WhisperX is configured to use CPU on Apple Silicon (MPS) devices due to compatibility issues
• Performance is suboptimal compared to native implementations
• Users with expensive Apple Silicon hardware aren't getting full value from their GPU

Proposed Solutions

Option 1: MLX-Whisper (Recommended)

Pros:

• 30-40% faster than whisper.cpp on Apple Silicon
• Native MLX framework designed specifically for Apple Silicon
• Excellent GPU utilization on M1/M2/M3 chips
• Python-based, easier integration with existing codebase
• Active development and Apple support

Cons:

• Only works on Apple Silicon (need to maintain WhisperX for other platforms)
• Smaller community compared to whisper.cpp
• May require significant refactoring of transcription service

Option 2: Lightning-Whisper-MLX

Pros:

• Claims 10x faster than whisper.cpp
• 4x faster than standard MLX-Whisper
• Optimized specifically for Apple Silicon
• Best-in-class performance for macOS

Cons:

• Very new project, stability concerns
• Limited documentation
• May lack features compared to WhisperX

Option 3: whisper.cpp

Pros:

• Cross-platform (works on Apple Silicon, CUDA, CPU)
• Very mature and stable
• 6-7x faster than vanilla Whisper on CPU
• Good Apple Silicon support via Metal
• Could potentially replace WhisperX entirely

Cons:

• Requires C++ integration (more complex)
• 30-40% slower than MLX-Whisper on Apple Silicon
• Would need Python bindings or subprocess calls

Performance Benchmarks (2024)

Based on recent benchmarks:

• MLX-Whisper: ~50% faster than vanilla Whisper on Apple Silicon
• Lightning-Whisper-MLX: 10x faster than whisper.cpp (claimed)
• whisper.cpp: 6-7x faster than vanilla Whisper on CPU, good Metal support

Implementation Plan

Phase 1: Research & Prototype

• Benchmark MLX-Whisper vs whisper.cpp on M1/M2/M3 hardware
• Test feature parity with WhisperX (timestamps, speaker alignment)
• Evaluate integration complexity for each option
• Create proof-of-concept implementation

Phase 2: Architecture Design

• Design abstraction layer for multiple transcription backends
• Create platform detection logic (Apple Silicon vs CUDA vs CPU)
• Plan migration strategy from WhisperX
• Design configuration system for backend selection

Phase 3: Implementation

• Implement chosen solution (likely MLX-Whisper)
• Create fallback mechanism to WhisperX for non-Apple platforms
• Update Docker configurations for Apple Silicon
• Implement proper error handling and logging

Phase 4: Testing & Optimization

• Comprehensive testing on M1, M2, M3 hardware
• Performance benchmarking vs current WhisperX implementation
• Memory usage optimization
• Edge case testing (long files, multiple speakers)

Phase 5: Documentation & Deployment

• Update documentation for Apple Silicon users
• Create migration guide
• Update setup scripts for macOS
• Release with clear performance expectations

Technical Requirements

Core Features to Maintain

• Word-level timestamps
• Speaker diarization compatibility
• Multiple language support
• Batch processing capability
• Progress callbacks for UI updates

New Requirements

• Automatic backend selection based on hardware
• Configurable backend via environment variables
• Performance metrics logging
• Graceful fallback on errors

Code Changes Required

Backend Changes

backend/app/tasks/transcription/
├── base_transcriber.py          # New: Abstract base class
├── whisperx_transcriber.py      # Refactored from whisperx_service.py
├── mlx_transcriber.py           # New: MLX-Whisper implementation
├── whisper_cpp_transcriber.py   # New: Optional whisper.cpp implementation
└── transcriber_factory.py       # New: Factory for backend selection

Configuration Updates

• Add TRANSCRIPTION_BACKEND environment variable
• Add APPLE_SILICON_OPTIMIZATION flag
• Update hardware detection to identify MLX availability
• Add backend-specific configuration options

Docker Updates

• Create Apple Silicon specific Dockerfile variant
• Update docker-compose with platform-specific service definitions
• Ensure proper MLX installation in containers

Acceptance Criteria

• 2x or better performance improvement on Apple Silicon vs current implementation
• No regression in transcription quality
• Seamless fallback to WhisperX on non-Apple hardware
• All existing features continue to work
• Clear documentation for users
• Automated backend selection based on hardware

Performance Targets

• M1 Pro: < 5 minutes for 1-hour audio (currently ~10 minutes)
• M2/M3: < 4 minutes for 1-hour audio
• Memory usage: < 8GB for large model
• GPU utilization: > 80% during transcription

Related Issues

• bug: Docker containers fail to start on ARM64/Apple Silicon Macs #38 - Docker containers fail to start on ARM64/Apple Silicon Macs (resolved)
• [FEATURE] Cross-Platform PyTorch Model Support for CUDA, MPS, and CPU #25 - Cross-Platform PyTorch Model Support

References

• MLX-Whisper GitHub
• Lightning-Whisper-MLX
• whisper.cpp
• Performance comparison article

Labels

enhancement, performance, macos, apple-silicon, transcription, mlx

Priority

High - Significant performance improvement for growing Apple Silicon user base

[TheFutureIsVoluntary]

@davidamacey May I suggest Parakeet MLX as well? I've been getting under 2 minutes for 1 hour audio on M1 Pro with it: https://github.com/senstella/parakeet-mlx

[Edit: Or, add functionality for calling an OpenAI compatible API endpoint - ie, allow changing the URL for the OpenAI API endpoint in the ASR Provider setup. I assume this would give users LOTS of options of what they can use, as many projects are supporting OpenAI compatible API endpoints. I was able to get an API endpoint for parakeet running using mlx-audio from https://github.com/Blaizzy/mlx-audio . I could post audio to transcribe like this:

curl -X POST http://localhost:8000/v1/audio/transcriptions
-F "file=@audio_file.opus"
-F "model=mlx-community/parakeet-tdt-0.6b-v2"
]

[m13v]

for macOS native transcription on Apple Silicon, WhisperKit has been really solid in production for us. the CoreML backend means you get proper GPU scheduling through the ANE (Apple Neural Engine) without fighting Metal directly. one gotcha we hit: the model download and CoreML compilation on first run can take a while, especially for larger models. worth caching the compiled model artifacts so subsequent launches are instant. also if you're doing long recordings, chunked processing is essential because loading the entire audio into memory for a 2 hour file will blow up. we split at silence boundaries detected by a simple energy threshold, not fixed time windows, so you don't cut mid-word.

[m13v]

our TranscriptionService that handles WhisperKit inference on Apple Silicon with the silence-boundary chunking and model caching I mentioned: https://github.com/m13v/fazm/blob/main/Desktop/Sources/TranscriptionService.swift

[davidamacey]

📋 Consolidated implementation plan — supersedes the original plan in this issue's body and all prior proposals

Full plan (detailed, with code references, capability tables, and per-item definitions of done):
👉 https://gist.github.com/davidamacey/53e5ac3de9d3f001bfce297534364c60

After a full architecture review, the question this issue originally asked ("MLX-Whisper vs whisper.cpp vs WhisperKit?") turns out to be the third problem, not the first. This comment replaces the issue body's implementation plan; the gist is the authoritative work breakdown for implementation.

Root cause analysis (verified against the codebase)

1. Docker on macOS runs a Linux VM — Metal/MPS/ANE are unreachable from containers, period. Today transcription and diarization run CPU-only on Macs. The TORCH_DEVICE=mps that opentr.sh exports is consumed by no compose file (dead config), and the docs promise MPS acceleration that never executes in Docker. No in-container engine swap can fix this — native acceleration requires a host-native process.
2. A latent arm64 release blocker exists: published images are multi-arch (amd64+arm64), but a 2026-05-29 Dockerfile.prod change added an unconditional onnxruntime-gpu==1.26.0 install that has no linux/arm64 wheel — the next arm64 image rebuild will fail. Plus the stack's default footprint (~15–20 GB; OpenSearch 4 GB locked heap + preloading workers) exceeds Docker Desktop's default VM.
3. Only then does engine choice matter — and it becomes pluggable rather than either/or.

The plan: three tracks

Track A — full-stack ARM64 parity (Docker best practices). Fix the arm64 build blocker (arch-gate the onnxruntime-gpu layer), defuse a CI workflow that would clobber the multi-arch manifest, ship an auto-loaded docker-compose.mac.yml resource overlay (OpenSearch heap 4 GB→2 GB, trimmed worker concurrency, longer health-check budgets), a Docker-on-macOS guide (VM sizing, OrbStack, VirtioFS/named volumes), BSD/bash-3.2 portability fixes in auxiliary scripts, and a macOS CI job.

Track B — Native ASR Sidecar (the M-chip acceleration). A small, stateless HTTP service running natively on the macOS host (loopback-only 127.0.0.1:5190, bearer-token auth, zero stack credentials), reached from the dockerized pipeline via host.docker.internal through the existing ASR-provider seam as a new mac_native provider. Full pipeline parity: word timestamps, SPEAKER_XX diarization via the MPS-optimized PyAnnote fork, native 256-d speaker embeddings, boundary smoothing, redaction, search indexing — identical downstream behavior to the Linux/CUDA path.

Pluggable engine matrix (model IDs encode the engine):

Engine	Models	Languages	Word timestamps	Status
mlx_whisper (primary)	tiny → large-v3-turbo (mlx-community)	100+	✅	beta
parakeet (NVIDIA TDT 0.6B v2, MLX)	parakeet-tdt-0.6b-v2	EN	✅	beta
moonshine (Useful Sensors)	tiny/base	EN	verify at impl	experimental (stretch)
speechanalyzer (Apple native, macOS 26+)	system model	per-locale	✅ (audioTimeRange)	first-class code, beta until verified on physical hardware
WhisperKit/Argmax	—	—	—	deferred engine; usable today via Track C

Diarization is engine-independent (sidecar-side PyAnnote-on-MPS + word→speaker assignment), so every engine — including Apple's — gets full speaker support.

Track C — BYO OpenAI-compatible endpoint + security/privacy. A new openai_compatible provider with user-configurable base URL (the plumbing was ~90 % built; it needs the base_url threaded into the client + SSRF guardrails), plus 14 security/privacy work items: sidecar authn (fail-closed token), structural + tested no-egress guarantees, supply-chain pinning (hashed lockfiles, pinned HF model revisions), privacy-tier labeling in the UI (local_stack / local_host / cloud_byok / remote_custom — a BYO endpoint is never labeled "local"), missing ASR-config audit events, rate limits, and contract tests. Cloud-seam compliance is preserved via two new capability keys — no cloud-specific logic enters this repo.

Acceptance gates (kept from this issue, plus quality gates)

• ≥ 2× end-to-end speedup vs the in-Docker CPU baseline on the same Mac
• 1 h audio < 5 min on M1-Pro-class hardware
• WER within 1 % absolute of the Linux large-v3-turbo baseline; DER within 2 % absolute (no quality regression)
• Graceful CPU fallback when the sidecar is absent; zero impact on Linux/CUDA deployments; no DB migration required anywhere

Responses to the suggestions in this thread

• @TheFutureIsVoluntary — both adopted. Parakeet MLX is a first-wave engine (Track B, phase B3); your OpenAI-compatible-endpoint idea is Track C item C1, and the mlx-audio setup you described will be a documented configuration for it.
• @m13v — thanks for the production WhisperKit insights. The CoreML compiled-artifact caching and long-file chunking concerns directly shaped the design (model warm-cache in the sidecar; silence-boundary chunking noted as a long-file optimization). WhisperKit itself is a deferred sidecar engine, but it's usable immediately via Track C by pointing the BYO provider at whisperkit-cli's OpenAI-compatible server.

Future direction (separate issue)

The sidecar is deliberately built as a backend-independent, versioned-API local inference core. Issue #283 tracks the cross-platform standalone desktop app (Tauri shell, SQLite + sqlite-vec local library) that embeds this same core — macOS first, then Windows/Linux — with its own plan gist. Nothing in this issue blocks on it.

---

Implementation is broken into independently landable phases sized for individual PRs — see the gist for per-phase files, interfaces, tests, and definitions of done. Work targets the v0.5.x cycle.