-
Notifications
You must be signed in to change notification settings - Fork 0
sttbench.stub
id: sttbench name: STTbench repo_path: ~/Documents/sw30labs/repos/STTbench remote_url: https://github.com/sw30labs/STTbench.git primary_language: Python framework: OpenAI SDK + MLX Whisper category: local-inference-mlx stacks: [mlx, converter, cli] ingested: 2026-07-09 last_commit_date: 2026-06-11
Benchmarks speech-to-text on the three axes that decide the bill: cost, speed, and accuracy. Runs the same audio through hosted OpenAI (gpt-4o-transcribe, gpt-4o-mini-transcribe) and local MLX Whisper, then emits a single self-contained report.md with an embedded chart, a Winners callout, and a WER table broken into substitution/deletion/insertion.
Single-input, three-backend runner. API legs compute real cost (duration × per-minute rate); the local leg estimates cost against owned hardware. Word-level accuracy is reported as WER with a Sub/Del/Ins split so verbosity/hallucination (insertions) is distinguished from genuine mishearing (substitutions). Local Whisper runs on Apple MLX with no OMLX server required.
matplotlib (cost sweep + plot), mutagen (audio duration, ffprobe fallback), openai (API SDK), python-dotenv, mlx-whisper (local Whisper via MLX, downloads model on first run).
No CI workflows. Conda env sttbench; start.sh launcher.
- qwenbench-mlx — sibling MLX benchmarking harness
- mlx-youtubescribe — shared local-Whisper transcription
- Tags: [python, mlx, apple-silicon, stt, benchmark, openai, local-inference, cli]