local inference mlx

---

id: local-inference-mlx title: Local Inference & MLX tags: [python, mlx, apple-silicon, local-inference, langgraph, tts, stt, audio, video, ocr, benchmark, fastapi, chromadb, lora, distillation] created: 2026-04-11 updated: 2026-05-29 freshness: fresh

Local Inference & MLX

Summary

Fourteen repositories forming a comprehensive local AI toolkit. The majority target Apple Silicon via MLX — inference serving (MLX Local Server), model benchmarking (QwenBench), model distillation (MLX Distillation Explained), vision-language captioning (ScreenLens, Qwen3 VL OCR), image generation (Bonsai Image 4B), video generation (Lance 3B Video, Sulphur 2 Base), speech synthesis (TARS-AI, Audiobook Generator, Supertonic 3 TTS), audio generation (Stable Audio 3), and speech recognition (MLX-YouTubeScribe). DeepSeek-VL2 PDF OCR provides GPU-based local inference on NVIDIA CUDA.

Key claims

• MLX Local Inference Server emulates 5 API surfaces (OpenAI Chat, OpenAI Responses, Azure OpenAI Chat, Azure OpenAI Responses, Anthropic Messages) as a drop-in local replacement with full tool/function calling support. ^src:local-mlx-responsesAPI-server

• TARS-AI runs a complete conversational AI (LLM + TTS) locally using mlx-lm (Llama 3.1 8B 4-bit, ~4.5GB) and mlx-audio Kokoro-82M (~164MB) with configurable personality sliders. ^src:tars-ai

• ScreenLens uses hybrid keyframe detection (SSIM + pHash + HSV) to only capture distinct frames, then captions via Qwen3.5-VL 122B and stores in ChromaDB for semantic video search. ^src:screenlens

• QwenBench MLX benchmarks 6 Qwen 3.5 model sizes (0.8B → 35B) with auto-judge scoring, cost-efficiency ranking, and crash-resilient incremental saves. ^src:qwenbench-mlx

• MLX Distillation Explained demonstrates a 5-step educational distillation pipeline from Claude Sonnet (teacher) to Llama 3.1 8B (student) via LoRA, orchestrated by LangGraph with verification gates. ^src:mlx-distillation-explained

• Audiobook Generator converts books to audiobooks using Qwen3-TTS with parallel generation, LangGraph workflow, QA verification via faster-whisper STT, and checkpoint-based resumption. ^src:audiobook-generator

• Qwen3 VL OCR Script batch-captions images using a local 30B vision-language model with auto-download. ^src:qwen3-vl-python-ocr-script-mlx

• MLX-YouTubeScribe transcribes YouTube videos/playlists offline using Whisper via MLX with optional Voxtral backend and speaker diarization. ^src:mlx-youtubescribe

• DeepSeek-VL2 PDF OCR extracts text from PDFs using the DeepSeek-VL2 Mixture-of-Experts vision-language model on NVIDIA CUDA, supporting both single-file and batch processing with multi-GPU pipeline parallelism. ^src:deepseekvl2-pdf-ocr

• Bonsai Image Ternary 4B uses a 2-bit ternary-quantized 4B image generation model via the PrismML runtime on Apple Silicon; requires full Xcode Metal toolchain (not just CLT). ^src:bonsai-image-ternary-4b-mlx-2bit

• Lance 3B Video BF16 wraps the xocialize/lance-mlx runtime to support both text-to-video (TextToVideoPipeline) and video Q&A/captioning (UnderstandingPipeline.generate_video) locally on Apple Silicon. ^src:lance-3b-video-bf16

• Stable Audio 3 supports four model tiers from CPU-only 433M (Small-Music, Small-SFX) to CUDA 1.4B (Medium, 380s max); streamlined for inference and fine-tuning with a Gradio UI. ^src:stable-audio-3

• Sulphur 2 Base wraps MLXBits/sulphur-2-distill-mlx-q4 for local video generation via the dgrauet/ltx-2-mlx runtime with an optional sulphur-promptenhancer-mlx-q8; follows the same install+GUI pattern as Stable Audio 3. ^src:sulphur-2-base

• Supertonic 3 MLX runs the mlx-community/supertonic-3 TTS model via a vendored supertonic_mlx graph executor using JSON topology + NPZ weights — a non-standard model format compared to GGUF/safetensors. ^src:supertonic-3-mlx

Open questions

• Could the MLX Local Inference Server be unified as the shared backend for TARS-AI, ScreenLens, and other MLX-based tools?
• What is the quality gap between QwenBench-measured model scores and real-world task performance in ScreenLens and TARS-AI?

Related

• agentic-frameworks (DeepAgent Azure CLI uses MLX server as backend, SST-AutoResearch uses MLX)
• ai-security-pentesting (TSLIT uses llama.cpp for local inference)
• developer-tools (GitNexus Fleet complements local tooling)

Articles

• timeline — 2024-07-12. [^src:asics-dethrone-gpus-llm]
• timeline — 2025-12-15. ^src:asics-vs-gpus-grid
• timeline — 2026-01-28. ^src:custom-silicon-gpt-story
• timeline — 2023-05-09. ^src:transforming-crypto-mining-hardware

Sources