Skip to content

local inference mlx

Jump to bottom
Nicolas Cravino edited this page Jul 10, 2026 · 9 revisions

id: local-inference-mlx title: Local Inference & MLX tags: [python, mlx, apple-silicon, local-inference, langgraph, tts, stt, audio, video, ocr, benchmark, fastapi, chromadb, lora, distillation] created: 2026-04-11 updated: 2026-07-09 freshness: fresh

Local Inference & MLX

Summary

Nineteen repositories forming a comprehensive local AI toolkit. The majority target Apple Silicon via MLX — inference serving (MLX Local Server), speculative-decoding acceleration (DFlash × MLX), model benchmarking (QwenBench), model distillation (MLX Distillation Explained), vision-language captioning (ScreenLens, Qwen3 VL OCR), image generation (Bonsai Image 4B), video generation (Lance 3B Video, Sulphur 2 Base, LTX 2.3 MLX, LongCat Video Avatar), speech synthesis (TARS-AI, Audiobook Generator, Supertonic 3 TTS), song and audio generation (Stable Audio 3, ACE-Step 1.5), and speech recognition (MLX-YouTubeScribe). DeepSeek-VL2 PDF OCR provides GPU-based local inference on NVIDIA CUDA.

Key claims

  • MLX Local Inference Server emulates 5 API surfaces (OpenAI Chat, OpenAI Responses, Azure OpenAI Chat, Azure OpenAI Responses, Anthropic Messages) as a drop-in local replacement with full tool/function calling support. ^src:local-mlx-responsesAPI-server

  • TARS-AI runs a complete conversational AI (LLM + TTS) locally, with both models served by a local OMLX server over the OpenAI protocol (/v1/chat/completions + /v1/audio/speech, Qwen3-TTS Voice Design) and configurable personality sliders; the July 2026 refactor moved it off in-process mlx-lm/mlx-audio. ^src:tars-ai

  • ScreenLens uses hybrid keyframe detection (SSIM + pHash + HSV) to only capture distinct frames, then captions via Qwen3.5-VL 122B and stores in ChromaDB for semantic video search. ^src:screenlens

  • QwenBench MLX benchmarks 6 Qwen 3.5 model sizes (0.8B → 35B) with auto-judge scoring, cost-efficiency ranking, and crash-resilient incremental saves. ^src:qwenbench-mlx

  • DFlash × MLX points DFlash block-diffusion speculative decoding at Qwen3.6-27B-bf16 on Apple Silicon, measuring ≈3.37× throughput (12.6 → 42.3 tok/s on an M3 Ultra) with output byte-for-byte identical to plain decoding — the drafter lands a mean 7.33 tokens per verify step, and the gains grow with target size. ^src:dflash-mlx-trial

  • MLX Distillation Explained demonstrates a 5-step educational distillation pipeline from Claude Sonnet (teacher) to Llama 3.1 8B (student) via LoRA, orchestrated by LangGraph with verification gates. ^src:mlx-distillation-explained

  • Audiobook Generator converts books to audiobooks using Qwen3-TTS with parallel generation, LangGraph workflow, QA verification via faster-whisper STT, and checkpoint-based resumption. ^src:audiobook-generator

  • Qwen3 VL OCR Script batch-captions images using a local 30B vision-language model with auto-download. ^src:qwen3-vl-python-ocr-script-mlx

  • MLX-YouTubeScribe transcribes YouTube videos/playlists offline using Whisper via MLX with optional Voxtral backend and speaker diarization. ^src:mlx-youtubescribe

  • DeepSeek-VL2 PDF OCR extracts text from PDFs using the DeepSeek-VL2 Mixture-of-Experts vision-language model on NVIDIA CUDA, supporting both single-file and batch processing with multi-GPU pipeline parallelism. ^src:deepseekvl2-pdf-ocr

  • Bonsai Image Ternary 4B uses a 2-bit ternary-quantized 4B image generation model via the PrismML runtime on Apple Silicon; requires full Xcode Metal toolchain (not just CLT). ^src:bonsai-image-ternary-4b-mlx-2bit

  • Lance 3B Video BF16 wraps the xocialize/lance-mlx runtime to support both text-to-video (TextToVideoPipeline) and video Q&A/captioning (UnderstandingPipeline.generate_video) locally on Apple Silicon. ^src:lance-3b-video-bf16

  • Stable Audio 3 supports four model tiers from CPU-only 433M (Small-Music, Small-SFX) to CUDA 1.4B (Medium, 380s max); streamlined for inference and fine-tuning with a Gradio UI. ^src:stable-audio-3

  • Sulphur 2 Base wraps MLXBits/sulphur-2-distill-mlx-q4 for local video generation via the dgrauet/ltx-2-mlx runtime with an optional sulphur-promptenhancer-mlx-q8; follows the same install+GUI pattern as Stable Audio 3. ^src:sulphur-2-base

  • Supertonic 3 MLX runs the mlx-community/supertonic-3 TTS model via a vendored supertonic_mlx graph executor using JSON topology + NPZ weights — a non-standard model format compared to GGUF/safetensors. ^src:supertonic-3-mlx

  • STTbench benchmarks speech-to-text across cost, speed, and accuracy — the same audio through hosted OpenAI (gpt-4o-transcribe, gpt-4o-mini-transcribe) and local MLX Whisper — and reports WER split into substitution/deletion/insertion, so hallucinated insertions are distinguished from genuine mishearing. ^src:sttbench

  • ACE-Step 1.5 MLX wraps the ACE Studio/StepFun text-to-song model on Apple Silicon, generating full songs with vocals from a style prompt + lyrics; follows the local-mlx-wrapper-pattern (GUI port 8789) with OMLX-powered prompt/lyrics helpers. ^src:ace-step-1-5-mlx

  • LTX 2.3 MLX runs Lightricks LTX 2.3 from native MLX weights (dgrauet/ltx-2.3-mlx-q4) with text-, image-, and audio-to-video modes, sharing the ltx-2-mlx runtime with Sulphur 2 Base (GUI port 8788). ^src:ltx-2-3-mlx

  • LongCat Video Avatar 1.5 MLX drives a talking-avatar video model (mlx-community/LongCat-Video-Avatar-1.5-q4-dmd-merged) from a portrait + audio + prompt, wrapping the xocialize/longcat-avatar-mlx runtime (~24 GB q4, GUI port 8796). ^src:longcat-video-avatar-1-5-mlx

Open questions

  • Could the MLX Local Inference Server be unified as the shared backend for TARS-AI, ScreenLens, and other MLX-based tools?
  • What is the quality gap between QwenBench-measured model scores and real-world task performance in ScreenLens and TARS-AI?

Related

Articles

Sources

Home | Index | Sitemap | Sitemap-Stacks | SCHEMA | Log

Clone this wiki locally