-
Notifications
You must be signed in to change notification settings - Fork 0
local inference mlx
Nicolas Cravino edited this page Apr 15, 2026
·
9 revisions
id: local-inference-mlx title: Local Inference & MLX tags: [python, mlx, apple-silicon, local-inference, langgraph, tts, stt, audio, video, ocr, benchmark, fastapi, chromadb, lora, distillation] created: 2026-04-11 updated: 2026-04-13 freshness: fresh
Nine repositories forming a comprehensive local AI toolkit. Eight target Apple Silicon via MLX — inference serving (MLX Local Server), model benchmarking (QwenBench), model distillation (MLX Distillation Explained), vision-language captioning (ScreenLens, Qwen3 VL OCR), speech synthesis (TARS-AI, Audiobook Generator), and speech recognition (MLX-YouTubeScribe). The ninth (DeepSeek-VL2 PDF OCR) provides GPU-based local inference on NVIDIA CUDA for PDF text extraction using the DeepSeek-VL2 MoE model.
- MLX Local Inference Server emulates 5 API surfaces (OpenAI Chat, OpenAI Responses, Azure OpenAI Chat, Azure OpenAI Responses, Anthropic Messages) as a drop-in local replacement with full tool/function calling support. ^src:local-mlx-responsesAPI-server
- TARS-AI runs a complete conversational AI (LLM + TTS) locally using mlx-lm (Llama 3.1 8B 4-bit, ~4.5GB) and mlx-audio Kokoro-82M (~164MB) with configurable personality sliders. ^src:tars-ai
- ScreenLens uses hybrid keyframe detection (SSIM + pHash + HSV) to only capture distinct frames, then captions via Qwen3.5-VL 122B and stores in ChromaDB for semantic video search. ^src:screenlens
- QwenBench MLX benchmarks 6 Qwen 3.5 model sizes (0.8B → 35B) with auto-judge scoring, cost-efficiency ranking, and crash-resilient incremental saves. ^src:qwenbench-mlx
- MLX Distillation Explained demonstrates a 5-step educational distillation pipeline from Claude Sonnet (teacher) to Llama 3.1 8B (student) via LoRA, orchestrated by LangGraph with verification gates. ^src:mlx-distillation-explained
- Audiobook Generator converts books to audiobooks using Qwen3-TTS with parallel generation, LangGraph workflow, QA verification via faster-whisper STT, and checkpoint-based resumption. ^src:audiobook-generator
- Qwen3 VL OCR Script batch-captions images using a local 30B vision-language model with auto-download. ^src:qwen3-vl-python-ocr-script-mlx
- MLX-YouTubeScribe transcribes YouTube videos/playlists offline using Whisper via MLX with optional Voxtral backend and speaker diarization. ^src:mlx-youtubescribe
- DeepSeek-VL2 PDF OCR extracts text from PDFs using the DeepSeek-VL2 Mixture-of-Experts vision-language model on NVIDIA CUDA, supporting both single-file and batch processing with multi-GPU pipeline parallelism. ^src:deepseekvl2-pdf-ocr
- Could the MLX Local Inference Server be unified as the shared backend for TARS-AI, ScreenLens, and other MLX-based tools?
- What is the quality gap between QwenBench-measured model scores and real-world task performance in ScreenLens and TARS-AI?
- agentic-frameworks (DeepAgent Azure CLI uses MLX server as backend, SST-AutoResearch uses MLX)
- ai-security-pentesting (TSLIT uses llama.cpp for local inference)
- developer-tools (GitNexus Fleet complements local tooling)
- timeline — 2024-07-12. ^src:asics-dethrone-gpus-llm
- timeline — 2025-12-15. ^src:asics-vs-gpus-grid
- timeline — 2026-01-28. ^src:custom-silicon-gpt-story
- timeline — 2023-05-09. ^src:transforming-crypto-mining-hardware