qwenbench mlx.stub

---

id: qwenbench-mlx name: QwenBench MLX repo_path: ~/Documents/sw30labs/repos/qwenbench-mlx remote_url: https://github.com/sw30labs/qwenbench-mlx.git primary_language: Python framework: mlx-lm ingested: 2026-04-11 last_commit_date: 2026-04-11 category: local-inference-mlx stacks: [mlx]

QwenBench MLX

Purpose

Single-script benchmarking suite for the entire Qwen 3.5 model family (0.8B → 35B) on Apple Silicon via MLX. Measures throughput, memory, response quality with auto-judge. Crash-resilient incremental saves.

Architecture

• Single Script: qwen_text.py with interactive TUI (questionary)
• Benchmark Runner: Iterates across model sizes × prompt sets
• Auto-Judge: Largest model (35B) scores quality of smaller models' responses
• Cost-Efficiency Scoring: Ranks models by throughput-per-quality ratio
• Checkpoint System: JSONL-based incremental saves, resumable after crashes

Models Benchmarked

• Qwen 3.5 0.8B
• Qwen 3.5 1.8B
• Qwen 3.5 3B
• Qwen 3.5 7B
• Qwen 3.5 14B
• Qwen 3.5 35B

Key Metrics

• Tokens per second (throughput)
• Memory usage (peak, average)
• Response quality (auto-judged)
• Cost-efficiency rank

Tags

python, mlx, apple-silicon, local-inference, benchmark, llm

Cross-references

• tars-ai — uses Qwen models
• screenlens — uses Qwen models