Install local LLM on new MacBook Pro M5 Max 128GB

[wphillipmoore]

Install Local LLM on New MacBook Pro

Context

A new MacBook Pro 16" M5 Max (fully maxed configuration) with 128GB unified memory is arriving in approximately two weeks (~2026-04-24). This is the top-of-the-line configuration with every performance option selected — 18-core CPU, 40-core GPU, 128GB unified memory, 614 GB/s memory bandwidth.

This machine is capable of running 70B parameter models at high quantization levels (Q6/Q8) with an estimated 10-16 tok/s throughput, and 14B-30B models at 30-75 tok/s. This will be our primary local LLM platform.

This issue covers the initial setup of a local LLM environment on that hardware for development testing and iteration.

Hardware specs (confirmed)

• Apple M5 Max (18-core CPU, 40-core GPU)
• 128GB unified memory
• 614 GB/s memory bandwidth
• Thunderbolt 5 ports

Tasks

• Install Ollama (brew install ollama)
• Pull initial model(s) — evaluate options and select based on research below
• Verify OpenAI-compatible API is accessible at localhost:11434/v1/
• Evaluate LM Studio as an alternative (MLX backend, 20-40% faster on Apple Silicon)
• Benchmark selected models: tokens/sec, memory usage, quality of structured JSON output
• Document findings: which models work well, recommended defaults, known limitations

Model selection research (to be expanded)

Initial candidates to evaluate for our use case (structured JSON output from research prompts):

Model	Size	Expected fit	Notes
Llama 3.1 70B	~40GB at Q4, ~55GB at Q6	Comfortable	General purpose, well-tested
Qwen 2.5 72B	Similar to Llama 70B	Comfortable	Strong at structured output
DeepSeek V2.5	~16GB at Q4 (MoE, 21B active)	Easy	Fast, MoE architecture
Mistral Large	Varies	TBD	Evaluate availability in GGUF/MLX
Smaller models (7B-14B)	Trivial	Trivial	For fast iteration when quality is irrelevant

Model selection should prioritize: (1) reliable structured JSON output, (2) reasonable instruction following, (3) speed for iteration. Output quality is secondary — this is for pipeline testing, not publication.

Integration with Diogenes pipeline

Once the provider-agnostic abstraction (#69) is implemented:

• Configure LiteLLM to route to local Ollama
• Run existing research pipeline end-to-end against local model
• Document what breaks and what works
• Establish a --profile local configuration for rapid iteration

Depends on

• New hardware arrival (~2026-04-24)
• Provider-agnostic LLM abstraction: migrate from direct Anthropic SDK to LiteLLM #69 (provider-agnostic API) for pipeline integration
• Related: Local LLM environment for development testing and iteration #70 (local LLM environment strategy)

[wphillipmoore]

Research: Local LLM Software Options for macOS on MacBook Pro M4 Max 128GB

The Big Three

1. Ollama

• What: CLI-first LLM runner, invisible inference infrastructure
• Setup: brew install ollama && ollama run llama3 — chatting in under 2 minutes
• Interface: Terminal only. Pair with Open WebUI (Docker) for ChatGPT-like GUI
• API: OpenAI-compatible at localhost:11434/v1/
• Apple Silicon: Metal GPU + MLX backend (v0.19, March 2026)
• Resources: Server uses ~1GB idle. Models auto-unload after 5 min. Can stop entirely
• Maturity: Most widely adopted local LLM tool. Largest ecosystem
• Cost: Free, open source (MIT)
• Best for: Programmatic API use, backend for other tools, ecosystem integration

2. LM Studio

• What: Desktop GUI app with polished ChatGPT-like interface
• Setup: Download app, browse/download models in-app, no terminal needed
• Interface: Full GUI — chat playground, model browser, parameter controls
• API: Built-in OpenAI-compatible local server
• Apple Silicon: Native MLX backend — 30-50% faster than GGUF on same hardware
• Resources: Only runs when open. Zero background impact
• Maturity: Mature, actively developed. Closed source
• Cost: Free for personal and commercial use
• Best for: Visual model exploration, chat interface, non-terminal users

3. Jan (jan.ai)

• What: Open-source desktop app for both local and cloud models
• Setup: Download app, pick model, start chatting
• Interface: Clean GUI. Switches between local models and cloud APIs (OpenAI, Claude, Groq)
• API: OpenAI-compatible server built in
• Maturity: Younger but actively developed. Apache 2.0 — fully auditable
• Cost: Free, open source
• Best for: Single app for local + cloud, open-source preference

Worth Knowing About

Tool	What	Best For
mlx-lm (Apple)	Python package, fastest Mac inference (20-30% faster than llama.cpp). No GUI/API	ML research, fine-tuning. Ollama/LM Studio use MLX under the hood
Msty	Desktop app with parallel model comparison. Native MLX. Apple-featured	Evaluating and comparing models visually
GPT4All	Desktop app with LocalDocs (point at folder, ask questions)	Document-based Q&A workflows
Llamafile (Mozilla)	Single-file executable, bundles model + engine. Double-click to run	Zero-install demos, portability
apfel	Wraps Apple's built-in ~3B model in macOS Tahoe. brew install apfel	Quick scripting, instant zero-download testing

Don't Bother

• LocalAI: Overkill for personal use, designed for multi-user servers. Apple Silicon secondary
• Raw llama.cpp: Ollama and LM Studio wrap it with better ergonomics
• vLLM: NVIDIA GPU-first. Not suited for Mac

Daily-Driver Reality

With 128GB unified memory, a 70B model at Q6 uses ~55GB, leaving 50-60GB free for browsers, IDEs, and normal work. All desktop apps run only when opened. Ollama is lightweight idle (~1GB) and auto-unloads models.

Recommendation for Our Use Case

Start with Ollama — primary use is programmatic (Diogenes pipeline integration via LiteLLM). Add LM Studio alongside for visual model exploration and chat when convenient. Both can coexist on the same machine.

Sources

• Ollama adopts MLX — 9to5Mac
• Local LLMs Apple Silicon 2026 — SitePoint
• Ollama vs LM Studio vs Jan Compared — DEV Community
• LM Studio MLX Guide — Markaicode
• Jan.ai Documentation
• apfel — Your Mac Already Has AI
• MLX GitHub — Apple

[wphillipmoore]

Update — add Gemma 4 to the candidate set (verified 2026-04-26)

Google released Gemma 4 on April 2, 2026 under the Apache 2.0 license — the first Gemma release without Google's custom Gemma Terms of Use restrictions. This changes the candidate calculus for this issue.

Sizes that fit on the M5 Max 128GB

Model	Parameters	Quantized fit	Notes
Gemma 4 31B Dense	31B	Q6 ~25GB / Q8 ~33GB	Top-capability open-weight; fine-tune-friendly
Gemma 4 26B MoE	26B total, 3.8B active per token	Q6 ~21GB	Iteration-tier sweet spot — speed of a small model with capability of a much larger one
Gemma 4 E4B	~4B effective	Trivial	Very fast iteration
Gemma 4 E2B	2.3B effective	Trivial	Smallest tier

Capabilities (Hugging Face / Google blog)

• 256K context window
• Native vision + native audio (multimodal in the same weights)
• 140+ languages

Apple Silicon / M5 Max specifics

• MLX backend support landed in Ollama 0.20. Ollama auto-uses MLX on Apple Silicon — no manual config — and leverages M5/M5 Pro/M5 Max GPU Neural Accelerators for both TTFT and tok/s.
• Reported throughput without MLX (i.e., llama.cpp baseline) on M5 Max: ~15 tok/s on 31B Dense, ~75 tok/s on 26B MoE. With MLX should improve.
• Known issues to track:
  • Ollama issue #15368 — Gemma 4 on M5 Max specifically: FA hang, /v1 streaming reasoning field, MLX-not-supported reports. Multi-symptom; partially open.
  • LM Studio bug #1741 — MLX runtime fails to load Gemma 4 26B with "Model type gemma4 not supported". Open.

So Ollama is the lower-friction local-serving path on M5 Max for Gemma 4 today; LM Studio MLX is broken until that bug closes — drop or de-prioritize it from the "Evaluate LM Studio as alternative" task in this issue until upstream fixes.

Recommended changes to this issue

1. Add Gemma 4 31B Dense and 26B MoE to the candidate model table — flag both as highest priority to evaluate first because of the Apache 2.0 license advantage over Llama 3.x and Qwen 2.5 (those have license terms that constrain commercial reuse paths).
2. Update the "Install Ollama" task: pin to Ollama ≥ 0.20 so the MLX backend is available.
3. Add a benchmarking sub-task specific to Gemma 4: validate structured JSON output on the 31B Dense and confirm whether the 26B MoE's per-token compute advantage actually translates to faster pipeline iteration in practice.
4. Defer the LM Studio evaluation task pending LM Studio bug #1741.
5. Track Ollama issue #15368 for M5 Max-specific Gemma 4 stability before any production-pipeline reliance.

Sources

• Welcome Gemma 4: Frontier multimodal intelligence on device — Hugging Face
• Gemma 4: Byte for byte, the most capable open models — Google Blog
• Gemma 4 — Google DeepMind
• Gemma 4: Expanding the Gemmaverse with Apache 2.0 — Google Open Source Blog
• Ollama is now powered by MLX on Apple Silicon in preview — Ollama Blog
• Gemma 4 on Apple Silicon: All Four Models Compared — SudoAll
• Running Gemma 4 with Ollama on Mac OS with Apple Silicon — Medium
• Gemma 4 - How to Run Locally — Unsloth Documentation