mlx_lm.server causes macOS kernel panic (IOGPUMemory crash) due to unbounded memory growth

[BrunoCerberus]

Summary

Running mlx_lm.server with Qwen3-Coder-30B-A3B-Instruct-8bit on a Mac Studio M3 Ultra (96GB) caused a full macOS kernel panic after the KV cache grew unboundedly during an agentic coding session (~58k+ tokens context). The system crashed and rebooted — not a process termination, but a kernel-level panic in IOGPUMemory.cpp.

System Information

• Hardware: Mac Studio (Mac15,14) — Apple M3 Ultra, 28 cores (20P + 8E), 60 GPU cores, 96 GB unified memory
• macOS: 26.3 (25D125)
• Kernel: Darwin 25.3.0 (xnu-12377.81.4~5/RELEASE_ARM64_T6031)
• Python: 3.12
• mlx-lm: 0.30.6
• mlx: 0.30.6

Model

mlx-community/Qwen3-Coder-30B-A3B-Instruct-8bit (~32GB on disk)

Steps to Reproduce

1. Start mlx_lm.server:

   mlx_lm.server --model mlx-community/Qwen3-Coder-30B-A3B-Instruct-8bit --port 8080 --max-tokens 64000

2. Use an agentic coding tool (e.g., OpenCode) that makes repeated /v1/chat/completions requests with growing conversation context
3. Let the context grow to ~58k+ tokens over multiple tool-call round trips
4. System crashes — full kernel panic, forced reboot

Kernel Panic Details

Panic string:

panic(cpu 18 caller 0xfffffe00428d679c): "completeMemory() prepare count underflow" @IOGPUMemory.cpp:550

Memory status at crash:

Metric	Value
Wired memory	80.14 GB (83.5% of 96GB)
Active memory	7.12 GB
Free memory	0.01 GB (essentially zero)
memoryPressure	false (system didn't detect pressure!)

Python/MLX process memory at crash:

PID 30289 (Python): residentMemoryBytes = 83,235,119,080 (~83.23 GB)

The MLX server process was consuming 83.23 GB of the 96 GB system at the time of the kernel panic.

Root Cause Analysis

1. mlx_lm.server starts by wiring ~75% of RAM via mx.set_wired_limit(max_recommended_working_set_size) — on a 96GB system, that's ~72GB. Wired memory cannot be swapped to disk.

2. The KV cache grows without bound — mlx_lm.server does not support --max-kv-size (see Feature Request: Add max-kv-size Support to MLX HTTP Server #615), so as the agentic tool sends longer and longer conversations, the KV cache keeps growing.

3. Memory breakdown at crash:

   • Model weights (8-bit): ~32 GB
   • KV cache (58k+ tokens, unbounded): ~10-20 GB
   • MoE activations (128 experts): ~3-5 GB
   • Metal buffer cache: ~2-4 GB
   • Total MLX process: ~83 GB

4. macOS couldn't recover — because the memory was wired (locked in RAM), macOS's Jetsam OOM killer couldn't reclaim it. The GPU driver (IOGPUMemory.cpp) hit a "prepare count underflow" when it couldn't allocate memory, causing a kernel panic instead of a graceful process termination.

5. memoryPressure: false in the panic report shows that macOS didn't even detect memory pressure — the wired memory bypassed the normal memory pressure monitoring system entirely.

Suggested Fixes

1. Add --max-kv-size to mlx_lm.server (already requested in Feature Request: Add max-kv-size Support to MLX HTTP Server #615) — this is the most critical fix. Without a KV cache cap, memory grows unboundedly with context length.

2. Add --memory-limit flag to mlx_lm.server — allow users to cap total memory usage via mx.metal.set_memory_limit(). The current default (wiring 75% of RAM) is dangerously aggressive for systems running other processes.

3. Reduce the default wired limit — mx.set_wired_limit(max_recommended_working_set_size) should not wire 75% of RAM by default. Consider 50-60%, or at minimum, make it configurable.

4. Add memory monitoring with graceful shutdown — the server should monitor its own memory usage and return an error response (e.g., HTTP 503) when approaching the limit, rather than growing until the kernel panics.

5. Document the risk — users should be warned that running large models (30B+) on systems with ≤128GB can cause kernel panics if context grows too large.

Related Issues

• Feature Request: Add max-kv-size Support to MLX HTTP Server #615 — mlx_lm.server does not support --max-kv-size
• mlx-lm#427 — System Crash while Running Local AI Models

Panic Log

Key fields from panic-full-2026-02-12-011744.0002.panic

{
  "bug_type": "210",
  "timestamp": "2026-02-12 01:17:44.00 -0300",
  "os_version": "macOS 26.3 (25D125)",
  "incident_id": "0BA6183D-4448-439B-B57E-EE3B2CD397AB"
}

Panic string:

panic(cpu 18 caller 0xfffffe00428d679c): "completeMemory() prepare count underflow" @IOGPUMemory.cpp:550
Debugger message: panic
Memory ID: 0xff
OS release type: User
OS version: 25D125
Kernel version: Darwin Kernel Version 25.3.0: Wed Jan 28 20:47:03 PST 2026; root:xnu-12377.81.4~5/RELEASE_ARM64_T6031

Memory status:

{
  "compressorSize": 47640,
  "compressions": 8141671,
  "decompressions": 4388400,
  "busyBufferCount": 0,
  "memoryPressureDetails": {
    "pagesWanted": 3027,
    "pagesReclaimed": 5807
  },
  "pageSize": 16384,
  "memoryPressure": false,
  "memoryPages": {
    "active": 466937,
    "throttled": 0,
    "fileBacked": 342561,
    "wired": 5251984,
    "purgeable": 69720,
    "inactive": 465071,
    "free": 973,
    "speculative": 54
  }
}

Calculated:

• Wired: 5,251,984 pages × 16,384 bytes = 80.14 GB
• Free: 973 pages × 16,384 bytes = 0.01 GB

MLX Python process:

PID 30289 (Python): residentMemoryBytes = 83,235,119,080 (~83.23 GB)

[angeloskath]

Hm the panic is probably related to the wired limit being too high. Did you run sudo sysctl iogpu.wired_limt_mb=xxx before?

I will probably merge the max-kv-size PR but it isn't necessarily the way to go because the model output will be kind of wrong when it reaches the limit.

I think a better (orthogonal) choice is to expose the number of caches in the prompt cache and/or adapt it so that when the memory is close to the limit perhaps some could be offloaded to disk...

[DXXS]

I'm getting regular out of memory issues when running mlx_lm.server also... Even though the model should fit into memory fine (and 'generate' seems to have no problems), querying it via the server (in larger context situations such as code assist) can cause the GPU to apparently overflow, and the top ~1/5 of the screen flashes purple+ish & does a little sort of ~paging dance!?, often thrashing slower & slower until the machine reboots itself. Here's hoping y'all may add memory flags to fix!

% sysctl debug.iogpu.wired_limit
debug.iogpu.wired_limit: 0

[yoyaku155]

This issue persists on macOS 26.4 (25E246) with the same panic signature. We are seeing 9 kernel panics in 6 days on a Mac Studio M3 Ultra 96GB running MLX inference with a 122B MoE model.

The completeMemory() prepare count underflow panic at IOGPUMemory.cpp:550 continues to occur, and we've also observed a second related panic: "Memory object unexpectedly not found in fPendingMemorySet" @IOGPUGroupMemory.cpp:219.

We did not run sudo sysctl iogpu.wired_limit_mb — this is default MLX behavior triggering the kernel bug.

Full details with timestamps and reproduction steps filed at: ml-explore/mlx#3346

This is a critical kernel-level regression that has not been fixed in macOS 26.4. Requesting this issue be reopened or the MLX team escalate an internal radar to the GPU driver team.