Decode speed ~30x below documented benchmarks on Apple M1 (8GB)

[Penscribe]

Decode speed ~30x below documented benchmarks on Apple M1 (8GB)

Summary

Running edge0-8b (Edge0-8B-A1B-preview) on a MacBook Pro with an M1 chip and 8GB unified memory produces decode speeds of ~0.7-0.8 tok/s, versus the ~23-25 tok/s reported in the model card for Apple Silicon. Peak memory usage matches the documented ~1GB, so the SSD expert offload mechanism itself appears to work correctly — only throughput is affected.

Environment

• Hardware: MacBook Pro, Apple M1, 8GB unified memory
• OS: macOS (Tahoe, per wheel tags macosx_26_0)
• edge0: 0.1.0 (fresh clone, pip install -e '.[dev,fetch]')
• mlx: 0.30.6 / mlx-metal: 0.30.6 / mlx-lm: 0.31.0
• Model: Edge0/Edge0-8B-A1B-preview, downloaded via scripts/fetch_models.py --tier edge0-8b
• Python: tested on both 3.14.3 and 3.12.13 — identical results (see below)

Steps to reproduce

python3.12 -m venv .venv && source .venv/bin/activate
pip install -e '.[dev,fetch]'
python scripts/fetch_models.py --tier edge0-8b --target-dir models
export EDGE0_8B_MODEL=$PWD/models/edge0-8b
edge0 chat --model-dir models/edge0-8b --prompt "Bonjour, présente-toi en une phrase."

Results across isolation tests

Condition	Result
Baseline (Python 3.14, ~7GB swap in use)	31 tokens / 46.6s → 0.66 tok/s
After closing background apps (swap ~5GB)	28 tokens / 38.4s → 0.73 tok/s
Repeat, same conditions	32 tokens / 40.2s → 0.80 tok/s
Same model_dir, fresh Python 3.12.13 venv	33 tokens / 40.8s → 0.81 tok/s
Control: mlx-community/Qwen3-4B-Instruct-2507-4bit via plain mlx_lm.generate, same machine, same session	11.38 tok/s (normal for a 4B dense model on M1)

Memory pressure/swap reduction (~7GB → ~5GB swap used) produced no meaningful change. Python version (3.14 vs 3.12) produced no meaningful change. The control test with a dense mlx-lm model on the same machine, run immediately after an edge0 test, ran at a normal speed — ruling out persistent thermal throttling as the cause (a throttled CPU/GPU would not recover instantaneously for the next process).

By elimination, the ~30x gap seems tied to edge0's own I/O pattern (frequent small mmap reads per expert per token) interacting poorly with the M1's SSD controller / memory bandwidth, which differs substantially from the M4 Pro used for the published benchmarks. kernel_task CPU usage spiked heavily during edge0 runs (20-49%) and dropped back to baseline immediately once generation ended, consistent with page-fault servicing rather than thermal state.

Expected behavior

Decode speed closer to the ~23-25 tok/s documented for Apple Silicon in the model card (understanding some slowdown vs the M4 Pro benchmark machine is expected — but not a ~30x factor).

Additional (unrelated) observation

pip install -e '.[dev,fetch]' pulls mlx-lm==0.31.0, which PyPI flags as yanked ("Batched KV cache cross contamination", see ml-explore/mlx-lm#965 and #975). Not believed to be the cause of the slowdown above (the bug is specific to batched/concurrent server requests, and this testing was single-request CLI), but flagging in case it's relevant to edge0's pinned dependency choice.

Happy to help debug

I can run additional diagnostics (Instruments trace, fs_usage during a chat run, testing with EDGE0_8B_MODEL on a different disk, etc.) if useful — let me know what would help narrow this down.

[linyubupa]

Thanks for the detailed report. Your I/O hypothesis matches the design: experts are mmap-backed and fetched per token through an LRU capped at 256MB by default, so decode on a memory-constrained machine is page-fault bound, not compute bound (consistent with the kernel_task spikes and the dense-4B control running at normal speed).

Two knobs to try before deeper debugging:

EDGE0_PREWARM=1 — warms the page cache over the checkpoint (madvise + sequential read) plus a dummy prefill/step
MLX_CACHE_LIMIT_MB=1024 — raises the expert LRU from 256MB so fewer evictions/refetches per token
Also run the same prompt twice: if the 2nd run is faster, it's cold-fetch cost. If it stays ~0.8 tok/s even warm on the internal disk, the next step is readahead/buffering in the loader — your Instruments/fs_usage traces would help.