server: force non-mmap load for MTP head to avoid Metal full-model duplicate

[frozename]

Targets the MTP path introduced in #22673 (currently a draft). Branched off the PR's head (5d5f1b46). Should be cherry-picked into #22673 / squashed into the same series before that PR lands. Filing as a separate PR for review visibility — happy to close once it's incorporated into #22673.

Bug

The MTP head is loaded by reopening the same GGUF with override_arch = qwen35_mtp[_moe], which registers tensors near both the start of the file (tok_embd) and the end (output, nextn.*, last transformer block). With the default mmap-backed buffer path at src/llama-model.cpp:1463-1483, the backend allocates a buffer covering the full [first_tensor_offset, last_tensor_offset) range — which for Qwen 3.6 27B spans nearly the entire file.

On Apple Silicon the result is a Metal buffer the same size as the main model, duplicated. Server log shows two MTL0_Mapped model buffer size entries with identical byte counts: one for the main model, one for the MTP head.

This is observable as unaccounted ≈ model_size in common_memory_breakdown_print (the MTP context's allocations live in a sibling llama_context and don't roll up into the target context's self).

Reproduction (M4 Pro 48 GB, Apple10 / pre-M5 Metal, PR head 5d5f1b46)

llama-server -m Qwen3.6-27B-Q5_K_M-mtp.gguf \
  --ctx-size 8192 --no-warmup -np 1 -ngl 99 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --spec-type mtp --spec-draft-n-max 3

Pre-fix server log:

load_tensors:  MTL0_Mapped model buffer size = 18760.13 MiB   <-- main model
load_tensors:  MTL0_Mapped model buffer size = 18760.13 MiB   <-- MTP head, full duplicate

OOMs mid-decode on the 38 GB Metal cap (M4 Pro). Q8_0 is even worse (~28 GB each).

Fix

Force use_mmap = false on the MTP load only. The non-mmap allocator path (llama-model.cpp:1492) sizes the backend buffer to the registered tensors instead of the mmap range — which for the MTP arch is ~1.4-1.7 GB instead of ~19-28 GB.

Before / after (M4 Pro 48 GB, Qwen 3.6 27B, froggeric's MTP GGUFs)

Quant	Metal MTP buffer pre-fix	Metal MTP buffer post-fix	Reduction
Q5_K_M-mtp (19 GB file)	18 760 MiB	1 425 MiB	13.2×
Q8_0-mtp (29 GB file)	28 213 MiB	1 719 MiB	16.4×

Both quants were OOMing mid-decode pre-fix; both now run cleanly.

Aggregate decode tok/s with the OP's recipe (-ctk q8_0 -ctv q8_0, no flash-attn, default ub), 9-prompt suite (am17an's gist, temperature=0 seed=42 n_predict=192):

Quant	Vanilla	MTP (post-fix)	Ratio	Accept
Q4_K_M	11.9	10.0	0.85×	0.701
Q5_K_M	9.5	8.6	0.91×	0.713
Q8_0	7.4	11.1	1.49×	0.725

Q4 is unchanged (it fit pre-fix; no regression). Q5 and Q8 are new datapoints because they OOM'd pre-fix. Q8 clears the speculative-decoding win threshold on this hardware — speculative savings scale with main-pass cost, so MTP shines at the largest quant the box can hold.

Testing scope

Built and tested only on M4 Pro 48 GB / Apple10 Metal (only platform I have access to). The non-mmap allocator path is well-exercised on all backends (it's the fallback when mmap fails or is disabled by --no-mmap), so functional correctness should be unchanged everywhere. Performance impact at MTP load time on other backends is a slower direct-read of ~1-2 GB of tensor data instead of mmap pointer aliasing — a one-time cost of a few seconds, no inference-time effect.

Would appreciate someone running this on CUDA / Vulkan to confirm no regression there.

Notes

• The fix only triggers when params_base.speculative.type == COMMON_SPECULATIVE_TYPE_MTP — vanilla / draft / mmproj load paths are untouched.
• Interactions with --no-mmap and --mlock were considered: the override is a no-op if the user already disabled mmap, and --mlock still works via the existing mlock_bufs path (llama-model.cpp:1497-1502).
• A more surgical alternative — conditionally skipping tok_embd / output registration in qwen35_mtp.cpp::load_arch_tensors when the nextn.* equivalents are present — was considered but rejected: it requires GGUF-metadata sniffing before arch instantiation, and breaks any future MTP GGUF that lacks the nextn variants. The non-mmap override keeps the arch logic simple and the failure mode contained.

[ggml-gh-bot]

Hi @frozename, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• Multiple backend changes in one PR: When adding support for a new model or feature, focus on CPU support only in the initial PR. Add support for other backends like CUDA in follow-up PRs. If you have a good reason to modify multiple backends in one PR, please explain it.

• AI-generated content: This project does not accept PRs, descriptions or commit messages that are fully or predominantly AI-generated. If you have used AI to assist you in writing code, please make sure to disclose that explicitly.

• Large PR: Large changes require prior discussion (e.g. an issue or RFC) and maintainers may not be able to review this PR as-is. Consider splitting it into smaller, focused PRs.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[0cc4m]

This should not be a PR, you made a 1 line change. Why not just discuss/propose it on the MTP PR?

[frozename]

This should not be a PR, you made a 1 line change. Why not just discuss/propose it on the MTP PR?

Ok I'll do that

[CISC]

I'd also like to stress that this kind of spam-by-ai will normally earn submitters a ban, the only reason you were not is because of the way you responded.

[frozename]

I'd also like to stress that this kind of spam-by-ai will normally earn submitters a ban, the only reason you were not is because of the way you responded.

Thanks for the heads up. I'll keep that in mind for future contributions