When does mmap actually help performance?

[jintakhan]

In almost every instance I've come across, mmap kneecaps prefill performance by at least 30% on both unified memory and non-UMA systems due to the added paging overhead. And from my understanding, while lazy loading helps with model loading times, it can also cause unexpected OOMs. Yet it remains the default load mode for llama.cpp. Is there a situation where mmap actually meaningfully helps with inference performance? Genuinely curious. Thanks.

[ekusiadadus]

I would not generally expect mmap itself to make steady-state inference compute faster. Its main advantages are around model loading and memory management rather than making the matmuls or attention kernels faster.

With mmap, llama.cpp can reference the model weights directly from file-backed mappings instead of first copying the entire model into separately allocated host buffers. That can reduce startup time and peak memory usage, and file-backed pages can also be shared through the OS page cache between processes.

There is an important implementation detail here, though: mmap and --no-mmap do not necessarily exercise identical loading paths in current llama.cpp.

For example, the mmap path directly returns pointers into the mapped file:

llama.cpp/src/llama-model-loader.cpp

Lines 1471 to 1490 in bd4f514

	const void * llama_model_loader::load_data_range(const llama_tensor_weight & w, size_t offs, size_t size, void * buf) const {
	GGML_ASSERT(offs + size <= ggml_nbytes(w.tensor));
	const void * data = buf;
	if (use_mmap) {
	data = (const uint8_t *) mappings.at(w.idx)->addr() + w.offs + offs;
	} else {
	GGML_ASSERT(buf != nullptr);
	GGML_ASSERT(w.idx < files.size());
	const auto & file = files.at(w.idx);
	file->seek(w.offs + offs, SEEK_SET);
	file->read_raw(buf, size);
	}
	if (check_tensors && !ggml_validate_row_data(w.tensor->type, data, size)) {
	throw std::runtime_error(format("tensor '%s' has invalid data", ggml_get_name(w.tensor)));
	}
	return data;

On Linux, llama.cpp also uses POSIX_FADV_SEQUENTIAL, MAP_POPULATE when applicable, and POSIX_MADV_WILLNEED for prefetched regions:

llama.cpp/src/llama-mmap.cpp

Lines 470 to 515 in bd4f514

	size = file->size();
	int fd = file->file_id();
	int flags = MAP_SHARED;
	if (numa) { prefetch = 0; }
	#ifdef __linux__
	if (posix_fadvise(fd, 0, 0, POSIX_FADV_SEQUENTIAL)) {
	LLAMA_LOG_WARN("warning: posix_fadvise(.., POSIX_FADV_SEQUENTIAL) failed: %s\n",
	strerror(errno));
	}
	// MAP_POPULATE would fault in the lazy ranges too
	if (prefetch && lazy_ranges.empty()) { flags |= MAP_POPULATE; }
	#endif
	addr = mmap(NULL, file->size(), PROT_READ, flags, fd, 0);
	if (addr == MAP_FAILED) {
	throw std::runtime_error(format("mmap failed: %s", strerror(errno)));
	}
	// page-aligned madvise over [beg, end), clamped to the file
	auto advise = [&](size_t beg, size_t end, int advice, const char * name) {
	const size_t page_size = sysconf(_SC_PAGESIZE);
	beg = beg & ~(page_size - 1);
	end = std::min((end + page_size - 1) & ~(page_size - 1), file->size());
	if (beg >= end) {
	return;
	}
	if (posix_madvise((char *) addr + beg, end - beg, advice)) {
	LLAMA_LOG_WARN("warning: posix_madvise(.., %s) failed: %s\n", name, strerror(errno));
	}
	};
	if (prefetch > 0) {
	for (const auto & range : ranges_complement(lazy_ranges, std::min(file->size(), prefetch))) {
	advise(range.first, range.second, POSIX_MADV_WILLNEED, "POSIX_MADV_WILLNEED");
	}
	}
	for (const auto & range : lazy_ranges) {
	advise(range.first, range.second, POSIX_MADV_RANDOM, "POSIX_MADV_RANDOM");
	}
	if (numa) {
	if (posix_madvise(addr, file->size(), POSIX_MADV_RANDOM)) {
	LLAMA_LOG_WARN("warning: posix_madvise(.., POSIX_MADV_RANDOM) failed: %s\n",
	strerror(errno));
	}
	}
	mapped_fragments.emplace_back(0, file->size());

Conversely, the non-mmap path can stage data through pinned host buffers and perform asynchronous uploads to the GPU. So especially with GPU offloading, a performance difference during loading or the first inference may come from the different I/O/upload paths rather than from page faults alone.

For steady-state prefill, though, I would expect the difference to mostly disappear once the relevant working set is resident. If mmap is still ~30% slower after several warmup runs, with no meaningful major-page-fault activity, that would be interesting and would suggest that paging alone is probably not the explanation.

I would benchmark these separately:

• model load time
• first prefill / TTFT
• subsequent prefills after warmup
• major/minor page faults
• RSS / resident model memory

and test both cold and warm page caches.

It would also be useful to know the exact model, llama.cpp commit, backend, -ngl/GPU offload settings, OS, and benchmark commands. A persistent ~30% steady-state difference under otherwise identical conditions would definitely be worth investigating.

[jintakhan]

Thank you. That’s what I’m confused about. I also expected mmap to perform the same as no mmap once pages are hot in VRAM. But I consistently get significantly worse prefill with mmap on, no matter how many warmup runs I give it. I always thought it was overhead inherent to mmap. I’ll take a look at page fault stats and see if there’s anything weird going on.

[dev-jinwoohong]

Hi @jintakhan,

Two things can keep mmap slower after warmup, and both are about whatever part of the model still lives in host memory, not the part in VRAM.

1. Huge pages. With --no-mmap the weights sit in anonymous memory, which THP can back with 2 MB pages. A file-backed mmap doesn't get that (kernel docs: "THP only works for anonymous memory mappings and tmpfs/shmem"), so CPU-side matmuls over those weights take noticeably more TLB misses. Warmup doesn't help because the pages are already resident, they're just 4 KB ones. Compare AnonHugePages in /proc/<pid>/smaps_rollup between the two runs, that's the tell. Same idea on Windows, mapped file views never get large pages.

2. macOS / Metal. ggml-metal wraps the mapped file directly with newBufferWithBytesNoCopy, so the "GPU" weights are literally the file cache pages. Under memory pressure the OS drops them and they get refaulted from disk on the next prefill, every time. --mlock (or --no-mmap) stops that.

If you're fully offloaded to a discrete GPU, the only thing left on the CPU is the token embedding table (llama-model.cpp: "there is very little benefit to offloading the input layer, so always keep it on the CPU"), and the lookup for a prompt is tiny, so mmap shouldn't be able to cost 30% there. In that case I'd double check that -ngl really covers every layer and post the CPU_Mapped model buffer size / CUDA0 model buffer size lines from the load log.