PLE Prefetch and Decode Overlap

PLE prefetch and decode overlap

moe-cache provides three separate experimental server flags. All default to off.

Flag	Environment variable	Action
--ple-prefetch	LLAMA_ARG_PLE_PREFETCH=1	Advises lazy-backed CPU row pages before reading them.
--decode-overlap	LLAMA_ARG_DECODE_OVERLAP=1	Queues eligible CUDA decode work while the CPU handles the previous result.
--decode-boundary-overlap	LLAMA_ARG_DECODE_BOUNDARY_OVERLAP=1	Overlaps eligible boundary preparation and CUDA graph updates. Use with --decode-overlap.

Omit a flag and unset its variable, or set the variable to 0, to disable it. Restart the server after changing these controls.

PLE prefetch

The row helper receives the real table and index tensor when GET_ROWS runs. It advises only rows inside a live lazy mapping. It skips unsupported layouts, non-lazy tables, and GPU gathers.

PLE prefetch:

• works in prefill and decode;
• does not require CUDA, decode overlap, or an expert cache;
• does not make arbitrary tensors lazy;
• does not preload or pin the complete table;
• keeps the original row order, dequantization, and thread count.

For Flash Next, use --load-mode none --lazy-mode on. This keeps ordinary expert sources allocated while PLE remains file-backed.

On the staged Flash Next path, the worker advises known PLE rows before filling its existing bounded buffer. Enabling PLE prefetch alone does not enable staged decode.

Platform behavior

• Linux uses page advice and has measured runs.
• Windows dynamically calls PrefetchVirtualMemory on supported targets. Native performance remains unverified.
• Other POSIX platforms remain unverified.
• Advice failure falls back to ordinary demand reads.

The helper can submit up to 256 coalesced page ranges per call. It adds bounded scratch, not a new worker or a full-table allocation.

Decode overlap

For target-only generation, --decode-overlap queues at most one eligible decode batch. Each sequence contributes one token, and the batch must fit one microbatch. The path requires:

• one CUDA GPU;
• GPU token embeddings or the supported Flash Next staged-input path;
• compatible backend sampling;
• compatible graph inputs and cache ownership;
• no active feature that requires incompatible output handling.

Unsupported sampling, probability output, LoRA, multimodal input, finite reasoning budgets, or incompatible graphs keep the normal path. Grammar and reasoning transitions discard queued work, restore sampler/model state, then continue normally.

For integrated MTP, the same flag can overlap the first next-draft pass after target sampling, acceptance, and rollback. That path requires one MTP head, one CUDA GPU, and an independently owned draft cache with one-token rollback.

For target-only Flash Next, staged host embeddings require -np 1 and speculation disabled. Look for:

Flash Next staged inputs enabled
staged PLE prefetch enabled

Decode-boundary overlap

Enable both flags:

--decode-overlap
--decode-boundary-overlap

The boundary path rebuilds eligible next-step graphs while prior GPU work runs. It proceeds without a wait only when existing buffers fit and ownership remains valid. Allocation growth, shared storage, unsupported placement, or failed graph updates keep the synchronized or re-instantiated path.

This flag targets boundary stalls, not average throughput. Measure coherent output, token gaps, graph reuse, and peak memory before keeping it.

Set GGML_CUDA_GRAPH_PROFILE=1 to log capture_us, update_us, launch_us, and cleanup_us. These values measure CPU time, not GPU idle time.

Test order

Start from a working command, then test:

1. --ple-prefetch alone;
2. --backend-sampling --decode-overlap;
3. --decode-boundary-overlap in addition to decode overlap;
4. MTP only after target-only overlap passes.

Keep the model, prompt, seed, context, cache, batch, ubatch, and output length fixed. Record output identity or divergence, prefill, decode, TTFT, token-gap percentiles, and memory.

On Windows, follow Windows WDDM setup and report the exact build, driver, storage, and path-enablement logs.

Source: current server documentation, llama-mmap.cpp, and server-context.cpp.

MoE flags - Grouped drafting - Feature index