Llama Main

llama/main

llama/main tracks upstream llama.cpp and carries the fork's conservative memory and MTP controls. Documented checkpoint: 5903044bc.

Note

BACKEND SUPPORT VARIES: These flags are not CUDA-only. The selected backend must support the requested storage and execution route.

Added features

• Pinned host KV
• Independent recurrent-state placement
• Partial target and draft KV residency
• Canonical host-KV quantization
• Independent draft ubatch
• Phase-aware workspace
• Capped MTP recurrent planes
• Sparse recurrent snapshots

All features remain opt-in. Without these flags, the branch keeps its normal llama.cpp behavior.

Key tradeoffs

• Host KV saves VRAM but adds host-device traffic.
• Partial GPU KV spends VRAM to reduce that traffic.
• Phase-aware workspace reduces eligible transient residency, not model weights or KV.
• Capped MTP state saves persistent rollback memory but can replay after a deep rejection.
• Sparse MTP replay currently supports declared Qwen 3.5 and Qwen 3.5 MoE GDN graphs on qualified backends.

Measure the intended context, parallelism, and draft depth. Do not treat memory savings as a guaranteed speedup.

Build

git clone --branch llama/main https://github.com/GenerelSchwerz/llama.cpp.git
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j 20
./build/bin/llama-server --help

Flags and setup - Hardware guides - Feature index - Home