[Showcase] MAPLE: Sub-2-BPW Pruning of Gemma 4 26B-A4B for 100% In-VRAM GPU Execution on 6GB VRAM (45.4 TPS)

[AmSach]

Hi llama.cpp community,

We wanted to share MAPLE (Manifold-Activated Pruning of Latent Experts), a post-training compression pipeline specifically created to solve the consumer VRAM bottleneck on fine-grained Mixture-of-Experts (MoE) architectures like Gemma 4 26B-A4B.

The Hardware Bottleneck on 6GB Laptop GPUs:

In its unpruned state (128 routed experts, 30 layers), even at extreme IQ2_XXS quantization, Gemma 4 26B requires 9.92 GB of memory. On consumer 6GB GPUs (such as an RTX 4050 Laptop GPU):

• Only 16 / 30 layers can fit into GPU VRAM (6,141 MiB allocated with system paging).
• PCIe system-RAM paging limits generation speed to 6.9 Tokens/Sec (144.9 ms / token).

The MAPLE 48-Expert Container:

Instead of uniform pruning, MAPLE profiles empirical forward-pass expert activations via llama-imatrix on a domain calibration corpus. Each expert is scored via a composite frequency-energy metric:
$$s_{e,l} = C_{e,l} \cdot \sqrt{\bar{\sigma}_{e,l} + \epsilon}$$

By slicing away 80 dormant experts per layer along expert axis 0 (ffn_down_exps, ffn_gate_up_exps) while locking layer scales at 1.0x, the resulting 48-expert container captures 79.88% of active forward-pass activation mass:

• Container Size: 4.75 GB (1.48 effective bits/param).
• GPU Residency: 100% In-VRAM (30 / 30 layers offloaded, 4,180 MiB VRAM used).
• Throughput: 45.4 Tokens/Sec (6.57x speedup, 22.0 ms / token).
• Task Fidelity: 93.8% keyword fidelity across Python code synthesis, BST data structures, physics reasoning, and formal syllogisms.

Run Directly via llama.cpp:

llama-cli \
  -hf AmSach/Gemma-4-26B-A4B-MAPLE-Sub5GB-GGUF \
  -ngl 99 \
  -c 2048 \
  -n 512 \
  --temp 0.4 \
  --top-p 0.9 \
  --repeat-penalty 1.1 \
  -r "<turn|>" \
  -p "<|turn>user\nExplain why the sky is blue using Rayleigh scattering.\n<turn|>\n<|turn>model\n"

• Preprint (Zenodo DOI): https://doi.org/10.5281/zenodo.24042480
• Hugging Face Model Container (4.75 GB GGUF): https://huggingface.co/AmSach/Gemma-4-26B-A4B-MAPLE-Sub5GB-GGUF
• GitHub Repository (Code & Pipeline): https://github.com/AmSach/maple

Feedback, benchmarks, and community tests are welcome!