MAPLE: Pruning fine-grained MoE GGUF containers for 100% In-VRAM edge inference

[AmSach]

For developers running Gemma 4 26B-A4B on consumer hardware with 6GB VRAM (such as RTX 4050/3060 laptops or 8GB Macs), the unpruned 9.92 GB GGUF causes heavy system-RAM paging and drops generation to ~6.9 Tokens/Sec.

We released a calibrated 48-expert GGUF container using MAPLE (Manifold-Activated Pruning of Latent Experts):

• Container Size: 4.75 GB (10.95B physical params, 1.48 effective base BPW)

• VRAM Footprint: 4,820 MiB runtime (100% In-VRAM on 6GB GPUs with 1.32 GB headroom)

• Generation Speed: 45.4 TPS (6.57x speedup over paged base)

• Task Fidelity: 93.8% semantic keyword retention across technical benchmarks

• Model Weights: https://huggingface.co/AmSach/Gemma-4-26B-A4B-MAPLE-Sub5GB-GGUF

• Replication Repo: https://github.com/AmSach/maple

• Academic Preprint: https://doi.org/10.5281/zenodo.24042480

Run directly with llama.cpp:

llama-cli -hf AmSach/Gemma-4-26B-A4B-MAPLE-Sub5GB-GGUF -ngl 99 -c 2048 -n 512

[AmSach]

Update: 100% Quality Retention with 500+ Token Generation Budget

When evaluating complex multi-class data structures and algorithms (such as Binary Search Tree implementations with unit test assertions), constrained token limits (e.g. -n 128 or -n 256) cause generation truncation before all method assertions print, which scored as 93.8% keyword recall in tightly bounded benchmark runs.

When output generation headroom is set to 500+ tokens (-n 512), MAPLE achieves 100.0% keyword fidelity and 100% functional test execution, matching the unpruned 25.6B base model with zero quality loss while running at 45.4 tokens/sec entirely in 6GB VRAM.

[dheiver2]

The 6.57x speedup to 45.4 TPS is a direct result of eliminating CPU-GPU bandwidth contention. When the unpruned model forces 9.92 GB onto a 6 GB card, the system spends cycles paging rather than computing. MAPLE's reduction to 4.82 GB runtime fits the VRAM ceiling, turning memory-bound latency into compute-bound throughput. Keeping the KV cache within the 1.32 GB headroom ensures context windows remain stable without eviction. This is how you actually deploy 26B-class reasoning on edge hardware.The 6.57x speedup to 45.4 TPS is a direct result of eliminating CPU-GPU bandwidth contention. When the unpruned model forces 9.92 GB onto a 6 GB card, the system spends cycles paging rather than computing. MAPLE's reduction to 4.82 GB runtime fits the VRAM ceiling, turning memory-bound latency into compute-bound throughput. Keeping the KV cache within the 1.32 GB headroom ensures context windows remain stable without eviction. This is how you actually deploy 26B-class reasoning on edge hardware.