"I didn't know it was IMPOSSIBLE!... I just needed it." - 400k+ Context on Qwen 3.6 MoE 35B on a Single 16GB GPU (RTX 5060 Ti) at 25 t/s

[0-KaM-0]

Hi everyone!

People often tell me that something is "impossible"... but when it actually works, they suddenly have nothing to say.
They claim that a 10M context with 90%+ coherence is undoable? Well, Grok publicly rated my solution on X (Twitter) as a 9.8 out of 10... for achieving a 60M context with 90%+ coherence.

Today, I want to share a successful stress-test optimization that allowed me to push the Qwen 3.6 MoE (35B-A3B) model far beyond its native limits on a single consumer-grade RTX 5060 Ti 16GB GPU, while keeping all 99 layers completely in VRAM.

My Setup & Metrics

• GPU: RTX 5060 Ti 16GB (GDDR7)
• Model: Qwen3.6-35B-A3B-12Gb-2.6763bpw.gguf
• Context processed: 353,993 tokens (real usage inside a live coding session)
• VRAM allocation: 15166 MiB / 16311 MiB (with around 1.1GB still free — and this includes the OS + 2 open browsers!)
• Prefill time: 8m 14s (~716 tok/s at the very end of the 353k chunk)
• Decode speed: 24.9 - 25.9 tok/s (Prompt Caching works perfectly on follow-up turns!)

Key Takeaways

1. You can easily push the limit up to 450k tokens without crashing the system. Pushing to 500k is possible, but it's right on the edge.
2. The model remains completely coherent (I haven't strictly verified the code execution accuracy at this scale yet, but the logic holds up).
3. Thanks to Prompt Caching, it works incredibly well. Getting 25 tok/s at this context scale speaks for itself. Massive kudos to the llama.cpp team!
4. The smart quant model squeezed down to exactly 11.7GB is beyond praise. Low bow to the author of this quant!

Deep Dive & The Solution (How to bypass HTTP 400 & Capping)

Standard parameters like qwen2.context_length or bare context_length overrides were ignored by the recent llama-server builds with CUDA 13.3, forcing a 256k slot cap. The hard limit of the model was successfully forced-overridden via the correct metadata key: --override-kv qwen35moe.context_length=int:400000.

Here is the exact launch command that made this rock-solid stable:

llama-server.exe --model Qwen3.6-35B-A3B-12Gb-2.6763bpw.gguf --ctx-size 400000 --temp 0.55 --cache-type-k q4_0 --cache-type-v q4_0 --gpu-layers 99 --batch-size 256 --ubatch-size 256 --parallel 1 --min-p 0.05 --repeat-penalty 1.1 --repeat-last-n 64 --n-predict -1 --flash-attn on --override-kv qwen35moe.context_length=int:400000 --rope-scaling yarn --rope-scale 1.526 --yarn-orig-ctx 262144

What do you think? Who can replicate or beat this result?

[0-KaM-0]

Final Sanity Check: Isolated Code Generation at 355k depth
To test the logical coherence and see if the context leaks into new tasks, I asked the model to write a simple, isolated "Back to Top" JavaScript button.

As you can see, even at a massive 355k tokens depth under a 1.526 YaRN scale, the syntax remains pristine. No hallucinations, perfect brackets closure, zero token repetition stuttering, and it came out instantly at 24.9 tok/s thanks to Prompt Caching!

[0-KaM-0]

🚀 UPDATE: Breaking the limits on an RTX 5090 32GB!

Just wanted to update this thread! Yesterday, we conducted the exact same stress-test architecture, but this time on a flagship RTX 5090 32GB GPU.

And the results are absolutely mind-blowing:

• Context window forced to: 1,500,000 tokens!!!!
• Total VRAM allocation: Only 23.8 GB out of 32 GB total (including Windows OS + 2 active browsers)!
• Live context filled to: 1.18 Million tokens (ingested around 2,000 pages of real heavy PDF documentation).

After the massive ~10-minute prefill phase, we ran a thorough sanity check testing the model on complex isolated code generation and abstract logic. The Qwen 3.6 MoE model remained 100% coherent, fully responsive, and perfectly functional, generating pristine code structures at a blistering 35 tok/s!

And yes — the entire 1.5M context pipeline runs COMPLETELY inside a single GPU! No system RAM offloading, no dual-card setups. Just raw, optimized single-card dominance. 🦖🔥🏆

[159753a52]

For anyone wondering why this fits at all, it's mostly the architecture: Qwen3.6-35B-A3B keeps a KV cache in only 10 of its 40 layers (the rest are linear attention with a fixed-size state), with 2 KV heads × 256. That's 20 KiB per token at f16 and ~10.6 KiB at q8_0. At 354K tokens it's ~6.75 GiB at f16 and ~3.6 GiB at q8_0, so next to an ~11 GiB file it only fits on 16 GB with a quantized cache.

If you'd rather keep a 4-bit quant than a 2.68 bpw one: UD-Q4_K_M (22.1 GB file) at 128K with f16 KV needs -ngl 99 --n-cpu-moe 17 on a 16 GB card, which is ~15.9 GiB on the card and ~8.2 GiB of experts in RAM. The full 256K needs --n-cpu-moe 23 (15.7 GiB on the card, 10.9 GiB in RAM). Decode drops because those experts are read from system RAM every token, so the small quant still wins on speed if its quality holds up for you.

I built a free planner that picks the --n-cpu-moe number from the GGUF's tensor sizes: [Link removed by moderator]/moe-offload-calculator/