Qwen3.6-27B on a single RTX 5090, 256K context, 160+ TPS

A validated recipe for serving Qwen3.6-27B on a single 32 GB RTX 5090 — full OpenAI API, vision, tool calling, streaming, speculative decoding, all verified end-to-end via scripts/verify-full.sh.

Based on Lorbus/Qwen3.6-27B-int4-AutoRound via vLLM with MTP speculative decoding + fp8_e4m3 KV cache. Built on Sandermage/genesis-vllm-patches + a CUDA graph capture fix that ships in this repo.

📖 Write-up for 3090: https://medium.com/@fzbcwvv/an-overnight-stack-for-qwen3-6-27b-85-tps-125k-context-vision-on-one-rtx-3090-0d95c6291914 🐛 Upstream bug reports: vllm-project/vllm#40807 (CUDA graph crash — worked around locally) · vllm-project/vllm#40831 (TurboQuant × spec-decode output-quality, isolated to cudagraph capture; root cause TBD)

Requirements

• GPU: 1× NVIDIA RTX 5090 (32 GB, Blackwell GB202).
• Driver: 580.x or newer (for CUDA 13 runtime in the pinned vLLM image).
• Chat template: included at compose/qwen3.5-enhanced.jinja — mounted automatically by the compose file.
• Disk: ~20 GB free for model weights.
• Software:
  • Docker with NVIDIA Container Toolkit
  • git, curl, sha256sum (setup script uses them)
  • hf CLI or huggingface-cli (install: pip install 'huggingface-hub[hf_transfer]')

No system Python required.

---

Quick start

# 1. Clone this repo
git clone https://github.com/CobraPhil/qwen36-27b-single-5090.git
cd qwen36-27b-single-5090

# 2. Fetch Genesis patches + download + SHA-verify the model (~20 GB, 10-30 min)
bash scripts/setup.sh

# 3. Start the server
cd compose && docker compose up -d

# 4. Watch it come up (~2 min for cold compile)
docker logs -f vllm-qwen36-27b
# Wait for "Application startup complete"

# 5. Sanity test
curl -sf http://localhost:8020/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3.6-27b-autoround",
       "messages":[{"role":"user","content":"Capital of France?"}],
       "max_tokens":30}'

# 6. Run the canonical benchmark
cd .. && bash scripts/bench.sh

That's it. The stack serves on http://localhost:8020/v1/* as a drop-in OpenAI-compatible endpoint — point any OpenAI SDK, Open WebUI, LM Studio, or Cline at it.

---