2080Ti LLM Toolbox

Recipes, patches, manifests, and benchmark summaries for running modern LLM serving stacks on RTX 2080 Ti / Turing SM75.

This is a working toolbox, not an official support matrix. The main target is a dual modified RTX 2080 Ti 22GB NVLink rig. The notes should still help other SM75 users understand which paths are realistic and which ones are dead ends.

Peak Result

Peak single-request speed on dual modified RTX 2080 Ti 22GB cards with NVLink, plus the largest measured KV cache from the 262K startup probe:

Model	Median Prefill	Median Decode	Max KV Cache
Qwen3.6-27B-AWQ	1841.7 tok/s	101.3 tok/s	735,084 tok

That is the result to compare against when judging whether an SM75 setup is actually in the right performance class. The detailed table and provenance are in reports/summaries/qwen36-27b-awq-vllm-peak-single-request.md. The speed columns use the PP4096/TG128 repeat. The max-cache column is the turboquant_4bit_nc max_model_len=262144 startup/cache probe.

Long-Context Capacity

The same dual 22GB 2080 Ti rig can create a real max_model_len=262144 vLLM cache and can complete real near-full-context requests. The most aggressive measured cache row is TurboQuant 4bit_nc:

KV dtype	vLLM reported KV cache	Total used VRAM / 2080 Ti
turboquant_4bit_nc	735,084 tok	20,595 MiB
turboquant_k8v4	520,461 tok	20,615 MiB
int8_per_token_head	518,397 tok	20,633 MiB
auto / FP16	272,938 tok	20,633 MiB

The VRAM column is total nvidia-smi device memory after startup, including weights, runtime/workspace, and KV cache. It is not KV-only footprint.

Real prompt gates also passed:

Route	Request	Prefill	Result
FP16/default KV, noMTP, original AWQ baseline	PP262000/TG1	785.26 tok/s	HTTP 200
TQ4NC KV, noMTP, current Qwopus quality lane	PP262000/TG1	783.09 tok/s	HTTP 200

Details: reports/summaries/qwen36-27b-awq-vllm-ctx262k-kv-vram.md. TurboQuant quality/stability details: reports/summaries/qwen36-27b-awq-vllm-turboquant-kv.md.

Recommended Route

This is the stack behind the peak result:

vLLM 0.21.0
torch 2.11 cu130
TP=2
AWQ Marlin
FlashInfer/FA2 attention
FlashQLA SM70/SM75 legacy GDN prefill
MTP K=3

Packaged runtime: weicj/vLLM-2080Ti-Definitive is the maintained 2080 Ti Definitive vLLM runtime that packages the current SM75 patches, launcher, profiles, and documentation for this route.

Use these first when reproducing the peak result or comparing other engines / KV experiments:

• Recipe: engines/vllm/recipes/qwen36-27b-awq-best-sm75.md
• Patch queue: engines/vllm/patches
• Lock file: manifests/dual-2080ti-vllm-qwen27-awq-best-sm75.lock
• Status page: STATUS.md

Reproduction order:

1. Start from the best SM75 recipe.
2. Apply the vLLM patch queue.
3. Validate the MTP K=3 route before enabling TurboQuant KV.
4. Compare results against BENCHMARKS.md.

Tested Hardware

The headline results are not from a stock 11GB card.

GPU: 2x modified RTX 2080 Ti 22GB
Interconnect: NVLink
Architecture: Turing SM75
Stepping: TU102-300A + TU102-300

Single-request results mean one active request at a time. Concurrent serving numbers are kept separate because they are not comparable with single-slot llama.cpp rows.

What Works

Route	Status	Use It For
vLLM + Qwen3.6-27B-AWQ + MTP K=3	Recommended	Fast dual-card serving, 4K and 64K workloads, repeated agent-style requests
vLLM + TurboQuant KV	Validated experimental	Testing compressed KV behavior; tq4nc is the best practical TQ row so far
llama.cpp + Qwen3.6 27B GGUF	Reliable baseline	Simpler fallback and sanity checks
SGLang + Qwen3.6-27B-AWQ	Smoke-only	SM75 compatibility research, not production comparison

The core SM75 unlock is FlashQLA for the GDN path. The separate FlashQLA backend is maintained at weicj/FlashQLA-SM70-SM75.

Why This Is Recommended

The vLLM path has real performance, quality, and stability evidence:

• PP4096/TG128 peak repeat: see the peak table at the top of this README.
• PP64K/TG512: 1294.3 tok/s prefill, 55.3 tok/s decode.
• Sequential 60-request Ragent6 run: 167.4s wall, average 700.9 tok/s prefill and 35.2 tok/s generation.
• LongGen3 4096/1024 warm sweep: Qwopus MTP improves decode from 32.26 tok/s noMTP to 54.14 tok/s at MTP K=3 and 55.20 tok/s at K=5.
• Concurrent serving validated up to max_num_seqs=4 after the FlashQLA legacy GDN multi-prefill patch.
• Ragent6 1/2/4-way shard checks completed without GDN errors, HTTP 500s, or quality regression: strict 43/60, weighted 82.5/100, invalid 0.

The highest-quality detailed tables live in BENCHMARKS.md. Curated run summaries live under reports/summaries.

Gemma4 31B GPTQ is also viable as a secondary route: FP16/default KV with assistant MTP5 reached a warm PP4096/TG128 peak of 1655.65 tok/s prefill and 99.64 tok/s decode on the same dual-card vLLM runtime. Its compressed-KV routes are more limited than Qwen's, so Gemma remains a secondary/experimental profile rather than the main toolbox route.

Required vLLM Patch

The current vLLM build needs a FlashQLA legacy GDN compatibility fix for multi-prefill batching:

• vLLM may pass multiple prefill sequences as packed cu_seqlens.
• The old FlashQLA legacy path only accepted one contiguous sequence.
• The patch splits the packed batch per sequence, calls the legacy GDN kernel, then reassembles output and final state.

Patch: engines/vllm/patches/0001-sm75-flashqla-gdn-ragged-prefill.patch

This is a compatibility loop, not a fused ragged GDN kernel. It makes real serving usable; it is not the final multi-prefill performance design.

Baselines And Rejected Paths

llama.cpp remains the sanity baseline. It is slower than the validated vLLM route on this hardware, but simpler and robust.

SGLang has reached a short HTTP 200 smoke with the Qwen3.6-27B-AWQ path, but it is not production-ready: the smoke output was bad, and there is no valid prefill/decode benchmark or repeated-request run yet.

The derivative Qwen3.6-35B-A3B-AWQ route reached near-100 tok/s decode, but is rejected because quality collapsed in Ragent6: strict 10/60, weighted 32.9/100.

Rejected and experimental routes are tracked in STATUS.md, model notes under models, and the relevant report summaries.

Repository Map

• engines/vllm: vLLM SM75 recipes, patch queue, and launch notes.
• engines/flashqla: FlashQLA SM70/SM75 kernel notes.
• engines/llamacpp: llama.cpp baselines.
• engines/sglang: SGLang experimental SM75 bring-up.
• models: model-specific compatibility notes.
• manifests: environment and workload locks.
• reports: curated summaries and raw log pointers.
• BENCHMARKS.md: full benchmark scorecard.
• STATUS.md: recommended, experimental, and rejected paths.
• scripts: small utilities for snapshots and repeatable calls.