Getting a NVIDIA Tesla P40 (Pascal, sm_61, 24 GB) working for local LLM
serving under WSL2 on Windows 11, with Docker GPU containers — and
benchmarking it on a modern 35B MoE model.
This is a working, verified setup, not a sketch. Every number below was measured on the machine described in Tested configuration.
Result: a 2016 datacenter card serving Qwen3.6-35B-A3B at ~51 tok/s with 128K context.
Three separate things block a P40 under WSL2, and none of them produce an error message that points at the actual cause:
- The GPU is invisible to WSL2 in TCC driver model — you get
Failed to initialize NVML: N/Ainside Linux whilenvidia-smiworks fine on Windows. No amount of reinstalling CUDA fixes this. WDDM is license-locked on Tesla cards; MCDM is the answer, and that is not documented for Pascal. - CUDA 13.x cannot target Pascal at all — offline compilation for
sm_61was removed in CUDA 13.0. You must pin to CUDA 12.x. - CUDA 12.9 does not compile on modern glibc — glibc 2.41/2.42 added C23 math functions that collide with CUDA's headers. NVIDIA fixed it in 13.2.1, which cannot target Pascal. You must patch.
Plus a fourth, if you build llama.cpp yourself: libcuda.so.1 does not exist at
Docker build time, only at run time.
See FINDINGS.md for the discoveries that are not documented elsewhere.
| Component | Version |
|---|---|
| GPU | NVIDIA Tesla P40, 24 GB, compute capability 6.1 |
| Secondary GPU | AMD Radeon iGPU (drives display) |
| Windows | 11 Pro, build 26200 |
| NVIDIA driver | 582.70 |
| WSL | 2.7.10, kernel 6.18.33 |
| Distro | Ubuntu 26.04 LTS (glibc 2.43, gcc 15) |
| CUDA toolkit | 12.9.2 |
| Docker Engine | 29.6.2 (native in WSL, not Docker Desktop) |
| NVIDIA Container Toolkit | 1.19.0 |
Other Pascal cards (P100, P4, GTX 10-series) should follow the same path, but only the P40 was tested. Newer GPUs do not need any of this.
Run these in order. Steps 1 is Windows-side; the rest run inside WSL.
# 1. Windows, ELEVATED PowerShell. Switches the P40 to MCDM. REQUIRES A REBOOT.
.\scripts\01-windows-enable-mcdm.ps1Reboot, then inside your WSL distro:
sudo ./scripts/02-wsl-cuda-toolkit.sh # CUDA 12.9 + gcc-14 + glibc patch
sudo ./scripts/03-wsl-docker-nvidia.sh # Docker Engine + container toolkit + CDI
./scripts/04-verify-stack.sh # 12-point verification, run as your user
./scripts/05-build-llamacpp.sh # builds llamacpp-p40:latest for sm_61Then fetch a model and serve it:
mkdir -p /opt/models/mtp && cd /opt/models/mtp
curl -L -C - -O "https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF/resolve/main/Qwen3.6-35B-A3B-UD-Q3_K_XL.gguf?download=true"
./scripts/06-run-server.sh04-verify-stack.sh is the one to run if anything misbehaves later — it isolates
which layer broke.
Qwen3.6-35B-A3B, UD-Q3_K_XL (16.85 GB / 17.23 GB MTP), full GPU offload.
| Configuration | Generation |
|---|---|
| Baseline, no speculative decoding | 37.9 tok/s |
| MTP weights, spec decoding off (control) | 38.3 tok/s |
MTP, --spec-draft-n-max 6 |
34.5 tok/s |
MTP, --spec-draft-n-max 4 |
41.9 tok/s |
MTP, --spec-draft-n-max 3 (default) |
48.3 tok/s |
MTP, --spec-draft-n-max 2 |
51.6 tok/s |
Prefill (pp512, llama-bench): 982 tok/s.
The control row matters: MTP weights with speculation disabled land on baseline, which is what proves the speedup is real speculative decoding rather than file-to-file variance.
| Context | VRAM | Generation |
|---|---|---|
| 4096 (KV f16) | 21710 MiB | 51.45 tok/s |
| 8192 | 21732 MiB | 50.34 tok/s |
| 16384 | 21836 MiB | 50.95 tok/s |
| 32768 | 22086 MiB | 50.79 tok/s |
| 65536 | 22614 MiB | 50.85 tok/s |
| 131072 | 22782 MiB | 50.54 tok/s |
32× the context for ~1 GB and ~2% throughput — see FINDINGS.md.
Needle-in-a-haystack, needle at 10% / 50% / 90% depth — 9/9 exact retrievals:
| Prompt size | Result | Prefill |
|---|---|---|
| 6315 tokens | 3/3 PASS | ~790 tok/s |
| 25111 tokens | 3/3 PASS | 422–609 tok/s |
| 78354 tokens | 3/3 PASS | ~216 tok/s |
Prefill is the real cost of long context, not memory. A 78K prompt takes ~6 minutes to ingest before the first token. Good for documents you query repeatedly (llama.cpp caches the prefix); poor for one-shot huge prompts.
- vLLM. Documented minimum is compute capability 7.5;
sm_61is not in any official build's architecture list, and Triton refuses to compile for it. The PR adding Pascal support (#4409) was closed unmerged. Community forks require--enforce-eagerand--max-num-seqs 1, which removes the reason to use vLLM at all. - CUDA 13.x. Pascal offline compilation was removed in 13.0.
- bf16 anything. Pascal has no bf16, and FP16 runs at 1:64 of FP32
(~183 GFLOPS vs ~11.7 TFLOPS). This is why llama.cpp with
GGML_CUDA_FORCE_MMQ=ON(INT8/DP4A) is the right engine for this card and FP16-tensor-core-oriented stacks are not. (An earlier version of this README said to avoid IQ i-quant GGUFs on Pascal. That advice was measured and did not hold — see FINDINGS.md.)
scripts/ numbered setup steps + verification
docker/ Dockerfile for the sm_61 llama.cpp image
patches/ CUDA 12.9 glibc 2.43 header fix
bench/ throughput and needle-in-haystack harnesses
FINDINGS.md undocumented discoveries
MIT — see LICENSE. Not affiliated with NVIDIA, Alibaba, or the llama.cpp project.