Skip to content

Latest commit

 

History

2 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Tesla P40 → WSL2 → CUDA → llama.cpp

Getting a NVIDIA Tesla P40 (Pascal, sm_61, 24 GB) working for local LLM serving under WSL2 on Windows 11, with Docker GPU containers — and benchmarking it on a modern 35B MoE model.

This is a working, verified setup, not a sketch. Every number below was measured on the machine described in Tested configuration.

Result: a 2016 datacenter card serving Qwen3.6-35B-A3B at ~51 tok/s with 128K context.


Why this repo exists

Three separate things block a P40 under WSL2, and none of them produce an error message that points at the actual cause:

  1. The GPU is invisible to WSL2 in TCC driver model — you get Failed to initialize NVML: N/A inside Linux while nvidia-smi works fine on Windows. No amount of reinstalling CUDA fixes this. WDDM is license-locked on Tesla cards; MCDM is the answer, and that is not documented for Pascal.
  2. CUDA 13.x cannot target Pascal at all — offline compilation for sm_61 was removed in CUDA 13.0. You must pin to CUDA 12.x.
  3. CUDA 12.9 does not compile on modern glibc — glibc 2.41/2.42 added C23 math functions that collide with CUDA's headers. NVIDIA fixed it in 13.2.1, which cannot target Pascal. You must patch.

Plus a fourth, if you build llama.cpp yourself: libcuda.so.1 does not exist at Docker build time, only at run time.

See FINDINGS.md for the discoveries that are not documented elsewhere.


Tested configuration

Component Version
GPU NVIDIA Tesla P40, 24 GB, compute capability 6.1
Secondary GPU AMD Radeon iGPU (drives display)
Windows 11 Pro, build 26200
NVIDIA driver 582.70
WSL 2.7.10, kernel 6.18.33
Distro Ubuntu 26.04 LTS (glibc 2.43, gcc 15)
CUDA toolkit 12.9.2
Docker Engine 29.6.2 (native in WSL, not Docker Desktop)
NVIDIA Container Toolkit 1.19.0

Other Pascal cards (P100, P4, GTX 10-series) should follow the same path, but only the P40 was tested. Newer GPUs do not need any of this.


Quick start

Run these in order. Steps 1 is Windows-side; the rest run inside WSL.

# 1. Windows, ELEVATED PowerShell. Switches the P40 to MCDM. REQUIRES A REBOOT.
.\scripts\01-windows-enable-mcdm.ps1

Reboot, then inside your WSL distro:

sudo ./scripts/02-wsl-cuda-toolkit.sh    # CUDA 12.9 + gcc-14 + glibc patch
sudo ./scripts/03-wsl-docker-nvidia.sh   # Docker Engine + container toolkit + CDI
./scripts/04-verify-stack.sh             # 12-point verification, run as your user
./scripts/05-build-llamacpp.sh           # builds llamacpp-p40:latest for sm_61

Then fetch a model and serve it:

mkdir -p /opt/models/mtp && cd /opt/models/mtp
curl -L -C - -O "https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF/resolve/main/Qwen3.6-35B-A3B-UD-Q3_K_XL.gguf?download=true"
./scripts/06-run-server.sh

04-verify-stack.sh is the one to run if anything misbehaves later — it isolates which layer broke.


Measured results

Qwen3.6-35B-A3B, UD-Q3_K_XL (16.85 GB / 17.23 GB MTP), full GPU offload.

Throughput

Configuration Generation
Baseline, no speculative decoding 37.9 tok/s
MTP weights, spec decoding off (control) 38.3 tok/s
MTP, --spec-draft-n-max 6 34.5 tok/s
MTP, --spec-draft-n-max 4 41.9 tok/s
MTP, --spec-draft-n-max 3 (default) 48.3 tok/s
MTP, --spec-draft-n-max 2 51.6 tok/s

Prefill (pp512, llama-bench): 982 tok/s.

The control row matters: MTP weights with speculation disabled land on baseline, which is what proves the speedup is real speculative decoding rather than file-to-file variance.

Context scaling with -ctk q8_0 -ctv q8_0

Context VRAM Generation
4096 (KV f16) 21710 MiB 51.45 tok/s
8192 21732 MiB 50.34 tok/s
16384 21836 MiB 50.95 tok/s
32768 22086 MiB 50.79 tok/s
65536 22614 MiB 50.85 tok/s
131072 22782 MiB 50.54 tok/s

32× the context for ~1 GB and ~2% throughput — see FINDINGS.md.

Long-context correctness

Needle-in-a-haystack, needle at 10% / 50% / 90% depth — 9/9 exact retrievals:

Prompt size Result Prefill
6315 tokens 3/3 PASS ~790 tok/s
25111 tokens 3/3 PASS 422–609 tok/s
78354 tokens 3/3 PASS ~216 tok/s

Prefill is the real cost of long context, not memory. A 78K prompt takes ~6 minutes to ingest before the first token. Good for documents you query repeatedly (llama.cpp caches the prefix); poor for one-shot huge prompts.


What does not work

  • vLLM. Documented minimum is compute capability 7.5; sm_61 is not in any official build's architecture list, and Triton refuses to compile for it. The PR adding Pascal support (#4409) was closed unmerged. Community forks require --enforce-eager and --max-num-seqs 1, which removes the reason to use vLLM at all.
  • CUDA 13.x. Pascal offline compilation was removed in 13.0.
  • bf16 anything. Pascal has no bf16, and FP16 runs at 1:64 of FP32 (~183 GFLOPS vs ~11.7 TFLOPS). This is why llama.cpp with GGML_CUDA_FORCE_MMQ=ON (INT8/DP4A) is the right engine for this card and FP16-tensor-core-oriented stacks are not. (An earlier version of this README said to avoid IQ i-quant GGUFs on Pascal. That advice was measured and did not hold — see FINDINGS.md.)

Repo layout

scripts/    numbered setup steps + verification
docker/     Dockerfile for the sm_61 llama.cpp image
patches/    CUDA 12.9 glibc 2.43 header fix
bench/      throughput and needle-in-haystack harnesses
FINDINGS.md undocumented discoveries

License

MIT — see LICENSE. Not affiliated with NVIDIA, Alibaba, or the llama.cpp project.

About

Run a Tesla P40 (Pascal, sm_61) under WSL2 for local LLM serving: MCDM driver model, CUDA 12.9, Docker GPU passthrough, and llama.cpp built for sm_61. Benchmarked on Qwen3.6-35B-A3B at ~51 tok/s with 128K context.

Topics

Resources

Stars

1 star

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages