[Bug]: FlashInfer sampler JIT fails on Blackwell SM120f with CUDA 13.3: CUDA compiler and toolkit headers are incompatible

[yzhbradoodrrpurp]

Your current environment

--2026-06-02 17:33:54-- https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/collect_env.py
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.110.133, 185.199.111.133, 185.199.108.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.110.133|:443... connected.
HTTP request sent, awaiting response...

🐛 Describe the bug

Description

I hit a FlashInfer sampler JIT compilation failure when launching vLLM with a Qwen3.5 model on NVIDIA RTX PRO 6000 Blackwell GPUs.

The model can be loaded by Transformers and vLLM can create the engine config successfully. vLLM also starts and responds correctly if I disable FlashInfer sampler with:

VLLM_USE_FLASHINFER_SAMPLER=0

However, when FlashInfer sampler is enabled, vLLM fails during startup while running the dummy sampler/profile stage. The failure happens in FlashInfer JIT compilation for cached_ops/sampling.

Environment

GPU: NVIDIA RTX PRO 6000 Blackwell Server Edition
Architecture: SM120f / compute_120f
Python: 3.12.13
PyTorch: 2.11.0+cu130
torch.version.cuda: 13.0
vLLM: 0.22.1rc1.dev58+g68dafcca7
Transformers: 5.9.0
FlashInfer: 0.6.12.dev20260602
flashinfer-cubin: 0.6.12.dev20260602
flashinfer-jit-cache: Not installed
CUDA_HOME: /root/miniconda3/envs/verl/lib/python3.12/site-packages/nvidia/cu13
CUDA_VERSION: 13.3
FLASHINFER_CUDA_VERSION: 13.3
NVCC: 13.3.33
FLASHINFER_CUDA_ARCH_LIST: {(12, '0f')}

python -m flashinfer show-config reports:

FlashInfer version: 0.6.12.dev20260602
flashinfer-cubin version: 0.6.12.dev20260602
flashinfer-jit-cache: Not installed
Torch version: 2.11.0+cu130
CUDA runtime available: Yes
FLASHINFER_CUDA_ARCH_LIST: {(12, '0f')}
FLASHINFER_CUDA_VERSION: 13.3
CUDA_VERSION: 13.3
CUDA_HOME: /root/miniconda3/envs/verl/lib/python3.12/site-packages/nvidia/cu13
NVCC found: Yes

Reproduction

Launch vLLM with FlashInfer sampler enabled:

python -m vllm.entrypoints.openai.api_server \
  --model /root/autodl-tmp/models/Qwen3.5-4B \
  --tokenizer /root/autodl-tmp/models/Qwen3.5-4B \
  --tensor-parallel-size 1 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.4 \
  --enforce-eager \
  --port 8008

vLLM fails during startup.

If I disable FlashInfer sampler:

VLLM_USE_FLASHINFER_SAMPLER=0 \
python -m vllm.entrypoints.openai.api_server \
  --model /root/autodl-tmp/models/Qwen3.5-4B \
  --tokenizer /root/autodl-tmp/models/Qwen3.5-4B \
  --tensor-parallel-size 1 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.4 \
  --enforce-eager \
  --port 8008

the server starts successfully and /v1/chat/completions works.

Error

The failure happens while compiling FlashInfer sampling JIT ops:

subprocess.CalledProcessError: Command '['ninja', '-v', '-C',
'/root/.cache/flashinfer/0.6.12.dev20260602/120f/cached_ops/sampling',
'-f',
'/root/.cache/flashinfer/0.6.12.dev20260602/120f/cached_ops/sampling/build.ninja']'
returned non-zero exit status 1.

Manually running ninja shows:

FAILED: /root/.cache/flashinfer/0.6.12.dev20260602/120f/cached_ops/sampling/csrc_renorm.cuda.o

/root/miniconda3/envs/verl/lib/python3.12/site-packages/nvidia/cu13/bin/nvcc
...
-gencode=arch=compute_120f,code=sm_120f
...
-c /root/miniconda3/envs/verl/lib/python3.12/site-packages/flashinfer/data/csrc/renorm.cu
-o /root/.cache/flashinfer/0.6.12.dev20260602/120f/cached_ops/sampling/csrc_renorm.cuda.o

In file included from
/root/miniconda3/envs/verl/lib/python3.12/site-packages/flashinfer/data/cccl/libcudacxx/include/cuda/std/__cccl/extended_data_types.h:26,
...
/root/miniconda3/envs/verl/lib/python3.12/site-packages/flashinfer/data/cccl/libcudacxx/include/cuda/std/__cccl/cuda_toolkit.h:41:8:
error: #error "CUDA compiler and CUDA toolkit headers are incompatible, please check your include paths"
   41 | #      error "CUDA compiler and CUDA toolkit headers are incompatible, please check your include paths"
      |        ^~~~~

ninja: build stopped: subcommand failed.

Expected Behavior

FlashInfer sampler JIT should compile successfully for Blackwell SM120f with CUDA 13.3, or the wheel should provide compatible cached sampling kernels for this environment.

Additional notes

Before setting CUDA 13.3 paths correctly, flashinfer show-config reported CUDA 12.8 and printed:

Failed to get device capability: SM 12.x requires CUDA >= 12.9

After exporting CUDA 13.3 paths, that issue disappeared:

export CUDA_HOME=/root/miniconda3/envs/verl/lib/python3.12/site-packages/nvidia/cu13
export PATH=$CUDA_HOME/bin:$PATH
export LD_LIBRARY_PATH=$CUDA_HOME/lib:$LD_LIBRARY_PATH

But the sampler JIT still fails with the CCCL/libcudacxx header compatibility error above.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[martin3252]

Same issue.

[yzhbradoodrrpurp]

Same issue.

I'm closing this issue because I found the root cause was my local CUDA environment configuration, not FlashInfer itself.

In my environment there were multiple CUDA installations:

• System CUDA: /usr/local/cuda -> CUDA 12.8
• Python/conda environment CUDA: /root/miniconda3/envs/verl/lib/python3.12/site-packages/nvidia/cu13 -> CUDA 13.0
• PyTorch: torch 2.11.0+cu130

Initially, some FlashInfer/vLLM JIT paths were picking up /usr/local/cuda instead of the CUDA runtime/toolkit installed inside the Python environment. This caused warnings/errors such as:

Failed to get device capability: SM 12.x requires CUDA >= 12.9
CUDA_HOME: /usr/local/cuda
CUDA_VERSION: 12.8
FLASHINFER_CUDA_VERSION: 12.8

Since the GPU is a Blackwell GPU with SM 12.x, CUDA 12.8 was too old.

After switching CUDA_HOME, PATH, LD_LIBRARY_PATH, and LIBRARY_PATH to the CUDA 13 path inside the conda environment, I also found that my CUDA Python packages were mixed:

nvcc / cccl: 13.3
cuda runtime / headers: 13.0
PyTorch: cu130

This caused:

CUDA compiler and CUDA toolkit headers are incompatible

I fixed the environment by aligning the CUDA toolchain with PyTorch cu130:

uv pip install --force-reinstall --no-deps \
  nvidia-cuda-nvcc==13.0.88 \
  nvidia-cuda-crt==13.0.88 \
  nvidia-nvvm==13.0.88 \
  nvidia-cuda-cccl==13.0.85

Then I set:

export CUDA_HOME=/root/miniconda3/envs/verl/lib/python3.12/site-packages/nvidia/cu13
export PATH=$CUDA_HOME/bin:$PATH
export LD_LIBRARY_PATH=$CUDA_HOME/lib:$CUDA_HOME/lib64:$LD_LIBRARY_PATH
export LIBRARY_PATH=$CUDA_HOME/lib:$CUDA_HOME/lib64:$LIBRARY_PATH

After that, FlashInfer JIT/autotuning ran successfully.
So the original issue was caused by my environment accidentally using the system CUDA 12.8 / mixed CUDA components, rather than a FlashInfer bug.
For reference, the working setup is:

GPU: NVIDIA RTX PRO 6000 Blackwell Server Edition
PyTorch: 2.11.0+cu130
CUDA runtime/toolchain used by the Python env: CUDA 13.0
vLLM: 0.22.1rc1.dev58+g68dafcca7
FlashInfer: 0.6.12.dev20260602
Python: 3.12

[martin3252]

My issue was coming from OOM in RAM. export MAX_JOBS to a smaller number solved the issue.

[arquiteto-raiz]

Confirming the workaround on RTX PRO 6000 Blackwell Max-Q (sm_120), vLLM 0.27.1, CUDA 13 pip install (no system toolkit).

Hit the same crash at the warmup sampler run serving Qwen3.8-27B-FP8 — RuntimeError: Could not find nvcc and default cuda_home='/usr/local/cuda' doesn't exist. Note that pointing CUDA_HOME at the pip-bundled nvidia/cu13 nvcc is not a fix — the JIT then fails on the header incompatibility described here.

VLLM_USE_FLASHINFER_SAMPLER=0 resolves it. Full working command for reference (hybrid GDN model, 262k context):

VLLM_USE_DEEP_GEMM=0 VLLM_MOE_USE_DEEP_GEMM=0 VLLM_USE_FLASHINFER_SAMPLER=0 \
vllm serve Qwen/Qwen3.8-27B-FP8 --tensor-parallel-size 1 --max-model-len 262144 \
  --max-num-seqs 8 --max-num-batched-tokens 8192 --gpu-memory-utilization 0.90 \
  --enable-prefix-caching --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder

Boots clean, 875k-token KV cache, stable under systemd.

[CRPONCELET]

vLLM + FlashInfer + MTP on Blackwell sm_120 — CUDA 13.3 from pip, no sudo

Fixes: SM 12.x requires CUDA >= 12.9, then headers are incompatible, then
ld: cannot find -lcudart. Keeps FlashInfer + fp8 KV cache. $VENV = venv root;
CU13=$VENV/lib/python3.12/site-packages/nvidia/cu13.

1. nvcc 13.3 + point FlashInfer at it (else it reads /usr/local/cuda = 12.8):
   pip install nvidia-cuda-nvcc==13.3.73
   export CUDA_HOME="$CU13"

2. Arch list — the var is FLASHINFER_CUDA_ARCH_LIST, not TORCH_CUDA_ARCH_LIST:
   export FLASHINFER_CUDA_ARCH_LIST="12.0"

3. Align runtime headers to 13.3 (nvcc is 13.3 but runtime/nvrtc default to 13.0):
   pip install nvidia-cuda-runtime==13.3.29 nvidia-cuda-nvrtc==13.3.33

4. Give the linker libcudart (package has .so.13 but no .so, and off-path):
   ln -s libcudart.so.13 "$CU13/lib/libcudart.so"
   export LIBRARY_PATH="$CU13/lib:$LIBRARY_PATH"
   export LD_LIBRARY_PATH="$CU13/lib:$LD_LIBRARY_PATH"

Verify before launch:
printf '#include <cuda_runtime.h>\n__global__ void k(){}\n' > t.cu
"$CU13/bin/nvcc" -arch=sm_120 -I"$CU13/include" -c t.cu -o t.o

Result: FlashInfer JIT-compiles in 120f (fast path). Serve with
--speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'.

Gotcha: under set -u, put the env block AFTER VENV is defined, or the
script dies on "unbound variable" before vLLM starts.