Setting up vLLM on NVIDIA Blackwell RTX Pro 6000 GPUs (Ubuntu 24.04 Server)

This guide documents a clean installation of CUDA 12.8 + NVIDIA drivers + vLLM on Ubuntu 24.04 x86_64, tested with NVIDIA RTX Pro 6000 (Blackwell) cards. It's based on NVIDIA's official instructions, vLLM community fixes, and extensive testing with multi-GPU Blackwell hardware.

⚡ Quick Install (TL;DR)

For experienced users who want the essential steps:

# 0. Clean old drivers
sudo apt-get remove --purge '^nvidia-.*'
sudo apt autoremove
sudo apt install -y pkg-config libglvnd-dev dkms build-essential \
    libegl-dev libegl1 libgl-dev libgl1 libgles-dev libgles1 \
    libglvnd-core-dev libglx-dev libopengl-dev \
    gcc make screen nano isc-dhcp-client python3-venv python3-pip

# 1. Install CUDA 12.8
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt-get update
sudo apt-get -y install cuda-toolkit-12-8

# 2. Install NVIDIA driver (choose one)
sudo apt-get install -y cuda-drivers   # proprietary (recommended)
# or
sudo apt-get install -y nvidia-open    # open kernel modules
sudo reboot

# 3. Python environment
python3 -m venv vllm-env
source vllm-env/bin/activate
pip install --upgrade pip setuptools wheel

# 4. Install PyTorch + NCCL for CUDA 12.8
pip install torch==2.8.0+cu128 torchvision --index-url https://download.pytorch.org/whl/cu128
pip install nvidia-nccl-cu12==2.27.3

# 5. Install vLLM
git clone https://github.com/vllm-project/vllm.git
cd vllm
pip install -e .

Run with 2× RTX Pro 6000 GPUs:

VLLM_ATTENTION_BACKEND=FLASHINFER \
torchrun --nproc_per_node=2 -m vllm.entrypoints.api_server \
    --model /path/to/model \
    --host 0.0.0.0 --port 9015 \
    --dtype float16 \
    --max-model-len 16384 \
    --gpu-memory-utilization 0.92 \
    --kv-cache-dtype auto \
    --calculate-kv-scales \
    --tensor-parallel-size 2 \
    --trust-remote-code

👉 Continue reading for detailed explanations and troubleshooting.

---

Introduction

Setting up NVIDIA GPUs on Linux can be challenging, especially with newer architectures. This guide provides a clean, step-by-step recipe for getting RTX Pro 6000 (Blackwell) cards running with vLLM on Ubuntu 24.04 server.

Between conflicting driver installs, CUDA version mismatches, and PyTorch's evolving support for Blackwell's sm_120 architecture, many common approaches fail. This guide eliminates the guesswork with a tested configuration that works.

Prerequisites

• Ubuntu 24.04 Server (x86_64)
• NVIDIA RTX Pro 6000 (Blackwell) GPU(s)
• Root or sudo access
• Internet connection

Architecture Overview

Understanding the dependency chain helps prevent issues:

   ┌──────────────────┐
   │      vLLM        │  ← Serving/inference engine
   ├──────────────────┤
   │     PyTorch      │  ← Must match CUDA version
   ├──────────────────┤
   │      NCCL        │  ← Multi-GPU communication
   ├──────────────────┤
   │   CUDA 12.8      │  ← Toolkit/runtime libraries
   ├──────────────────┤
   │  NVIDIA Driver   │  ← Open or proprietary
   ├──────────────────┤
   │  Ubuntu 24.04    │
   └──────────────────┘

Key principle: Install from the bottom up. Each layer depends on the ones below it.

Common Pitfalls to Avoid

Before starting, be aware of these critical issues:

1. Always purge old drivers completely - Partial removals cause conflicts
2. Use CUDA 12.8 or newer - Earlier versions don't support Blackwell
3. Pin NCCL version - vLLM requires nvidia-nccl-cu12==2.27.3 for multi-GPU. There are known bugs with earlier versions.
4. Match PyTorch to CUDA - Mismatched versions won't recognize sm_120
5. Use virtual environments - System Python conflicts are hard to debug

Detailed Installation Steps

Step 0: Clean Previous Installations

Remove any existing NVIDIA software:

sudo apt-get remove --purge '^nvidia-.*'
sudo apt autoremove

Install required build dependencies:

sudo apt install -y pkg-config libglvnd-dev dkms build-essential \
    libegl-dev libegl1 libgl-dev libgl1 libgles-dev libgles1 \
    libglvnd-core-dev libglx-dev libopengl-dev \
    gcc make screen nano isc-dhcp-client python3-venv python3-pip

Step 1: Install CUDA 12.8 Toolkit

Add NVIDIA's repository and install CUDA:

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt-get update
sudo apt-get -y install cuda-toolkit-12-8

Step 2: Install NVIDIA Drivers

Choose between proprietary (recommended) or open-source drivers:

# Option A: Proprietary drivers (recommended for stability)
sudo apt-get install -y cuda-drivers

# Option B: Open-source kernel modules
sudo apt-get install -y nvidia-open

Reboot the system:

sudo reboot

Verify installation:

nvidia-smi

You should see your GPUs listed with driver version 570+ and CUDA 12.8.

Step 3: Setup Python Environment

Create an isolated environment:

python3 -m venv vllm-env
source vllm-env/bin/activate
pip install --upgrade pip setuptools wheel

Step 4: Install PyTorch and NCCL

Install PyTorch with CUDA 12.8 support:

pip install torch==2.8.0+cu128 torchvision --index-url https://download.pytorch.org/whl/cu128

Install the correct NCCL version for multi-GPU:

pip install nvidia-nccl-cu12==2.27.3

Step 5: Install vLLM

Clone and install from source (recommended for latest fixes):

git clone https://github.com/vllm-project/vllm.git
cd vllm
pip install -e .

Alternatively, install from PyPI:

pip install vllm

Step 6: Run vLLM Server

Example configuration for dual RTX Pro 6000:

VLLM_ATTENTION_BACKEND=FLASHINFER \
torchrun --nproc_per_node=2 -m vllm.entrypoints.api_server \
    --model /path/to/your/model \
    --host 0.0.0.0 \
    --port 9015 \
    --dtype float16 \
    --max-model-len 16384 \
    --gpu-memory-utilization 0.92 \
    --kv-cache-dtype auto \
    --calculate-kv-scales \
    --tensor-parallel-size 2 \
    --trust-remote-code

Verification

Run these commands to verify your setup:

# Check GPU visibility
nvidia-smi

# Verify CUDA version
nvcc --version

# Check NCCL installation
python -c "import torch; print(f'NCCL version: {torch.cuda.nccl.version()}')"

# Test PyTorch GPU access
python -c "import torch; print(f'CUDA available: {torch.cuda.is_available()}')"
python -c "import torch; print(f'GPU count: {torch.cuda.device_count()}')"

Troubleshooting

GPU Not Detected

• Ensure drivers are properly installed: sudo dmesg | grep nvidia
• Check if secure boot is disabled in BIOS
• Verify kernel module is loaded: lsmod | grep nvidia

CUDA Version Mismatch

• Check installed versions: nvcc --version vs nvidia-smi
• Ensure PyTorch matches CUDA: python -c "import torch; print(torch.version.cuda)"

Multi-GPU Communication Issues

• Verify NCCL: python -c "import torch; print(torch.cuda.nccl.version())"
• Check peer access: nvidia-smi topo -m
• Ensure all GPUs are on the same NUMA node if possible

vLLM Import Errors

• Rebuild vLLM after driver updates: pip install -e . --force-reinstall
• Check for conflicting packages: pip list | grep -E "torch|cuda|nccl"

Performance Optimization

For production deployments:

1. GPU Memory: Start with --gpu-memory-utilization 0.9 and increase carefully
2. Batch Size: Adjust --max-num-seqs based on your workload
3. KV Cache: Use --kv-cache-dtype fp8 for memory savings (if supported)
4. Tensor Parallelism: Match --tensor-parallel-size to your GPU count

References

• NVIDIA CUDA Installation Guide
• vLLM Documentation
• PyTorch CUDA Compatibility
• NCCL Documentation
• vLLM Multi-GPU Issue #19166

Contributing

Found an issue or have improvements? Please open an issue or submit a PR. Blackwell-specific optimizations and experiences are especially welcome.

License

This guide is provided as-is under the MIT License. Use at your own risk.

---

Tested Configuration:

• Ubuntu 24.04.1 LTS
• NVIDIA Driver 570.86.16
• CUDA 12.8
• PyTorch 2.8.0+cu128
• vLLM 0.6.4
• 2× RTX Pro 6000 Ada (96GB VRAM total)