Optimizing LLM inference performance with Dual RTX 3090 + NVLink on Proxmox

[duart]

Optimizing LLM inference performance with 2x RTX 3090 + NVLink on Proxmox

I used AI to summarize my experiences, so the text below may not sound fully natural, even after I've edited it.

Problem

I was struggling to achieve the benchmarked performance declared in documentation using 2x RTX 3090 GPUs with NVLink. After spending several hours investigating potential solutions, I finally achieved a 3x performance improvement (from ~20 tokens per second to 60 tokens per second ) through VM configuration changes. It still not perfect, but already enough to make this configuration usable for agentic tasks.

Hardware & Software Stack

Component	Specification
Server	Dell T630
GPUs	2x RTX 3090 with NVLink
RAM	> 32GB DDR4
Hypervisor	Proxmox
Workload	Dedicated VM for LLM inference, 2x RTX3090 PCI pass-through to VM

The Solution

Three key optimizations were essential to achieving the expected performance:

1. CPU Dedication (CPU Pinning)

Pinning vCPUs to specific physical cores prevents the hypervisor from migrating VM threads, which can cause performance degradation and unpredictable latency spikes. This is important for latency-sensitive workloads like LLM inference (as some forums suggest).

2. HugePages

Enabling HugePages reduces the number of pages the CPU must manage, improving memory performance. Instead of managing hundreds of thousands of default 4KB pages, HugePages with 1GB pages, reducing TLB misses and memory translation overhead (as some forums suggest).

3. Better CPU frequencies setup

Setting the CPU scaling governor to ~ performance (accelerator-performance in tuned ) mode ensures the CPU runs at optimal frequencies by maintaining constant clock speeds at the highest available frequency within limits. This prevents the CPU from scaling down during idle periods and eliminates latency from frequency transitions.

Results

Configuration	Throughput
Before optimization	~20 tokens per second
After optimization	~60 tokens per second
Improvement	3x

Key Takeaways

• VM configuration matters: Even with server class hardware (Dell server, dual RTX 3090 + NVLink), improper hypervisor anbd host settings can be a bottleneck for performance.
• CPU pinning also matters: For stable, high-performance workloads, dedicate specific CPU cores to your LLM inference VM
• HugePages provide significant gains: Memory management optimizations can give substantial performance improvements for memory-intensive workloads.
• CPU frequency management: CPU frequency management plays vital role.

If you're running LLM inference on Proxmox with similar types of GPUs, ensure you've properly configured CPU dedication and frequencies management and HugePages before assuming your hardware is underperforming.

---

This experience highlights that achieving benchmarked performance requires attention to the entire stack—not just the GPU hardware.
Some references:

• Proxmox HugePages for VMs - DEV Community (35%)
• Proxmox Virtual Machine optimization - Deep Dive - DEV Community (22%)
• Hey Proxmox & Community - Let's talk about resources isolation (18%)
• Windows VM really Bad Memory Performance! | Proxmox Support... (17%)
• best settings for better LLM performance? - Proxmox Support Forum (8%)

Complete Guide: Proxmox VM Optimization for LLM Inference

📋 Overview

Goal: Improve LLM inference performance from ~20 tokens/second to 60+ tokens/second on VM with dual RTX 3090 GPUs.

Hardware: Proxmox host with dual RTX 3090s (NVLink), VM ID 200 in my case

Result Achieved: All optimizations successfully implemented and verified ✅

---

🎯 Optimizations Applied

Optimization	Benefit	Status
1GB Hugepages	Reduced memory overhead	✅ Active
CPU Performance Mode	Max CPU frequency	✅ Active
Memory Compaction	Ensures hugepage allocation	✅ Automatic
NUMA Alignment	CPUs + memory + GPUs on Node 0	✅ Configured
1:1 vCPU Pinning	Cache locality, no migration	✅ Working
Overhead Isolation	Non-vCPU threads on separate cores	✅ Working

Expected Total Gain: 200% improvement (20 → 60 tokens/second)

---

🔧 Settings Needed

1. Host System Settings

GRUB Configuration

# /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pci=realloc 
isolcpus=6,8,10,12,14,16,18,20 nohz_full=6,8,10,12,14,16,18,20 
rcu_nocbs=6,8,10,12,14,16,18,20 default_hugepagesz=1G hugepagesz=1G"



# Update GRUB
update-grub
reboot

Sysctl Configuration

# /etc/sysctl.conf
vm.nr_hugepages = 80
vm.compaction_proactiveness = 20
vm.extfrag_threshold = 500

# Apply settings
sysctl -p

Tuned Profile (better CPUs frequency management for GPU accelerators)

# Install tuned if not present
apt install tuned

# Enable and start tuned
systemctl enable tuned
systemctl start tuned

# Set performance profile
tuned-adm profile accelerator-performance

# Verify
tuned-adm active

---

2. VM 200 Configuration

# VM ID
VMID=200

# Memory: 32GB
qm set $VMID --memory 32768

# CPU: 8 vCPUs
qm set $VMID --vcpus 8 --cores 8 --cpulimit 8 --sockets 1

# CPU flags: 1GB page support
qm set $VMID --cpu host,flags=+pdpe1gb,+kvm_pv_eoi,+kvm_pv_unhalt

# Hugepages: 1GB pages
qm set $VMID --hugepages 1024 --keephugepages 1

# NUMA configuration
qm set $VMID --numa 1
qm set $VMID --numa0 cpus=0-7,hostnodes=0,memory=32768,policy=bind

# Verify configuration
qm config $VMID | grep -E "cpu|vcpu|core|sockets|numa|hugepages|memory"

Expected output:

cpu: host,flags=+pdpe1gb,+kvm_pv_eoi,+kvm_pv_unhalt
cores: 8
cpulimit: 8
hugepages: 1024
keephugepages: 1
memory: 32768
numa: 1
numa0: cpus=0-7,hostnodes=0,memory=32768,policy=bind
sockets: 1
vcpus: 8

---

💻 Code to Achieve (AI generated by Claude Sonnet)

Hook Script (configuration need to be applied before and after VM runs)

cat > /var/lib/vz/snippets/hook-200.sh << 'EOF'
#!/bin/bash
# Unified hook script for VM 200 with vCPU pinning and memory compaction

VMID=200
PHASE="${2}"
CORES=(6 8 10 12 14 16 18 20)  # Host cores for 1:1 vCPU mapping

# Only run for VM 200
if [ "$VMID" != "${1}" ]; then
    exit 0
fi

echo "[$(date)] Hook script for VM $VMID: Phase = $PHASE"

case "$PHASE" in
    pre-start)
        echo "[$(date)] Pre-start: Compacting memory for hugepages..."
        sync
        echo 1 > /proc/sys/vm/compact_memory
        echo "[$(date)] Pre-start: Memory compaction complete"
        ;;
    
    post-start)
        echo "[$(date)] Post-start: Starting 1:1 vCPU pinning..."
        
        # Get QEMU PID
        PID=$(ps aux | grep qemu | grep "\-id $VMID" | grep -v grep | awk '{print $2}')
        
        if [ -z "$PID" ]; then
            echo "[$(date)] Post-start: ERROR - VM $VMID not running"
            exit 1
        fi
        
        echo "[$(date)] Post-start: VM $VMID PID: $PID"
        
        # Wait for all vCPU threads to appear (max 30 seconds)
        echo "[$(date)] Post-start: Waiting for all 8 vCPU threads to appear..."
        expected_vcpus=8
        timeout=30
        waited=0
        
        while [ $waited -lt $timeout ]; do
            VCPU_TIDS=($(ps -T -p $PID | grep "CPU.*KVM" | awk '{print $2}'))
            vcpu_count=${#VCPU_TIDS[@]}
            
            echo "[$(date)] Post-start: Found $vcpu_count vCPU threads (expected $expected_vcpus)"
            
            if [ "$vcpu_count" -ge "$expected_vcpus" ]; then
                echo "[$(date)] Post-start: All $expected_vcpus vCPU threads are now available"
                break
            fi
            
            sleep 2
            waited=$((waited + 2))
        done
        
        if [ $waited -ge $timeout ]; then
            echo "[$(date)] Post-start: WARNING - Timeout after $timeout seconds"
            echo "[$(date)] Post-start: Only found $vcpu_count vCPU threads"
        fi
        
        # Get final vCPU thread list
        VCPU_TIDS=($(ps -T -p $PID | grep "CPU.*KVM" | awk '{print $2}'))
        
        if [ ${#VCPU_TIDS[@]} -eq 0 ]; then
            echo "[$(date)] Post-start: ERROR - Could not find vCPU threads"
            exit 1
        fi
        
        echo "[$(date)] Post-start: Found ${#VCPU_TIDS[@]} vCPU threads"
        
        # Pin each vCPU thread to its dedicated core
        for i in "${!VCPU_TIDS[@]}"; do
            TID=${VCPU_TIDS[$i]}
            CORE=${CORES[$i]}
            
            if [ -n "$TID" ] && [ -n "$CORE" ]; then
                echo "[$(date)] Post-start: vCPU $i (tid $TID) → host core $CORE"
                taskset -cp $CORE $TID
            fi
        done
        
        # Move non-vCPU overhead threads to non-isolated cores
        echo "[$(date)] Post-start: Moving overhead threads to cores 0,2,4..."
        for tid in $(ls /proc/$PID/task/); do
            name=$(cat /proc/$PID/task/$tid/comm 2>/dev/null)
            if [[ "$name" != *"KVM"* ]]; then
                taskset -cp 0,2,4 $tid 2>/dev/null
            fi
        done
        
        echo "[$(date)] Post-start: Done. Overhead threads moved to cores 0,2,4"
        echo "[$(date)] Post-start: vCPU pinning complete (pinned ${#VCPU_TIDS[@]} vCPUs)"
        ;;
        
    pre-stop|post-stop)
        echo "[$(date)] $PHASE: No action needed"
        ;;
esac

exit 0
EOF

# Make executable
chmod +x /var/lib/vz/snippets/hook-200.sh

# Attach to VM
qm set 200 --hookscript local:snippets/hook-200.sh

---

✅ Code to Test

1. Verify VM Configuration

# Check VM configuration
qm config 200 | grep -E "cpu|vcpu|core|sockets|numa|hugepages|memory"

# Expected output:
# cpu: host,flags=+pdpe1gb,+kvm_pv_eoi,+kvm_pv_unhalt
# cores: 8
# cpulimit: 8
# hugepages: 1024
# keephugepages: 1
# memory: 32768
# numa: 1
# numa0: cpus=0-7,hostnodes=0,memory=32768,policy=bind
# sockets: 1
# vcpus: 8

---

2. Verify Hugepages

# Check hugepages allocation and usage
cat /proc/meminfo | grep -i huge

# Expected output (VM running, similar to below):
# HugePages_Total:      80
# HugePages_Free:       48
# HugePages_Rsvd:       0
# HugePages_Surp:       0
# Hugepagesize:    1048576 kB

---

3. Verify vCPU Pinning

# Get VM PID
VM_PID=$(ps aux | grep qemu | grep "\-id 200" | grep -v grep | awk '{print $2}')

# Check vCPU threads and their CPUs
ps -T -p $VM_PID -o tid,comm,psr | grep "CPU"

# Expected output (similar to below):
#   862546 CPU 0/KVM         6
#   862547 CPU 1/KVM         8
#   862548 CPU 2/KVM        10
#   862549 CPU 3/KVM        12
#   862550 CPU 4/KVM        14
#   862551 CPU 5/KVM        16
#   862552 CPU 6/KVM        18
#   862553 CPU 7/KVM        20

---

4. Verify Tuned Profile

# Check active tuned profile
tuned-adm active

# Expected output:
# Current active profile: accelerator-performance

---

5. Verify Hook Script Execution

# Watch hook script logs
tail -f /var/log/syslog | grep -E "Hook script|Pre-start|Post-start"

# Expected output on VM start (similar to below):
# [Mon May 18 11:05:55 2026] Hook script for VM 200: Phase = pre-start
# [Mon May 18 11:05:55 2026] Pre-start: Compacting memory for hugepages...
# [Mon May 18 11:05:55 2026] Pre-start: Memory compaction complete
# [Mon May 18 11:06:01 2026] Hook script for VM 200: Phase = post-start
# [Mon May 18 11:06:01 2026] Post-start: Found 8 vCPU threads
# [Mon May 18 11:06:01 2026] Post-start: vCPU pinning complete (pinned 8 vCPUs)

---

6. Verify NUMA Alignment

# Check NUMA node for GPUs
lspci -nn | grep -E "VGA|3D|Display"

# Check NUMA node for CPUs
lscpu | grep -E "NUMA node|CPU\(s\)"

# Expected: GPUs and isolated CPUs (6,8,10,12,14,16,18,20) on NUMA Node 0

---

7. Monitor CPU Usage

# Real-time CPU usage
htop

# Or use mpstat
mpstat -P ALL 1

# Expected: High usage on CPUs 6,8,10,12,14,16,18,20 (vCPUs)
# Low usage on CPUs 0,2,4 (overhead)

---

8. Monitor GPU Usage

# Inside the VM
nvidia-smi

# Monitor in real-time
watch -n 1 nvidia-smi

# Expected: Both GPUs being utilized during inference

---

9. Performance Test (Inside VM)

# Test with Ollama
ollama run llama3 --verbose

# Monitor tokens/second in output
# Expected: ~ 60 tokens/second in CODE bench (~ 3x improvement from ~20)

URL=http://localhost:8010 CONTAINER=vllm-qwen36-27b-dual ./scripts/bench.sh  && ./scripts/verify-stress.sh

---

10. Complete Verification Script

#!/bin/bash
echo "=== Complete Verification for VM 200 ==="
echo ""

# 1. VM Configuration
echo "1. VM Configuration:"
qm config 200 | grep -E "cpu|vcpu|core|sockets|numa|hugepages|memory"
echo ""

# 2. Hugepages
echo "2. Hugepages:"
cat /proc/meminfo | grep -i huge
echo ""

# 3. vCPU Pinning
echo "3. vCPU Pinning:"
VM_PID=$(ps aux | grep qemu | grep "\-id 200" | grep -v grep | awk '{print $2}')
ps -T -p $VM_PID -o tid,comm,psr | grep "CPU"
echo ""

# 4. Tuned Profile
echo "4. Tuned Profile:"
tuned-adm active
echo ""

# 5. Sysctl Settings
echo "5. Sysctl Settings:"
sysctl vm.nr_hugepages vm.compaction_proactiveness vm.extfrag_threshold
echo ""

echo "=== Verification Complete ==="

---

📊 Architecture Summary

┌─────────────────────────────────────────────────────────────┐
│                    Proxmox Host (pvedell)                   │
├─────────────────────────────────────────────────────────────┤
│  NUMA Node 0 (GPU Node)                                     │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐          │
│  │ Host CPU 6  │  │ Host CPU 8  │  │ Host CPU 10 │          │
│  │ vCPU 0      │  │ vCPU 1      │  │ vCPU 2      │          │
│  └─────────────┘  └─────────────┘  └─────────────┘          │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐          │
│  │ Host CPU 12 │  │ Host CPU 14 │  │ Host CPU 16 │          │
│  │ vCPU 3      │  │ vCPU 4      │  │ vCPU 5      │          │
│  └─────────────┘  └─────────────┘  └─────────────┘          │
│  ┌─────────────┐  ┌─────────────┐                           │
│  │ Host CPU 18 │  │ Host CPU 20 │                           │
│  │ vCPU 6      │  │ vCPU 7      │                           │
│  └─────────────┘  └─────────────┘                           │
│  ┌─────────────┐  ┌─────────────┐                           │
│  │ Host CPU 0  │  │ Host CPU 2  │                           │
│  │ Overhead    │  │ Overhead    │                           │
│  └─────────────┘  └─────────────┘                           │
│                                                             │
│  ┌─────────────┐  ┌─────────────┐                           │
│  │ RTX 3090 #1 │  │ RTX 3090 #2 │  NVLink Bridge            │
│  │ GPU 0       │  │ GPU 1       │  (PCIe x16)               │
│  └─────────────┘  └─────────────┘                           │
│                                                             │
│  Memory: 32GB (1GB hugepages) → NUMA Node 0                 │
└─────────────────────────────────────────────────────────────┘

---

🎯 Expected Performance

Metric	Before	After	Improvement
Tokens/second	~20	~60	~ 3x
Latency	Higher	Lower	Better responsiveness
CPU Utilization	Suboptimal	Optimal	All 8 vCPUs efficient
Memory Overhead	Higher	Lower	1GB pages
NUMA Locality	Poor	Excellent	All on Node 0

---

📝 Key Files Reference

File	Purpose
/etc/default/grub	Kernel parameters (hugepages, CPU isolation)
/etc/sysctl.conf	Runtime sysctl settings
/etc/pve/qemu-server/200.conf	VM configuration
/var/lib/vz/snippets/hook-200.sh	Hook script (memory compaction + vCPU pinning)

---

✅ Checklist

• GRUB configured with 1GB hugepages and CPU isolation
• Sysctl settings applied (hugepages, compaction)
• Tuned profile set to accelerator-performance
• VM configured with 8 vCPUs, 32GB memory, 1GB hugepages
• NUMA configured (cpus=0-7, hostnodes=0)
• Hook script created and attached
• vCPU pinning verified (8 vCPUs → 8 cores)
• Hugepages verified (32GB allocated to VM)
• Performance tested (~ 60 CODE benchmark tokens/second)

[noonghunna]

Really appreciate you writing this up @duart — and for flagging it's AI-summarized. The host-tuning checklist (NUMA alignment, CPU pinning, HugePages, performance governor) is solid general hygiene and the meta-point ("attention to the whole stack, not just the GPU") is right.

One reframe that should help you recover the rest of the performance, not just the 3×:

Our own reference rig also runs under Proxmox — 2× RTX 3090, PCIe-only (no NVLink), GPUs passed through to the VM, and zero of your optimizations applied (stock governor, no HugePages, no manual pinning). It sustains the documented dual-card baselines untuned: dual.yml ~69/89, dual-turbo ~81/108, dual-dflash ~127 code tok/s. So out-of-box Proxmox is not an inherent 3× tax — the ~20 tok/s you started from was a specific pathology, and your tuned ~60 is still below our untuned PCIe-only numbers.

The one hardware difference between your rig and ours explains both: you have NVLink across two passed-through GPUs; we don't. NVLink-over-VM-passthrough is a notoriously fragile path — when IOMMU grouping / ACS / NUMA placement is off, NCCL silently can't use the NVLink peer link and collapses to a slow cross-bridge fallback (exactly a ~3×-down floor). Your NUMA-alignment + pinning changes most likely let NVLink actually engage, rather than fixing "Proxmox" per se. This is a known recurring class here — see #137 (NVLink-not-engaging under container/VM passthrough).

The tell: a correctly engaged NVLink 2×3090 runs ~10–15% faster than our PCIe-only baseline (per our controlled NVLink A/Bs) — comfortably north of 80–90 on the same workload, not 60. So you've recovered most of a collapsed-NVLink penalty but likely not all of it; there's real headroom left.

To confirm whether NVLink is now actually being used and find the remaining gap: bash scripts/report.sh --full and share it (a Q&A discussion is ideal). It now includes a PCIe/NVLink topology + lspci + NVLink-detection section built for exactly this — it'll show definitively whether NCCL is on the NVLink peer link or still falling back, plus your model/quant/compose so the 60-vs-baseline comparison is apples-to-apples.

Thanks again — the NUMA-alignment angle for multi-socket passthrough hosts is genuinely useful; I've folded a credited pointer into our container-runtimes notes.