Inference Platform — vLLM vs Triton

Benchmarking platform that runs vLLM and NVIDIA Triton Inference Server side-by-side on a Lambda Labs A10 GPU VM, serving TinyLlama-1.1B. KEDA autoscales both engines based on Prometheus queue-depth metrics.

Purpose: compare throughput, latency, and autoscaling behavior between the two inference engines.

Stack

Component	Role
k3s	Lightweight Kubernetes on the VM
vLLM	OpenAI-compatible HTTP API (/v1/chat/completions)
Triton 23.10	KFServing v2 API, Python backend
KEDA	Event-driven autoscaling (1–5 replicas each)
Prometheus	Metrics scraping from both engines
Grafana	Dashboard + alerting
Locust	Load generation from local machine

Prerequisites

Lambda Labs A10 VM (or equivalent with NVIDIA GPU):

./scripts/check-prereqs.sh

Requires: Docker, kubectl/k3s, Helm, nvidia-smi, nvidia-ctk, Python 3.10+

Setup

# Full setup (~2 min for model download on A10)
./setup.sh

What it does:

1. Installs k3s and configures NVIDIA container runtime
2. Installs KEDA via Helm
3. Deploys namespace, Prometheus, Grafana, vLLM, Triton, KEDA ScaledObjects
4. Init containers pull TinyLlama weights (~1.1 GB) to PVCs on first run

Usage

# On the VM — keep running in a dedicated terminal
./scripts/port-forward.sh

# From your Mac — SSH tunnel
ssh -L 8000:localhost:8000 -L 8080:localhost:8080 \
    -L 9090:localhost:9090 -L 3000:localhost:3000 ubuntu@<VM_IP>

# Single-request latency check
./scripts/benchmark.sh

# Full load test (100 users, runs against both engines)
cd load-test && VM_IP=<your-ip> ./run_loadtest.sh

# Custom parameters
USERS=50 SPAWN_RATE=5 DURATION=2m cd load-test && ./run_loadtest.sh

Service Endpoints

Service	URL	Notes
vLLM	http://localhost:8000	OpenAI-compatible
Triton	http://localhost:8080	KFServing v2
Prometheus	http://localhost:9090
Grafana	http://localhost:3000	admin / inference123

Architecture

inference namespace (k3s)
├── vllm/           — Deployment + Service + PVC
├── triton/         — Deployment + Service + PVC + ConfigMap (model.py + config.pbtxt)
├── prometheus/     — Deployment + RBAC + DCGM exporter
├── grafana/        — Deployment + ConfigMap (dashboard JSON)
└── keda/           — ScaledObjects for vllm and triton

vLLM runs with bfloat16, GPU device. Init container downloads weights to PVC at /model-cache/tinyllama.

Triton uses a Python backend (triton-models/tinyllama/1/model.py) that wraps HuggingFace transformers. Model interface:

• Inputs: prompt (STRING), max_tokens (INT32, default 128, clamped to [1, 512])
• Outputs: generated_text (STRING), num_output_tokens (INT32)

KEDA ScaledObjects:

• vLLM: scale up when queue depth > 3 OR p99 latency > 20s
• Triton: scale up when pending requests > 3 OR avg queue wait > 5s
• Scale-up stabilization: 30s | Scale-down cooldown: 2 min

Useful Commands

# Watch pods / autoscaling
kubectl get pods -n inference -w
kubectl get hpa -n inference -w

# Logs
kubectl logs -n inference -l app=vllm -f
kubectl logs -n inference -l app=triton -f

# Free up VRAM by scaling one engine to zero
kubectl scale deployment triton -n inference --replicas=0

# After editing triton-models/tinyllama/ — re-apply and restart
kubectl apply -f k8s/triton/configmap.yaml
kubectl rollout restart deployment/triton -n inference

Load Test Results

Results (CSV + HTML) written to load-test/results/ after each run.

Teardown

./teardown.sh

Deletes all k8s resources in the inference namespace, uninstalls KEDA, and kills port-forwards. Does not uninstall k3s.