Gateway API Inference Extension — EKS Lab

A hands-on lab demonstrating intelligent LLM inference routing on Amazon EKS using the Kubernetes Gateway API Inference Extension. Real GPU nodes, real routing decisions, observable in logs and Grafana in real time.

---

What This Lab Demonstrates

KV cache–aware load balancing — The Endpoint Picker Extension (EPP) scores every vLLM pod on three signals before routing each request: queue depth, KV cache utilisation, and prefix cache hit rate. You watch the weighted scores and selection reason in logs as load builds.

Dynamic LoRA adapter routing — A LoRA adapter is loaded into running vLLM pods at runtime via the API, no restart required. InferenceModelRewrite maps a stable client-facing model name (qwen2-lora) to the adapter (ru-lora). The EPP confirms per-pod adapter availability via vllm:lora_requests_info and routes accordingly.

The EPP does the model rewrite — not the gateway — Agentgateway delegates routing logic entirely to the EPP via ext-proc. The EPP parses the request body, resolves the InferenceModelRewrite mapping, scores pods, rewrites the model field, and returns the selected endpoint — all in one gRPC round trip before the request is forwarded.

---

Architecture

  Your laptop       ┌─────────────────────────────────────────────┐
  kubectl/curl ─────┤  AWS NLB → agentgateway-system              │
                    │  (agentgateway controller + Gateway CR)     │
                    │                                             │
                    │  ai-workloads namespace                     │
                    │  ┌─────────────┐   ┌─────────────┐          │
                    │  │  vllm pod 1 │   │  vllm pod 2 │          │
                    │  │  + ru-lora  │   │  + ru-lora  │          │
                    │  └─────────────┘   └─────────────┘          │
                    │  ┌──────────────────────────────────┐       │
                    │  │  EPP (Endpoint Picker Extension)  │      │
                    │  │  scrapes /metrics every 15s       │      │
                    │  │  KV cache · queue · prefix cache  │      │
                    │  │  InferenceModelRewrite lookup     │      │
                    │  └──────────────────────────────────┘       │
                    │                                             │
                    │  GPU NodePool (Karpenter — spot + on-demand)│
                    │  g4dn/g5/g6 — provisioned on pod schedule   │
                    └─────────────────────────────────────────────┘

---

Stack

Layer	Technology
Kubernetes	EKS 1.32, BYOCNI
CNI	Cilium (VXLAN overlay)
Node provisioning	Karpenter (GPU spot + on-demand fallback)
Gateway	Agentgateway (Gateway API conformant)
Inference routing	Gateway API Inference Extension — EPP
Model server	vLLM (Qwen/Qwen2-0.5B-Instruct)
LoRA adapter	sikoraaxd/Qwen2-0.5B-Instruct-ru-lora (dynamically loaded)
Observability	Prometheus + Grafana (official inference extension dashboard)

---

What You Will See

EPP scoring in logs — every request produces a trace like this:

incomingModelName: "qwen2"  →  targetModelName: "Qwen/Qwen2-0.5B-Instruct"

pod x7pqq   kv=0.007  queue=0  running=5   → weighted score: 3.99
pod 5ldmj   kv=0.032  queue=0  running=74  → weighted score: 6.94  ← SELECTED

reason: prefix-cache-scorer gave 5ldmj score=1 (weight 3) — cached prompt tokens

LoRA routing validated end-to-end — send "model":"qwen2-lora", receive a Cyrillic response proving the Russian adapter served the request:

{ "model": "ru-lora", "text": "Привет! Как я могу помочь вам сегодня?" }

Grafana dashboard — inference_pool_average_kv_cache_utilization, inference_pool_per_pod_queue_size, and inference_pool_ready_pods update in real time as load runs.

---

Prerequisites

• AWS account with permissions to create EKS clusters, IAM roles, and EC2 instances
• macOS or Linux workstation

brew install awscli eksctl helm kubectl jq cilium-cli

Configure AWS credentials:

aws configure   # Region: us-east-1, Output: json

---

Estimated Cost

Resource	Type	$/hr
EKS control plane	Managed	$0.10
m5.xlarge × 2 (system nodes)	On-demand	$0.38
g4dn.xlarge × 2 (GPU — vLLM)	Spot / on-demand fallback	$0.16–0.64
NLB × 2	Per hour	$0.016
Total		~$0.65–1.15/hr

A typical lab run (cluster up, demo, teardown) takes 2–3 hours.

---

Lab Guide

→ Full Step-by-Step Lab Guide

The guide covers all 9 parts end-to-end:

1. EKS cluster (BYOCNI — no VPC CNI)
2. Cilium CNI (VXLAN overlay)
3. Karpenter (GPU NodePool with spot + on-demand fallback)
4. Agentgateway + Gateway API CRDs
5. Prometheus + Grafana
6. vLLM deployment with dynamic LoRA support
7. EPP under load — live scoring logs
8. LoRA adapter routing validation
9. Scale to zero + full cleanup

---

Key Concepts

InferencePool — groups vLLM pods into a routable backend. The EPP is deployed alongside it and continuously scrapes /metrics from each pod.

InferenceModelRewrite — maps client-facing model names to backend model IDs. Resolved by the EPP, not the gateway.

EPP scoring pipeline — three plugins run per request, each producing a score per pod:

• queue-scorer (weight 2) — penalises pods with waiting requests
• kv-cache-utilization-scorer (weight 2) — penalises high GPU memory pressure
• prefix-cache-scorer (weight 3) — rewards pods with cached prompt prefixes

The pod with the highest weighted total is selected. Weights are configurable via inferenceExtension.pluginsCustomConfig in the InferencePool Helm chart.

Dynamic LoRA loading — vLLM exposes POST /v1/load_lora_adapter when started with --enable-lora and VLLM_ALLOW_RUNTIME_LORA_UPDATING=true. The EPP discovers loaded adapters via vllm:lora_requests_info and uses this for model-aware routing.