v0.9.0 Release

llm-d v0.9.0 Release

Release goal and issues tracked here: #1945, although not all of that was accomplished. Thank you to all our new and old contributors.

LLM-D v0.9.0 Component Summary

Component	Version	Previous Version	Type
llm-d/llm-d-router-endpoint-picker	v0.10.0	v0.9.0	Image + Helm Chart
llm-d/llm-d-router-disagg-sidecar	v0.10.0	v0.9.0	Image
llm-d/llm-d-uds-tokenizer	—	v0.9.0	Image
llm-d/llm-d-kv-cache	v0.9.0	v0.8.0	Library
llm-d/llm-d-inference-sim	v0.10.2	v0.9.2	Image
llm-d/llm-d-cuda	v0.9.0	v0.8.1	Image
llm-d/llm-d-aws (EFA)	v0.9.0	v0.8.1	Image
llm-d/llm-d-rocm	v0.9.0	v0.8.1	Image
llm-d/llm-d-xpu	v0.9.0	v0.8.1	Image
llm-d/llm-d-xpu-sglang	v0.9.0	—	Image
llm-d/llm-d-cpu	v0.9.0	v0.8.1	Image
llm-d/llm-d-kv-cache/llmd-fs-connector	0.23	0.23	Wheel installed in llm-d
llm-d/llm-d-benchmark	v0.8.0	v0.6.8.1	Image
llm-d/llm-d-workload-variant-autoscaler	v0.9.0	v0.8.0	Helm Chart + Image
llm-d/llm-d-async	v0.9.0	v0.8.0	Helm Chart + Image
llm-d/llm-d-batch-gateway	v0.5.0	—	Image + Helm Chart
llm-d/llm-d-latency-predictor	0.9.0	—	Image
llm-d/mooncake-master-store	v0.8.0	v0.8.0	Image
vllm-project/vllm	v0.26.0	v0.23.0	Wheel installed in llm-d
kubernetes-sigs/gateway-api-inference-extension	v1.5.0	v1.5.0	Helm Chart + CRDs
kubernetes-sigs/inference-perf	v0.6.1	v0.6.0	Tool

Upstream Model Server Images

Engine	Image	Tag	Previous Tag
vLLM	docker.io/vllm/vllm-openai	v0.26.0	v0.23.0
vLLM Omni	docker.io/vllm/vllm-omni	v0.26.0	—
vLLM TPU	docker.io/vllm/vllm-tpu	v0.26.0	v0.25.0
vLLM XPU	docker.io/vllm/vllm-openai-xpu	v0.26.0	v0.23.0
vLLM ROCm	docker.io/vllm/vllm-openai-rocm	v0.26.0	v0.23.0
vLLM ROCm Omni	docker.io/vllm/vllm-omni-rocm	v0.24.1	—
vLLM CPU	docker.io/vllm/vllm-openai-cpu	v0.26.0	v0.23.0
SGLang	docker.io/lmsysorg/sglang	v0.5.16	v0.5.13.post1
TRT-LLM	nvcr.io/nvidia/tensorrt-llm/release	1.3.0rc23	—

---

Infrastructure Changes

Component	Version	Previous Version
Gateway API CRDs	v1.5.1	v1.5.1
GAIE CRDs	v1.5.0	v1.5.0
Istio	1.29.4	1.29.4
AgentGateway	v1.1.0	v2.3.3 (as kgateway)
Envoy Gateway / Envoy AI Gateway	v1.8.1 / v0.7.0	—

What's Changed

• bump wide-ep images by @Gregory-Pereira in #1937
• Add benchmark results for SGLang with tiered prefix cache by @rahulgurnani in #1921
• pin inference-perf version by @diegocastanibm in #1946
• Add Intel XPU vLLM support for multimodal optimized-baseline by @joshuayao in #1772
• docs: list founders inline instead of logos on getting-started intro by @Ibrahim2595 in #1960
• Create skeleton for sig-llm-d-inference-payload-processor by @petecheslock in #1499
• Add cleanup parameter to slash command by @diegocastanibm in #1961
• update-badge parameter by @diegocastanibm in #1951
• docs: add Deploy section to batch-gateway well-lit path by @lioraron in #1965
• Update links in kv-offloader documentation to point to the GitHub repo for deployment manifests by @petecheslock in #1962
• fix: Add missing update_badge parameters to ci workflows by @amirfr3 in #1969
• docs: fix broken well-lit-path links in README by @Ibrahim2595 in #1975
• Fix metric names for SGLang in Grafana Dashboards by @rahulgurnani in #1970
• docs(flowcontrol): map shutdown-drained requests to 503 by @LukeAVanDrie in #1801
• Add Snowflake as a user of llm-d by @robertgshaw2-redhat in #1978
• [Docs] Update PD flow to include KV lease extension by @NickLucche in #1543
• deps(actions): bump actions/checkout from 6 to 7 by @dependabot[bot] in #1976
• fix: correct the typos in the project by @AayushSaini101 in #1971
• Add sig-batch-inference to SIGS.md by @jtechapps in #1922
• proposal: Observability integration across the llm-d stack by @gyliu513 in #1685
• docs: add XPU WideEP guide backend by @yao531441 in #1925
• Add Novita as a user of llm-d by @UranusSeven in #1986
• [ROCm] Docker AMD ROCm image update by @vcave in #1977
• fix(ci): fix dry_run default value for wide-ep-lws update-badge jobs by @weizhoublue in #1955
• fix: xpu rdma deviceclass name by @poussa in #1811
• Support tiered prefix cache on Intel XPU with LMCache connector by @XinyuYe-Intel in #1731
• Update version badge to v0.8 by @chcost in #1989
• [Autoscaling]: Update autoscaling guide OWNERS file by @lionelvillard in #1990
• [Autoscaling] Unpin WVA release 0.8 by @lionelvillard in #1979
• chore: Remove superfluous nightly prefix. by @lionelvillard in #1991
• [Fix] Add new config required by actions/checkout@v7 by @maugustosilva in #1992
• [Fix] Add allow-unsafe-pr-checkout: true to all instances of actions/checkout by @maugustosilva in #1993
• docs(ipp): add Inference Payload Processor documentation by @yehuditkerido in #1884
• docs: fix broken links and script paths in flow control guide by @ishwar170695 in #1997
• Add DigitalOcean as an adopter in ADOPTERS.md by @robertgshaw2-redhat in #2004
• docs(wva): add KEDA support and shared-cluster setup by @omerap12 in #1792
• Fix return type annotation by @diegocastanibm in #2005
• docs: fix typo for vllm attribute by @zdtsw in #2000
• update: add GH nightly for optimized baseline on SGLang by @zdtsw in #1999
• feat(monitoring): ship default EPP Prometheus alerting rules by @sudoalok in #1972
• Add license scan report and status by @fossabot in #2013
• Resync nightly matrix by @diegocastanibm in #2018
• Add engine-type: sglang label to optimized-baseline kustomization by @Amit-Berman in #2012
• docs(agentic-serving): add TPU v7 disaggregated P/D benchmark results and manifests by @yangspirit in #1903
• Add new adopter of llm-d by @robertgshaw2-redhat in #2028
• Docs: add menu-config.json to control the documentation sidebar by @Ibrahim2595 in #2021
• Docs: fix broken multimodal guide link in EPP HTTP APIs reference by @Ibrahim2595 in #2031
• Prevent Incorrect Load-Aware Routing for AMD SGLang Backends by @weizhoublue in #2025
• Update README.md with two Docs links by @Ibrahim2595 in #2007
• fix(wva): migrate nightly + guide from Prometheus Adapter to KEDA by @mamy-CS in #2038
• docs(autoscaling): convert EPP autoscaling guide to KEDA by @nourey in #1981
• fix(wva): authenticate KEDA to Thanos and assert the metric path by @mamy-CS in #2039
• Initial Details for sig-agentic-inference by @petecheslock in #1868
• feat(guides): add GKE A4X (GB200) P/D disaggregation overlay by @huaxig in #2043
• refactor(a4x): update A4X kustomization without gib and patching by @huaxig in #2058
• support ipv6 setting by @eating-chen in #1974
• fix: Move the router's grafana dashboard to llm-d/llm-d by @gyliu513 in #1984
• fix: CI dry-run and optimized-baseline guide by @zdtsw in #2035
• update(CI): convert wva dry-run job to do recursive find for kustomization by @zdtsw in #2063
• predicted-latency-routing: add the P/D disaggregated deployment and its benchmark by @kaushikmitr in #2048
• refactor(gke): restructure GKE overlays to gke/base and gke/a4x by @liu-cong in #2062
• Optimized Baseline Architectural Shift & Updated Benchmarks by @kaushikmitr in #1651
• [Minor] Update machine types in GKE README by @seanhorgan in #2056
• feat(a4xmax): ...

Release v0.8.1

LLM-D v0.8.0 Component Summary

Themes: solidify CI coverage & project operations, expand accelerator coverage, graduate multimodal/batch/flow-control to production, introduce initial RL support.

Component	Version	Previous Version	Type	Notes
llm-d/llm-d-router-endpoint-picker	v0.9.0	v0.8.0	Image + Helm Chart	Core EPP image (renamed from llm-d-inference-scheduler)
llm-d/llm-d-router-disagg-sidecar	v0.9.0	v0.8.0	Image	P/D routing sidecar (renamed from llm-d-routing-sidecar)
llm-d/llm-d-uds-tokenizer	vllm-v0.23.0	vllm-v0.19.1	Image	Tokenizer sidecar aligned with vLLM version
llm-d/llm-d-kv-cache	v0.9.0	v0.8.0	Library	HMA support, storage events, multi-tier offloading
llm-d/llm-d-inference-sim	v0.9.2	v0.8.2	Image	Multimodal support, Mooncake bootstrap, configurable latency
llm-d/llm-d-cuda	v0.8.0	v0.7.0	Image	vLLM v0.23.0, CUDA 13.0.2
llm-d/llm-d-aws (EFA)	v0.8.0	v0.7.0	Image
llm-d/llm-d-hpu	v0.8.0	v0.7.0	Image
llm-d/llm-d-kv-cache/llmd_fs_backend_connector	v0.23	v0.19.1	Wheel installed in llm-d	Migrated to vLLM 0.23.0 offload API
llm-d/llm-d-benchmark	v0.7.0	v0.6.8.1	Image	Benchmark workload launcher
llm-d/llm-d-workload-variant-autoscaler	v0.8.0	v0.7.0	Helm Chart + Image	CRD migration to llm-d.ai API group, improved observability
vllm-project/vllm	v0.23.0	v0.19.1	Wheel installed in llm-d	Confirmed by @Gregory-Pereira and @tessapham
kubernetes-sigs/gateway-api-inference-extension	v1.5.0	v1.5.0	Helm Chart	Charts now published from llm-d-router OCI registry

Upstream vLLM Images (replacing llm-d-built images)

Per PR #1791, the following platforms now use upstream vLLM images directly instead of llm-d-built custom images:

Platform	New Image	Tag	Previous llm-d Image
GPU (CUDA)	vllm/vllm-openai	v0.23.0	ghcr.io/llm-d/llm-d-cuda (still available for advanced builds)
ROCm (AMD)	vllm/vllm-openai-rocm		ghcr.io/llm-d/llm-d-rocm
CPU	ghcr.io/llm-d/llm-d-cpu	v0.7.0	Same image, version bump
XPU (Intel)	vllm/vllm-openai	v0.23.0	ghcr.io/llm-d/llm-d-xpu

---

Infrastructure Changes

Component	Version	Previous Version	Notes
Gateway API	v1.5.1	v1.5.1	No change
Istio	1.29.4	1.29.1	Patch update
kgateway (agentgateway)	v2.3.3	v2.2.1

---

Deprecated / Removed Components

Component	Status	Replaced By
llm-d/llm-d-inference-scheduler	Renamed	ghcr.io/llm-d/llm-d-router-endpoint-picker
llm-d/llm-d-routing-sidecar	Renamed	ghcr.io/llm-d/llm-d-router-disagg-sidecar
llm-d/llm-d-cuda (debug)	Removed	N/A
llm-d/llm-d-cuda-gb200	Removed	N/A
llm-d/llm-d-xpu	Removed	vllm/vllm-openai
llm-d/llm-d-rocm	Removed	vllm/vllm-openai-rocm

---

New Capabilities in This Release

Capability	Component(s)	Status
Multi-modal serving (production)	llm-d-router, llm-d-inference-sim	Graduated
Batch gateway (production)	llm-d-router	Graduated
Flow control (production)	llm-d-router	Graduated
Non-Kubernetes mode (RL/Slurm)	llm-d-router (FileDiscovery plugin)	New
Responses API support	llm-d-router	New
Multi-tier KV offloading (CPU → storage)	llm-d-kv-cache	New
HMA (Heterogeneous Memory Allocation) support	llm-d-kv-cache, fs-connector	New
DP-Aware scheduling	llm-d-router	Graduating
Mooncake connector	llm-d-kv-cache	New
Predicted latency scheduling	llm-d-router	New
Agentic workload routing	llm-d-router	New
TPU nightly tests	CI/CD	New

What's Changed

• Point to patch v0.8.1 by @maugustosilva in commit d6ffb5c
• Updated the branch to clone in the guides to release-0.8 branch by @ahg-g in #1958
• Pin inference-perf version by @diegocastanibm in #1946

Full Changelog: v0.8.0...v0.8.1

Release v0.8.0

LLM-D v0.8.0 Component Summary

Themes: solidify CI coverage & project operations, expand accelerator coverage, graduate multimodal/batch/flow-control to production, introduce initial RL support.

Component	Version	Previous Version	Type	Notes
llm-d/llm-d-router-endpoint-picker	v0.9.0	v0.8.0	Image + Helm Chart	Core EPP image (renamed from llm-d-inference-scheduler)
llm-d/llm-d-router-disagg-sidecar	v0.9.0	v0.8.0	Image	P/D routing sidecar (renamed from llm-d-routing-sidecar)
llm-d/llm-d-uds-tokenizer	vllm-v0.23.0	vllm-v0.19.1	Image	Tokenizer sidecar aligned with vLLM version
llm-d/llm-d-kv-cache	v0.9.0	v0.8.0	Library	HMA support, storage events, multi-tier offloading
llm-d/llm-d-inference-sim	v0.9.2	v0.8.2	Image	Multimodal support, Mooncake bootstrap, configurable latency
llm-d/llm-d-cuda	v0.8.0	v0.7.0	Image	vLLM v0.23.0, CUDA 13.0.2
llm-d/llm-d-aws (EFA)	v0.8.0	v0.7.0	Image
llm-d/llm-d-hpu	v0.8.0	v0.7.0	Image
llm-d/llm-d-kv-cache/llmd_fs_backend_connector	v0.23	v0.19.1	Wheel installed in llm-d	Migrated to vLLM 0.23.0 offload API
llm-d/llm-d-benchmark	v0.7.0	v0.6.8.1	Image	Benchmark workload launcher
llm-d/llm-d-workload-variant-autoscaler	v0.8.0	v0.7.0	Helm Chart + Image	CRD migration to llm-d.ai API group, improved observability
vllm-project/vllm	v0.23.0	v0.19.1	Wheel installed in llm-d	Confirmed by @Gregory-Pereira and @tessapham
kubernetes-sigs/gateway-api-inference-extension	v1.5.0	v1.5.0	Helm Chart	Charts now published from llm-d-router OCI registry

Upstream vLLM Images (replacing llm-d-built images)

Per PR #1791, the following platforms now use upstream vLLM images directly instead of llm-d-built custom images:

Platform	New Image	Tag	Previous llm-d Image
GPU (CUDA)	vllm/vllm-openai	v0.23.0	ghcr.io/llm-d/llm-d-cuda (still available for advanced builds)
ROCm (AMD)	vllm/vllm-openai-rocm		ghcr.io/llm-d/llm-d-rocm
CPU	ghcr.io/llm-d/llm-d-cpu	v0.7.0	Same image, version bump
XPU (Intel)	vllm/vllm-openai	v0.23.0	ghcr.io/llm-d/llm-d-xpu

---

Infrastructure Changes

Component	Version	Previous Version	Notes
Gateway API	v1.5.1	v1.5.1	No change
Istio	1.29.4	1.29.1	Patch update
kgateway (agentgateway)	v2.3.3	v2.2.1

---

Deprecated / Removed Components

Component	Status	Replaced By
llm-d/llm-d-inference-scheduler	Renamed	ghcr.io/llm-d/llm-d-router-endpoint-picker
llm-d/llm-d-routing-sidecar	Renamed	ghcr.io/llm-d/llm-d-router-disagg-sidecar
llm-d/llm-d-cuda (debug)	Removed	N/A
llm-d/llm-d-cuda-gb200	Removed	N/A
llm-d/llm-d-xpu	Removed	vllm/vllm-openai
llm-d/llm-d-rocm	Removed	vllm/vllm-openai-rocm

---

New Capabilities in This Release

Capability	Component(s)	Status
Multi-modal serving (production)	llm-d-router, llm-d-inference-sim	Graduated
Batch gateway (production)	llm-d-router	Graduated
Flow control (production)	llm-d-router	Graduated
Non-Kubernetes mode (RL/Slurm)	llm-d-router (FileDiscovery plugin)	New
Responses API support	llm-d-router	New
Multi-tier KV offloading (CPU → storage)	llm-d-kv-cache	New
HMA (Heterogeneous Memory Allocation) support	llm-d-kv-cache, fs-connector	New
DP-Aware scheduling	llm-d-router	Graduating
Mooncake connector	llm-d-kv-cache	New
Predicted latency scheduling	llm-d-router	New
Agentic workload routing	llm-d-router	New
TPU nightly tests	CI/CD	New

What's Changed

• Simplify WVA guide test by @lionelvillard in #1072
• fix concurrency group to sha not PR by @Gregory-Pereira in #1073
• fix block-size alignment by @vMaroon in #1084
• Revise maturity status and TPU VM type details by @seanhorgan in #1085
• Fix formatting of automated test status in README by @seanhorgan in #1087
• Fix image on pd user guide by @Edwinhr716 in #1086
• Updated maturity testing level on all guides by @maugustosilva in #1094
• Skip latest tag for release candidates by @diegocastanibm in #1034
• fix(xpu): enable TP=2 for Qwen3-32B for fixing XPU prefix-cache test failed by @yuanwu2017 in #1081
• [guides] Add a commented priorityClassName for use in nightly CI/CD by @maugustosilva in #1062
• deps(actions): bump docker/build-push-action from 6 to 7 by @dependabot[bot] in #1089
• deps(actions): bump actions/github-script from 7 to 8 by @dependabot[bot] in #1091
• deps(actions): bump google-github-actions/auth from 2.1.12 to 3.0.0 by @dependabot[bot] in #1090
• deps(actions): bump dorny/paths-filter from 3 to 4 by @dependabot[bot] in #1092
• deps(actions): bump docker/login-action from 3 to 4 by @dependabot[bot] in #1093
• Add shared drive for sig-rl by @petecheslock in #1109
• Fix llm-d performance dashboard queries by @danehans in #1098
• Fix/e2e validate single curl pod by @yuanwu2017 in #1078
• Add Moreh as a contributor to the adopters list by @hhk7734 in #1111
• Update guides with status badges. by @maugustosilva in #1110
• fix: use vllmServe modelCommand in precise-prefix-cache-aware XPU values by @sharvil10 in #1101
• Add deployment health-check smoke test for llm-d clusters by @lisperz in #767
• [1/N] Documentation Revamp by @robertgshaw2-redhat in #1100
• fix: split vLLM command array and fill in quickstart TODOs by @madhugoutham in #1126
• add OCI well-lit-path for P/D disaggregation by @hexfusion in #1123
• fix: use the proper git ref for tag extraction by @Gregory-Pereira in #1133
• Update latency-predictor.md diagram by @LukeAVanDrie in #1127
• docs: fix typos detected by nightly scan (issue #1056) by @ianliuy in #1135
• [Docs][4/N] Update architecture/core/epp/scheduling.md docs by @ahg-g in #1122
• [2/N] Proxy Doc by @robertgshaw2-redhat in #1121
• [Docs][6/N]: add Istio gateway setup guide by @madhugoutham in #1146
• docs: add EPP flow control reference by @LukeAVanDrie in #1130
• [Docs] [3/N] Add Disaggregation Architecture Docs by @robertgshaw2-redhat in #1138
• [Docs] Add glossary page by @ianliuy in #1148
• Add agentgateway guide by @danehans in #1159
• fix(docs): correct typos from nightly scan, add false-positive config by @ianliuy in #1160
• [Docs] Guides Directory Cleanup by @robertgshaw2-redhat in #1163
• [Docs][EPP] Doc for request handling and control by @zetxqx in #1128
• fix(docker): add LIBRARY_PATH and ldconfig to CUDA runtime stage by @ianliuy in #1147
• Update ms-inference-scheduling/values_tpu_v7.yaml to use RunAI model streamer by @amacaskill in #1102
• deps(actions): bump hashicorp/setup-terraform from 3.1.2 to 4.0.0 by @dependabot[bot] in #1153
• deps(actions): bump aws-actions/configure-aws-credentials from 4.3.1 to 6.1.0 by @dependabot[bot] in #1152
• deps(actions): bump actions/github-script from 8 to 9 by @dependabot[bot] in #1150
• deps(actions): bump j178/prek-action from 1 to 2 by @dependabot[bot] in #1151
• deps(actions): bump actions/upload-artifact from 6 to 7 by @dependabot[bot] in #1149
• [Docs] Remove customizing-a-guide.md by @robertgshaw2-redhat in #1165
• [Docs] Move guides/benchmarks to helpers/benchmark.md by @robertgshaw2-redhat in #1164
• [Docs][Istio] Align to AgentGateway Doc by @robertgshaw2-redhat in #1161
• deps(docker): bump gdrcopy from v2.5.1 to v2.5.2 by @ianliuy in #1171
• [Docs] Envoy Proxy -> GAIE-Conformant Proxy by @robertgshaw2-redhat in #1162
• Updated the basic architecture diagram by @ahg-g in #1173
• [Docs] Fix Broken Quickstart Links by @robertgshaw2-redhat in #1175
• [Docs] Remove guide/pre...

Release v0.7.0

LLM-D Component Summary

• ⚠️ BREAKING CHANGE — CUDA 13.0.2 runtime: All llm-d CUDA images now ship with CUDA 13.0.2 (upgraded from 12.x). This requires NVIDIA driver 580 or later on the host. Nodes running older drivers must be upgraded before deploying v0.7.0 images.
• UX Change - due to the difficulty configuring gateways for many adopters, we have made the default deployment of llm-d to use "standalone mode" where we use a generic proxy instead of the more feature full gateway. We still recomend a fully gateway for customers in production.

Component	Version	Previous Version	Type
llm-d/llm-d-inference-scheduler	v0.8.0	v0.7.1	Image
llm-d/llm-d-uds-tokenizer	vllm-v0.19.1	v0.7.1	Image
llm-d/llm-d-kv-cache	v0.8.0	v0.7.1	Library
llm-d/llm-d-routing-sidecar	v0.8.0	v0.7.1	Image
llm-d/llm-d-inference-sim	v0.8.2	v0.7.1	Image
llm-d/llm-d-cuda	v0.7.0	v0.6.0	Image
llm-d/llm-d-cuda (debug)	v0.7.0	v0.6.0	Image
llm-d/llm-d-cuda-gb200	v0.7.0	N/A	Image (New)
llm-d/llm-d-aws (EFA)	v0.7.0	v0.6.0	Image
llm-d/llm-d-xpu	v0.7.0	v0.6.0	Image
llm-d/llm-d-hpu	v0.7.0	v0.6.0	Image
llm-d/llm-d-cpu	v0.7.0	v0.6.0	Image
llm-d/llm-d-rocm	v0.7.0	v0.6.0	Image
llm-d/llm-d-kv-cache/llmd_fs_backend_connector	v0.19.1	v0.17.1	Wheel installed in llm-d
llm-d/llm-d-workload-variant-autoscaler	v0.7.0	v0.6.0	Helm Chart + Image
llm-d-incubation/llm-d-infra (Deprecated)	N/A	v1.4.0	Helm Chart
llm-d-incubation/llm-d-modelservice (Deprecated)	N/A	v0.4.9	Helm Chart
vllm-project/vllm	v0.19.1	v0.17.1	Wheel installed in llm-d
kubernetes-sigs/gateway-api-inference-extension	v1.5.0	v1.4.0	Helm Chart

---

Infrastructure Changes

Component	Version	Previous Version
Gateway API	v1.5.1	v1.4.0
Istio	1.29.1	1.28.1
agentgateway (old KGateway)	v2.2.1	v2.1.1

What's Changed

• Simplify WVA guide test by @lionelvillard in #1072
• fix concurrency group to sha not PR by @Gregory-Pereira in #1073
• fix block-size alignment by @vMaroon in #1084
• Revise maturity status and TPU VM type details by @seanhorgan in #1085
• Fix formatting of automated test status in README by @seanhorgan in #1087
• Fix image on pd user guide by @Edwinhr716 in #1086
• Updated maturity testing level on all guides by @maugustosilva in #1094
• Skip latest tag for release candidates by @diegocastanibm in #1034
• fix(xpu): enable TP=2 for Qwen3-32B for fixing XPU prefix-cache test failed by @yuanwu2017 in #1081
• [guides] Add a commented priorityClassName for use in nightly CI/CD by @maugustosilva in #1062
• deps(actions): bump docker/build-push-action from 6 to 7 by @dependabot[bot] in #1089
• deps(actions): bump actions/github-script from 7 to 8 by @dependabot[bot] in #1091
• deps(actions): bump google-github-actions/auth from 2.1.12 to 3.0.0 by @dependabot[bot] in #1090
• deps(actions): bump dorny/paths-filter from 3 to 4 by @dependabot[bot] in #1092
• deps(actions): bump docker/login-action from 3 to 4 by @dependabot[bot] in #1093
• Add shared drive for sig-rl by @petecheslock in #1109
• Fix llm-d performance dashboard queries by @danehans in #1098
• Fix/e2e validate single curl pod by @yuanwu2017 in #1078
• Add Moreh as a contributor to the adopters list by @hhk7734 in #1111
• Update guides with status badges. by @maugustosilva in #1110
• fix: use vllmServe modelCommand in precise-prefix-cache-aware XPU values by @sharvil10 in #1101
• Add deployment health-check smoke test for llm-d clusters by @lisperz in #767
• [1/N] Documentation Revamp by @robertgshaw2-redhat in #1100
• fix: split vLLM command array and fill in quickstart TODOs by @madhugoutham in #1126
• add OCI well-lit-path for P/D disaggregation by @hexfusion in #1123
• fix: use the proper git ref for tag extraction by @Gregory-Pereira in #1133
• Update latency-predictor.md diagram by @LukeAVanDrie in #1127
• docs: fix typos detected by nightly scan (issue #1056) by @ianliuy in #1135
• [Docs][4/N] Update architecture/core/epp/scheduling.md docs by @ahg-g in #1122
• [2/N] Proxy Doc by @robertgshaw2-redhat in #1121
• [Docs][6/N]: add Istio gateway setup guide by @madhugoutham in #1146
• docs: add EPP flow control reference by @LukeAVanDrie in #1130
• [Docs] [3/N] Add Disaggregation Architecture Docs by @robertgshaw2-redhat in #1138
• [Docs] Add glossary page by @ianliuy in #1148
• Add agentgateway guide by @danehans in #1159
• fix(docs): correct typos from nightly scan, add false-positive config by @ianliuy in #1160
• [Docs] Guides Directory Cleanup by @robertgshaw2-redhat in #1163
• [Docs][EPP] Doc for request handling and control by @zetxqx in #1128
• fix(docker): add LIBRARY_PATH and ldconfig to CUDA runtime stage by @ianliuy in #1147
• Update ms-inference-scheduling/values_tpu_v7.yaml to use RunAI model streamer by @amacaskill in #1102
• deps(actions): bump hashicorp/setup-terraform from 3.1.2 to 4.0.0 by @dependabot[bot] in #1153
• deps(actions): bump aws-actions/configure-aws-credentials from 4.3.1 to 6.1.0 by @dependabot[bot] in #1152
• deps(actions): bump actions/github-script from 8 to 9 by @dependabot[bot] in #1150
• deps(actions): bump j178/prek-action from 1 to 2 by @dependabot[bot] in #1151
• deps(actions): bump actions/upload-artifact from 6 to 7 by @dependabot[bot] in #1149
• [Docs] Remove customizing-a-guide.md by @robertgshaw2-redhat in #1165
• [Docs] Move guides/benchmarks to helpers/benchmark.md by @robertgshaw2-redhat in #1164
• [Docs][Istio] Align to AgentGateway Doc by @robertgshaw2-redhat in #1161
• deps(docker): bump gdrcopy from v2.5.1 to v2.5.2 by @ianliuy in #1171
• [Docs] Envoy Proxy -> GAIE-Conformant Proxy by @robertgshaw2-redhat in #1162
• Updated the basic architecture diagram by @ahg-g in #1173
• [Docs] Fix Broken Quickstart Links by @robertgshaw2-redhat in #1175
• [Docs] Remove guide/prereq/infrastructure by @robertgshaw2-redhat in #1168
• Added GKE gateway guide by @ahg-g in #1174
• [Docs] Move Client Tools from Guides --> Helpers by @robertgshaw2-redhat in #1167
• feat: add Rebellions as a supported accelerator vendor by @rebel-jinmoo in #1115
• docs: rewrite predicted latency architecture and add well-lit path by @kaushikmitr in #1166
• [Docs][Autoscaling][2/N] hpa/keda vs hpa design choices/features by @lionelvillard in #1157
• [Docs][7/N] KV Indexer by @vMaroon in #1143
• [Docs][autoscaling][1/N] Autoscaling intro by @lionelvillard in #1145
• [Docs] Refine RDMA docs by @praveingk in #1181
• Fixup merge conflict markers in tiered-prefix-cache/storage/README.md by @tlrmchlsmth in #1190
• [Docs] fix broken link by @lionelvillard in #1191
• docs: llm-d-inference-payload-processor proposal by @nilig in #1184
• [Docs] Well Lit Paths Doc by @chcost in #1156
• Tweaks to the latency predictor docs by @ahg-g in #1201
• [Docs] Restructure Predicted Latency Well Lit Path Doc by @robertgshaw2-redhat in #1200
• [Docs][Autoscaling][4/N] first pass at the HPA+IGW doc by @lionelvillard in #1192
• Updating native HPA-based autoscaling guide to reference EPP instead of IGW by @ahg-g in #1212
• [Docs] Rename Well-Lit-Paths -> Guides, Guides -> Resources by @robertgshaw2-redhat in https://github.com/llm-d/llm-d/pul...

Release v0.6.0

LLM-D Component Summary

Component	Version	Previous Version	Type
llm-d/llm-d-inference-scheduler	v0.7.1	v0.6.0	Image
llm-d/llm-d-uds-tokenizer	v0.7.1	v0.6.0	Image
llm-d/llm-d-kv-cache	v0.7.1	v0.6.0	Library
llm-d/llm-d-routing-sidecar	v0.7.1	v0.6.0	Image
llm-d/llm-d-inference-sim	v0.8.2	v0.7.1	Image
llm-d/llm-d-cuda	v0.6.0	v0.5.1	Image
llm-d/llm-d-cuda (debug)	v0.6.0	v0.5.1	Image
llm-d/llm-d-aws (EFA)	v0.6.0	v0.5.1	Image
llm-d/llm-d-xpu	v0.6.0	v0.5.1	Image (Temporarily Unavailable)
llm-d/llm-d-hpu	v0.6.0	v0.5.1	Image (Temporarily Unavailable)
llm-d/llm-d-cpu	v0.6.0	v0.5.1	Image
llm-d/llm-d-rocm	v0.6.0	v0.5.1	Image
llm-d/llm-d-kv-cache/llmd_fs_backend_connector	v0.17.1	v0.15.1	Wheel installed in llm-d
llm-d/llm-d-workload-variant-autoscaler	v0.6.0	v0.5.1	Helm Chart + Image
llm-d-incubation/llm-d-infra	v1.4.0	v1.3.6	Helm Chart
llm-d-incubation/llm-d-modelservice	v0.4.9	v0.4.7	Helm Chart
vllm-project/vllm	v0.17.1	v0.15.1	Wheel installed in llm-d
kubernetes-sigs/gateway-api-inference-extension	v1.4.0	v1.3.1	Helm Chart

---

Infrastructure Changes

Component	Version	Previous Version
Gateway API	v1.5.1	v1.4.0
Istio	1.29.1	1.28.1
agentgateway (old KGateway)	v2.2.1	v2.1.1

---

What's Changed

• Add SGLang option for inference-scheduling well-lit path by @andreyod in #527
• add step to build hpu by @diegocastanibm in #917
• 🌱 Remove per-repo gh-aw typo/link/upstream workflows by @clubanderson in #920
• docs: Small fixes in the inference-scheduling installation guide by @roytman in #924
• Fix link to vLLM Native CPU Offloading documentation by @petecheslock in #928
• docs: Remove unnecessary backticks around llm-d by @terrytangyuan in #933
• Partial enablement of CICD for GKE by @maugustosilva in #934
• Add hpu to ci-release by @diegocastanibm in #918
• Only report on 404 broken links and not temp issues or scraper blocks by @petecheslock in #846
• hpu: update images by @poussa in #936
• fix: correct typos in documentation and code comments by @BaskDuan in #973
• Fix the HPU failed issue and enable HPU PR checks in CI by @yuanwu2017 in #965
• chore: update xpu image build inputs by @VincyZhang in #911
• Add HPA + IGW metrics autoscaling guide to the autoscaling well-lit path by @aishukamal in #972
• deps(actions): bump actions/checkout from 4 to 6 by @dependabot[bot] in #836
• Build llm-d-cuda locally by @diegocastanibm in #865
• build: add CUDA versions into central build file by @zdtsw in #982
• Mark myself as inactive project maintainer while on leave by @smarterclayton in #985
• update the result for inference-scheduling and precise-prefix-cache-aware by @zetxqx in #722
• install storage offloading connector by @Gregory-Pereira in #640
• try set and unset cuda stubs in LIBRARY_PATH by @Gregory-Pereira in #987
• upgrading to vLLM v0.17.1 by @Gregory-Pereira in #977
• Refresh support for kgateway agentgateway GIE and GW API by @danehans in #421
• Update gke inference scheduling test to address failures by @rlakhtakia in #649
• deps(actions): bump google-github-actions/setup-gcloud from 2.2.0 to 3.0.1 by @dependabot[bot] in #835
• updating common configurations with infra refactor 272 by @Gregory-Pereira in #997
• ci test helm chart schemas + helmfile examples + dry run k8s apply by @Gregory-Pereira in #998
• any build triggered from push to main should be latest by @Gregory-Pereira in #996
• fix: use VLLM_PRECOMPILED_WHEEL_COMMIT from vllm 0.17.1 by @zdtsw in #1007
• enable building from open PRs by @Gregory-Pereira in #1011
• build: reduce cuda runtime image size by switch base image by @zdtsw in #606
• github: pin to a version for trivy than mastser by @zdtsw in #1018
• ci: use pull_request for upstream PRs, pull_request_target for forks by @Gregory-Pereira in #1023
• removing sm70 and 75 because lack of cuda 12.9.X support rdma-tools by @Gregory-Pereira in #1024
• docs: llm-d planner proposal by @anfredette in #963
• Image verification workflow by @diegocastanibm in #1010
• deps(actions): bump actions/setup-go from 5 to 6 by @dependabot[bot] in #1016
• deps(actions): bump github/codeql-action from 3 to 4 by @dependabot[bot] in #1015
• deps(actions): bump lycheeverse/lychee-action from 2.7.0 to 2.8.0 by @dependabot[bot] in #930
• Add benchmarking numbers for Lustre with LMcache connector by @Sneha-at in #1020
• Add P/D Disaggregation with RoCE/GDR for OpenShift by @maugustosilva in #1025
• Move benchmark templates into the specific guide directories by @deanlorenz in #784
• build: add infiniband-diags to rdma-tools runtime image by @rajkiranjoshi in #1040
• fix: typos from nightly scan by @tessapham in #1037
• Attempted fix for CI/CD GKE by explicitly exporting LD_LIBRARY_PATH by @maugustosilva in #1032
• Change HELMFILE_ENV on P/D Disaggregation (OpenShift) to ocp by @maugustosilva in #1041
• Adding Topology Aware Scheduling labels on Wide-EP GKE example by @Edwinhr716 in #531
• Fix llm-d.ai/guide for pods on PD guide by @maugustosilva in #1047
• Async Processor well-lit path guide by @shimib in #1031
• And llm_d_ref defaults to main by @diegocastanibm in #1049
• docs: expand suspended hyphens in Inference Scheduler description by @skmahe1077 in #1050
• deps(actions): bump docker/setup-buildx-action from 3 to 4 by @dependabot[bot] in #1044
• deps(actions): bump actions/github-script from 7 to 8 by @dependabot[bot] in #1043
• deps(actions): bump docker/metadata-action from 5 to 6 by @dependabot[bot] in #1042
• llm-d skills proposal by @rachelt44 in #1039
• Additional fix for gke ld library path by @maugustosilva in #1052
• 🐛 Fix kubectl attach race condition in e2e-validate.sh by @clubanderson in #1051
• Fix pd on gke by @maugustosilva in #1053
• ✨ Add /test-nightly slash command for fork PRs by @clubanderson in #1048
• Bump simulator version to v0.8.2 by @mayabar in #1045
• build: add libibverbs-utils to rdma-tools runtime image by @rajkiranjoshi in #1055
• fix: add missing sizeLimit and fix label typo in inference-scheduling configs by @wenhug in #1038
• docs: replace Envoy proxy references with Gateway API by @tessapham in #1036
• docs: update KV events topic to ip:port format for scheduler v0.7.0 by @bongwoobak in #1008
• deps(actions): bump haya14busa/action-update-semver from 1.3.0 to 1.5.1 by @dependabot[bot] in #754
• deps(actions): bump actions/setup-python from 5 to 6 by @dependabot[bot] in #753
• docs: assign a level of maturity for each guide by @maugustosilva in #1058
• update versions for v0.6.0 release by @diegocastanibm in #1064
• Add SGLang option to PD disaggregation well-lit path by @andreyod in #580
• docs: bump inference-scheduler image to v0.7.1 by @bongwoobak in #1069
• ✨ Add glob patterns and run URL feedback to /test-nightly by @clubanderson in #1065
• ✨ Replace blocking lychee CI with gh-aw nightly checkers by @clubanderson in #1066
• Images Preparation for v0.6.0 by @diegocastanibm in #1067
• Last version bump before cutting v0.6 release by @maugustosilva in htt...

Release v0.5.1

LLM-D Component Summary

Component	Version	Previous Version	Type
llm-d/llm-d-inference-scheduler	v0.6.0	v0.5.0	Image
llm-d/llm-d-kv-cache	v0.6.0	v0.5.0	Library
llm-d-incubation/llm-d-modelservice	v0.4.7	v0.4.5	Helm Chart
llm-d/llm-d-routing-sidecar	v0.6.0	v0.5.0	Image
llm-d/llm-d-inference-sim	v0.7.1	v0.7.1	Image
llm-d/llm-d-cuda	v0.5.1	v0.5.0	Image
llm-d/llm-d-cuda (debug)	v0.5.1	NA	Image (New)
llm-d/llm-d-aws (EFA)	v0.5.1	Deprecated in v0.5.0	Image (Re-enabled)
llm-d/llm-d-xpu	v0.5.1	v0.5.0	Image
llm-d/llm-d-cpu	v0.5.1	v0.5.0	Image
llm-d/llm-d-rocm	v0.5.1	NA	Image (New)
llm-d/llm-d-hpu	v0.5.1	NA	Image (New)
vllm-project/vllm	v0.15.1	v0.14.1	Wheel installed in llm-d
llm-d-incubation/llm-d-infra	v1.3.6	v1.3.6	Helm Chart
kubernetes-sigs/gateway-api-inference-extension	v1.3.1	v1.3.0	Helm Chart (Pending upstream release)
llm-d/llm-d-workload-variant-autoscaler	v0.5.1	v0.5.0	Helm Chart + Image

---

Infrastructure Changes

Component	Version	Previous Version
Gateway API	v1.3.1	v1.3.0
Istio	1.28.1	1.28.1
KGateway	v2.1.1	v2.1.1

---

What's Changed

• Fix XPU example errors by @yuanwu2017 in #696
• Simplify storage offloading guides into a single one by @liu-cong in #683
• feat: remove standalone by @capri-xiyue in #688
• update permissions for lustre guide setup by @Sneha-at in #710
• Fix: add retries for model discovery by @yuanwu2017 in #704
• [BUGFIX] Comment caching to avoid error by @diegocastanibm in #719
• Fix lint issues by @diegocastanibm in #721
• llm-d install doc on openshift 4.20 by @fgharo in #538
• Enables AMD inference scheduling well-lit path by @vcave in #642
• [build fix] Remove linking compat dir by @Gregory-Pereira in #727
• just deprecate queue scorrer params by @Gregory-Pereira in #730
• Add Gaudi inference scheduling CI workflow test by @yuanwu2017 in #486
• fix gke monitoring in helmfile by @zetxqx in #697
• add intel HPU for llm-d P/D disaggregation by @ZhengHongming888 in #372
• Add the XPU CI test by @yuanwu2017 in #380
• feat: add feature request issue template by @thillai-c in #733
• Add the format hint for CI by @yuanwu2017 in #717
• feat: add support to build for CPU on AMX by @zdtsw in #725
• fix: add missing check-buildah target in Makefile by @GuilinDev in #735
• docs: fix typos and spelling errors across docs, guides, and Dockerfile by @thillai-c in #731
• docs: update container image link in PROJECT.md by @petecheslock in #738
• Add debug variant support for llm-d images across CUDA by @kapiljain1989 in #734
• 🌱 Add typos config and Dependabot by @clubanderson in #746
• chore: update bug report issue template with current versions by @thillai-c in #732
• ✨ Add nightly E2E OpenShift workflows for 5 guides by @clubanderson in #747
• 🌱 Standardize governance workflows via llm-d-infra by @clubanderson in #744
• 🌱 Remove dependabot configuration by @clubanderson in #756
• linting links by @Gregory-Pereira in #757
• 🐛 Fix broken reusable workflow references by @clubanderson in #762
• 🐛 Bump nightly pod_wait_timeout from 15m to 30m by @clubanderson in #760
• 🐛 Fix PD disaggregation nightly decode tensor parallelism by @clubanderson in #764
• 🐛 Fix tiered-prefix-cache CrashLoopBackOff: num_cpu_blocks → cpu_bytes_to_use by @clubanderson in #768
• ⚠️ Revert: Fix tiered-prefix-cache CrashLoopBackOff (#768) by @clubanderson in #769
• 🐛 Fix tiered-prefix-cache CrashLoopBackOff: num_cpu_blocks → cpu_bytes_to_use by @clubanderson in #770
• adding required packages for LMCache runtime by @Gregory-Pereira in #701
• ✨ Add upstream dependency monitor, dependabot, and remove legacy CI callers by @clubanderson in #771
• Set accelerator_type to H100 and rename OCP nightly workflows by @diegocastanibm in #777
• ✨ Add CKS nightly E2E workflows for IS and PD by @clubanderson in #780
• ressurect efa by @Gregory-Pereira in #741
• ✨ Add wide-ep and benchmark CKS nightly callers by @clubanderson in #782
• deps(actions): bump docker/build-push-action from 5 to 6 by @dependabot[bot] in #755
• deps(actions): bump actions/upload-artifact from 4 to 6 by @dependabot[bot] in #752
• ✨ Fix dev image latest tag + nightly build + image_override for all E2E nightlies by @clubanderson in #783
• deps(actions): bump actions/checkout from 4 to 6 by @dependabot[bot] in #751
• Update interactive-pod to use latest guidellm by @natoscott in #716
• 🌱 OCP H100 nodeSelector + CKS GPU preemption for nightly E2E by @clubanderson in #788
• 🐛 Fix typo in CKS benchmark caller workflow dispatch by @clubanderson in #790
• 🐛 Build WVA image from main + prevent fork schedule runs by @clubanderson in #794
• 🐛 Fix CKS WVA test target: use test-e2e-openshift (no kind dependency) by @clubanderson in #797
• 🐛 Build WVA controller image from main for CKS nightly by @clubanderson in #798
• 🐛 Add docker build retry for WVA nightly E2E workflows by @clubanderson in #801
• ✨ Enable GPU preemption on all nightly E2E workflows by @clubanderson in #810
• bump to nixl 0.10.0 and vllm v0.15.1 by @Gregory-Pereira in #713
• Replace RHEL packages with static CentOS Stream 9 RPMs by @dagrayvid in #669
• sa was missing, copy it in by @Gregory-Pereira in #813
• ocp nightly wide-ep sa was missing, copy it in by @Gregory-Pereira in #814
• Fix typos tracked in _typos.toml by @lisperz in #786
• feat: add support to use kv-cache UDS tokenizer sidecar in guide by @zdtsw in #740
• Add RDMA tools container image for RoCE/InfiniBand validation by @dagrayvid in #626
• docs: update README for v0.5 release by @chcost in #822
• Proposal: Prism - Performance analysis for distributed inference systems by @seanhorgan in #796
• 🐛 Fix uds-tokenizer image tag in PPC guide (v0.5.0-rc1 → v0.5.1-rc1) by @clubanderson in #829
• Add gke rilb gateway class by @liu-cong in #815
• Release doc by @diegocastanibm in #819
• Add gateway provider prereq to the tiered prefix cache guides by @liu-cong in #761
• [BUG] sourcing docker/vllm-version in ci-release by @diegocastanibm in #823
• ⚠️ Skip gateway provider install on OCP nightly E2Es by @clubanderson in #833
• Add a MAINTAINERS file for current and future maintainers list by @petecheslock in #817
• update: bump llm-d-modelservice chart to 0.4.6 by @zdtsw in #644
• Add sig-rl structure proposal by @smarterclayton in #840
• Update benchmark templates to 0.5 by @dmitripikus in #816
• Revert "Update benchmark templates to 0.5" by @Gregory-Pereira in #841
• Add ADOPTERS.md to document project contributors and supporters by @petecheslock in #830
• [ADOPTERS.md] Add Tesla as a user of llm-d by @robertgshaw2-redhat in #845
• Change the model name and workflow name by @yuanwu2017 in #742
• 🐛 fix: increase startup probe and pod wait timeouts for IS nightly E2E by @clubanderson in #852
• [Benchmark Guide] [WVA + IS] Document and Provide a WVA + IS Benchmark Guide by @Vezio in #820
• Update benchmark templates to 0.5 (#816) by ...

Release v0.4.0

📦 llm-d v0.4.0 Release Notes

This release of the llm-d repo will capture the release for the entirety of the project, guides, components, and all.

Release Date: 2025-11-26

---

🧩 Component Summary

Component	Version	Previous Version	Type
llmd/llm-d-inference-scheduler	v0.4.0-rc.1	v0.3.1	Image
llm-d-incubation/llm-d-modelservice	v0.3.8	v0.2.10	Helm Chart
llm-d/llm-d-routing-sidecar	v0.4.0-rc.1	v0.3.1	Image
llm-d/llm-d-cuda	v0.4.0	v0.3.1	Image
llm-d/llm-d-aws	v0.4.0	v0.3.1	Image
llm-d/llm-d-xpu	v0.4.0	v0.3.1	Image
llm-d/llm-d-cpu	v0.4.0	v0.3.1	Image (New)
llm-d-incubation/llm-d-infra	v1.3.4	v1.3.3	Helm Chart
kubernetes-sig/gateway-api-inference-extension	v1.2.0-rc.1	v1.0.1	Helm Chart
llm-d/llm-d-workload-variant-autoscaler	v0.0.8	NA (new)	Helm Chart + Image

---

🔹 lmd/llm-d-inference-scheduler

• Description: This scheduler that makes optimized routing decisions for inference requests to the llm-d inference framework.
• Diff: v0.3.1 → v0.4.0-rc.1

---

🔹 llm-d-incubation/llm-d-modelservice

• Description: modelservice is a Helm chart that simplifies LLM deployment on llm-d by declaratively managing Kubernetes resources for serving base models. It enables reproducible, scalable, and tunable model deployments through modular presets, and clean integration with llm-d ecosystem components (including vLLM, Gateway API Inference Extension, LeaderWorkerSet).
• Diff: v0.2.10 → v0.3.8

---

🔹 llm-d/llm-d-routing-sidecar

• Description: A reverse proxy redirecting incoming requests to the prefill worker specified in the x-prefiller-host-port HTTP request header.
• Diff: v0.3.1 → v0.4.0-rc.1

---

🔹 llm-d/llm-d

• Description: A midstreamed image of vllm-project/vllm for inferencing, supporting features such as PD disaggregation, KV cache awareness and more.
• Diff: v0.3.1 → v0.4.0
• Image Variants: Different image variants of this component:
  • XPU: ghcr.io/llm-d/llm-d-xpu:v0.4.0
  • AWS: ghcr.io/llm-d/llm-d-aws:v0.4.0
  • CUDA: ghcr.io/llm-d/llm-d-cuda:v0.4.0
  • CPU: ghcr.io/llm-d/llm-d-cpu:v0.4.0

---

🔹 llm-d-incubation/llm-d-infra

• Description: A helm chart for deploying gateway and gateway related infrastructure assets for llm-d.
• Diff: v1.3.3 → v1.3.4

---

🔹 kubernetes-sig/gateway-api-inference-extension

• Description: A Helm chart to deploy an InferencePool, a corresponding EndpointPicker (epp) deployment, and any other related assets.
• Diff: v1.0.1 → v1.2.0-rc.1

---

🔹 llm-d/llm-d-workload-variant-autoscaler (New - Experimental)

• Description: Variant optimization autoscaler for distributed inference workloads
• History (new): v0.0.5
• Note: This is an experimental component being included in this release for early testing and feedback.

---

For more information on any of the component project or versions, please checkout their repos directly. For information on installing and using the new release refer to our guides. Thank you to all contributors who helped make this happen. Automated release notes will be included below, but it should be noted this only tracks work in the main repo, and does not fully reflect a changelog across the project

What's Changed

• Add umbrella kv cache offloading well-lit path folder structure by @liu-cong in #401
• Correct wide-ep resource requirements. by @liu-cong in #373
• add information about component testing by @Gregory-Pereira in #361
• doc(guides): Introduce standardized recipes for Gateway, InferencePool, and vLLM by @zetxqx in #444
• Fix a broken link in the cpu prefix cache readme by @smarterclayton in #451
• Add more GKE specific workarounds and known issues by @smarterclayton in #419
• Update SIGs documentation to remove outdated schedule details. by @petecheslock in #431
• Update links to deploying vLLM multi-host in stable docs by @smarterclayton in #436
• fix kutomization error and model flag error in cpu offloading. by @zetxqx in #453
• Add GKE B200 readme notes by @smarterclayton in #454
• doc: enrich the prefix-cache-storage vllm cpu native offloading with benchmark results by @zetxqx in #438
• Add CPU for llm-d Inference Scheduling by @ZhengHongming888 in #428
• Add cpu offloading example for GKE + LMCache by @dannawang0221 in #318
• Add tab format for better UX on the website by @liu-cong in #452
• Rename prefix-cache-storage to tiered-prefix-cache by @vMaroon in #468
• Remove the dockerfile.gke as it is no longer used by @smarterclayton in #462
• Token credentials fix + vLLM v0.11.1 by @Gregory-Pereira in #456
• guides: Make vLLM log more useful in inference-scheduling by @russellb in #439
• Inference scheduling support for Intel Gaudi accelerator by @poussa in #374
• Add JIT directories and model directories by @smarterclayton in #418
• Use markdown comments for Tabs support on docusaurus by @petecheslock in #474
• Highlight P/D benefits with throughput-interactivity tradeoff by @liu-cong in #472
• add benchmark results lmcache results and tuned epp scorers by @zetxqx in #457
• Add step by step guide for setting up p/d with TPU on GKE by @yangligt2 in #443
• refactor: restructure vllm recipe with base and overlay pattern by @diego-torres in #475
• [Build] Add FI JIT Cache to Image by @robertgshaw2-redhat in #482
• Add instructions to clone git repo and checkout the release by @liu-cong in #477
• Create CPU dockefile for PD and Inference Scheduling by @ZhengHongming888 in #465
• guides/prereq/client-setup/install-deps.sh - increment HELMFILE_VERSION to 1.2.1 by @herbertkb in #492
• docs: Addresses CPU support added in PR #428 by @aneeshkp in #466
• Infra, MS and GAIE bumps + istio change compat by @Gregory-Pereira in #459
• Update release version for cpu offloading guide by @liu-cong in #495
• enable TLS in monitoring for prom by @Gregory-Pereira in #496
• helmfile and supporting artifacts for wva by @clubanderson in #464
• updating LMCACHe to be non fork by @Gregory-Pereira in #501
• component bumps for WVA guide by @Gregory-Pereira in #502
• Build vLLM 0.11.2 + patches for 0.4 by @smarterclayton in #461
• Avoid defining LMCACHE_COMMIT_SHA in multiple places by @terrytangyuan in #503
• WVA guide integration targeting v0.4 by @mamy-CS in #470
• fixing AWS image by @Gregory-Pereira in #506
• remove pre-passing values for VLLM by @Gregory-Pereira in #507

New Contributors

• @zetxqx made their first contribution in #444
• @ZhengHongming888 made their first contribution in #428
• @dannawang0221 made their first contribution in #318
• @russellb made their first contribution in #439
• @poussa made their first contribution in #374
• @yangligt2 made their first contribution in #443
• @diego-torres made their first contribution in #475
• @herbertkb made their first contribution in #492
• @aneeshkp made their first contribution in #466
• @mamy-CS made their first contribution in #470

Full Changelog: v0.3.1...v0.4.0

Release v0.5.0

📦 llm-d v0.5.0 Release Notes

This release of the llm-d repo will capture the release for the entirety of the project, guides, and components.

Release Date: 2026-02-03

---

Core Objectives

• Reproducible benchmarking
• Scaling KV-cache
• Autoscaling improvements
• Greater Metrics + tracing story

🏗️ Infrastructure Changes - BREAKING CHANGES

Component	Version	Previous Version
Gateway API	v1.4.0	v1.3.0
Istio	1.28.1	1.28-alpha.89f30b26ba71bf5e538083a4720d0bc2d8c06401
KGateway	v2.1.1	v2.0.3
GKE Gateways	NA - tied to GKE	NA - tied to GKE

NOTE: We upgraded to the following versions to consume the new v1 Inference Pool API. This means cluster admins should descale their workloads and upgrade these Infrastructure level components before proceeding. You should be able to use the old inference.networking.x-k8s.io/v1alpha2 API, however the guides will require changes to make them work - use this at your own risk.

🧩 LLM-D Component Summary

Component	Version	Previous Version	Type
llm-d/llm-d-inference-scheduler	v0.5.0	v0.4.0-rc1	Image
llm-d/llm-d-kv-cache	v0.5.0	v0.4.0	Library
llm-d-incubation/llm-d-modelservice	v0.4.5	v0.3.8	Helm Chart
llm-d/llm-d-routing-sidecar	v0.5.0	v0.4.0-rc1	Image
llm-d/llm-d-inference-sim	v0.7.1	v0.6.1	Image
llm-d/llm-d-cuda	v0.5.0	v0.4.0	Image
llm-d/llm-d-aws	Deprecated	v0.4.0	Image
llm-d/llm-d-xpu	v0.5.0	v0.4.0	Image
llm-d/llm-d-cpu	v0.5.0	v0.4.0	Image (New)
vllm-project/vllm	v0.14.1	v0.11.2 + additional cherry-picks (built from fork)	Wheel installed in llm-d
llm-d-incubation/llm-d-infra	v1.3.6	v1.3.4	Helm Chart
kubernetes-sig/gateway-api-inference-extension	v1.3.0	v1.2.0-rc1	Helm Chart
llm-d/llm-d-workload-variant-autoscaler	v0.5.0	v0.0.8	Helm Chart + Image

Deprecations and changes

Temporary Deprecation: EFA

We ran into a bug 2 days before release, between EFA and and our regular path. This was that the base image install version of libiverbs / rdma core is different than the version being packaged by EFA. Having both these versions causes nvshmem initialization errors for anything over RDMA. As a temporary measure we have dissabled EFA while we figure out a proper way to deine who owns the core RDMA core user space packages.

WVA promotion:

Our Workload-Variant-Autoscaler has graduated from a experimental to core component of llm-d! Congratulations to the SIG team!

Migrations

The llm-d/llm-d-routing-sidecar image has been moved under the llm-d/llm-d-inference-scheduler repo, and its previous one archieved.

🧩 CUDA Image Specific Component Summary

Component	Version	Previous Version
LMCache	v0.3.13	v0.3.8
UCX	v1.20.0	v1.19.0
NVSHMEM	v3.4.5-0 (git)	v3.3.20 (developer source distribution)
NIXL	v0.9.0	v0.6.0
GDR Copy	v2.5.1	Commit 0f7366e (maps to v2.5.1, no change)
Infinistore	v0.2.33	NA (New)
EFA installer (Temporary deprecation)	v1.46.0	v1.43.3
Flashinfer	v0.5.3	v0.5.2
DEEPGEMM	v0.5.3	v0.5.2
DEEPEP	v0.5.3	v0.5.2
PPLX	v0.5.3	v0.5.2

RC Images in v0.4.0 release 🤦🏼

Due to some unfortunate circumstances last release, we shipped the official release with many code compelete RC images. We plan to have better release hygeine going forward.

Meta changes

• Change the default NIXL port to 5600 to be consistent with vLLM v0.11.1 +
• Adopt the new v1 inferencepool API
• Remove routing sidecar from non-pd deployments as its not necesary
  • in those cases adjust decode port 8200 back to default 8000 port as it no longer needs to be proxied
• More accurate labels for model and accelerator type on inference servers
• And many more! Please see each component's diff / changelog for a more complete understanding of the changes across the project.

---

🔹 llm-d/llm-d-inference-scheduler

• Description: The scheduler that makes optimized routing decisions for inference requests to the llm-d inference framework.
• Diff: v0.4.0-rc1 → v0.5.0

---

🔹 llm-d/llm-d-kv-cache

• Description: The libraries for tokenization, KV-events processing, and KV-cache indexing and offloading.
• Diff: v0.4.0 → v0.5.0

---

🔹 llm-d-incubation/llm-d-modelservice

• Description: modelservice is a Helm chart that simplifies LLM deployment on llm-d by declaratively managing Kubernetes resources for serving base models. It enables reproducible, scalable, and tunable model deployments through modular presets, and clean integration with llm-d ecosystem components (including vLLM, Gateway API Inference Extension, LeaderWorkerSet).
• Diff: v0.3.8 → v0.4.5

---

🔹 llm-d/llm-d-routing-sidecar

• Description: A reverse proxy redirecting incoming requests to the prefill worker specified in the x-prefiller-host-port HTTP request header.
• Diff: [v0.4.0-rc.1 → v0.5.0]
  • Note, no compare link can be provided because this component was moved under the inference-scheduler repo.

---

🔹 llm-d/llm-d-inference-sim

• Description: A light-weight inference simulator.
• Diff: v0.6.1 → v0.7.1

---

🔹 llm-d/llm-d

Note: in the v0.4.0 release the guides were not as updated as we would have liked. Because of this we built a v0.4.0 image but we used the v0.3.1 image in the guides.

• Description: A midstreamed image of vllm-project/vllm for inferencing, supporting features such as PD disaggregation, KV cache awareness and more.
• Diff: v0.4.0 → v0.5.0
• Image Variants: Different image variants of this component:
  • XPU: ghcr.io/llm-d/llm-d-xpu:v0.5.0
  • AWS: Deprecated
  • CUDA: ghcr.io/llm-d/llm-d-cuda:v0.5.0
  • CPU: ghcr.io/llm-d/llm-d-cpu:v0.5.0

---

🔹 vllm-project/vllm

Note: last release we built our inferencing images off of the nerual magic fork of vLLM. This was built off of vLLM v0.11.2 with additional cherry-picks for feature work that did not make it in on time.

As vLLM has an extreemly high contribution velocity, rather than listing a Diff it makes more sense to refer to the release notes for the releases that happened in between directly:

• Release v0.12.0
• Release v0.13.0
• Release v0.14.0
• Release v0.14.1

---

🔹 llm-d-incubation/llm-d-infra

• Description: A helm chart for deploying gateway and gateway related infrastructure assets for llm-d.
• Diff: v1.3.4 → v1.3.6

---

🔹 kubernetes-sig/gateway-api-inference-extension

• Description: A Helm chart to deploy an InferencePool, a corresponding EndpointPicker (epp) deployment, and any other related assets.
• Diff: v1.2.0-rc1 → v1.3.0

---

🔹 llm-d/llm-d-workload-variant-autoscaler (New - Experimental)

• Description: [TODO: Add description of the workload variant autoscaler]
• Diff: v0.0.8 → v0.5.0
• NOTE: This component was previously experimental, but it has graduated to a core component of LLM-D. We hoped to move it from the llm-d-incubation org to the llm-d main org for this release but did not have time - stay tuned for this in the next release.

---

For more information on any of the component project or versions, please checkout their repos directly. For information on installing and using the new release refer to our guides. Thank you to all contributors who helped make this happen.

In repo change-log - inference image and guides only

• docs: added standalone EPP without gateway api guide by @capri-xiyue in #508
• refactor wideep to use recipes folder. by @zetxqx in #455
• Fix dead link by @yuanwu2017 in #510
• Add tiered prefix cache to the top level guide by @liu-cong in #516
• Update the correct vllm kv cache utilization metric name in inference scheduler by @liu-cong in #509
• fix: EPP configuration issues in precise-prefix-cache-aware guide by @yuanwu2017 in #512
• Add tiered prefix cache to GKE doc by @liu-cong in #523
• Update TPU and XPU supported well-lit paths by @liu-cong in #524
• Link to the 0.4 release blog by @smarterclayton in #525
• Update wide-ep example to 0.4.0 image ...

v0.3.1 Release

Release overview

This release was focused on following up on our objectives from the v0.3.0 that could not make it into that release. A few key stories to highlight:

• ARM support
• Refactor image build process to scripts
• Unifying the GKE image into our core Cuda image
• Adding AKS cloud provider support

Welcome to all our new contributors, and thanks to the team for their hard work.

Component version bumps:

• Inference SIM (v0.5.1 --> v0.6.1)
• llm-d image (v0.3.0 --> v0.3.1, diff encapsulated in change-log below)

What's Changed

• [bugfix] changing filename to not reference old plugin by @Gregory-Pereira in #353
• Deprecate all InferenceModel in XPU guides by @yankay in #358
• version bump on vllm release v0.11.0 tag move by @Gregory-Pereira in #365
• Update SIGS.md owners by @petecheslock in #363
• Clear /dev/shm before process startup to prevent crashloops by @smarterclayton in #364
• Add hardware and platform support issue template by @Ayobami-00 in #359
• minor typo in the inference guide by @effi-ofer in #377
• docs: introduce AKS as a well-lit infra provider by @chewong in #335
• Correct # of measured output tokens / s by @smarterclayton in #350
• refactor dockerfiles to a set bash scripts by @wseaton in #324
• Add wide ep gke test by @rlakhtakia in #367
• Intel pd workflow + v0.3 lagging updates by @Gregory-Pereira in #310
• Use the vLLM image for xpu by @yuanwu2017 in #357
• Fix markdown-link-checker failed issue by @yuanwu2017 in #385
• feat: Add updated readiness probe for vLLM containers by @rajinator in #330
• Fix queries and load script in monitoring by @Hritik003 in #383
• Arm cuda support (from clean branch) by @wseaton in #382
• Fix deadlink error in markdown-link-check by @yuanwu2017 in #404
• Update bug report template by @Gregory-Pereira in #413
• Fix release image tagging by @Gregory-Pereira in #379
• Update monitoring install for CKS by @Gregory-Pereira in #375
• Patch nvshmem to avoid an uninitialized value passed to RoCE by @smarterclayton in #407
• [Docs] Fix InferencePool version number getting cut off in WideEP guide by @tlrmchlsmth in #414
• Fix broken monitoring dashboard link by @smarterclayton in #422
• Set correct variables for built NVSHMEM by @smarterclayton in #417
• Update DeepEP to a version with a patch for setting NVSHMEM HCA mappings to CUDA device by @smarterclayton in #397
• swap release to tag creation image tagging by @Gregory-Pereira in #423
• pr vs release tag by @Gregory-Pereira in #424
• enable cache busting by @Gregory-Pereira in #425
• release.tag_name does not exist for a tag not part of a release by @Gregory-Pereira in #427
• unify darwin/arm64 with other platforms when install helmfile in install-deps.sh by @yitingdc in #267
• Set LD_LIBRARY_PATH for nvshmem appropriately by @smarterclayton in #429
• Update GKE to align to UBI images by @smarterclayton in #415
• updating to tags for v0.3.1 release by @Gregory-Pereira in #432
• bugfixing by @Gregory-Pereira in #433

New Contributors

• @yankay made their first contribution in #358
• @Ayobami-00 made their first contribution in #359
• @effi-ofer made their first contribution in #377
• @chewong made their first contribution in #335
• @rlakhtakia made their first contribution in #367
• @rajinator made their first contribution in #330
• @Hritik003 made their first contribution in #383
• @yitingdc made their first contribution in #267

Full Changelog: v0.3.0...v0.3.1

v0.3.0

📦 llm-d v0.3.0 Release Notes

This release of the llm-d repo will capture the release for the entirety of the project, guides, components, and all.

Release Date: 2025-10-10

---

Core Objectives

This release had a few key objectives:

• Increase support for specialized hardware backends (TPU, XPU)
• Increase cloud provider support (DOKS)
• Establish a metrics and observability story
• Wide-ep optimizations (EPLB, DBO, Async Scheduling, etc.)

🧩 Component Summary

Component	Version	Previous Version	Type
llm-d/llm-d-inference-scheduler	v0.3.2	v0.2.1	Image
llm-d-incubation/llm-d-modelservice	v0.2.10	v0.2.0	Helm Chart
llm-d/llm-d-routing-sidecar	v0.3.0	v0.2.0	Image
vllm-project/vllm	v0.11.0	v0.10.0	Editable install based on precompiled wheel
llm-d/llm-d-cuda	v0.3.0	v0.2.0	Image
llm-d/llm-d-gke	v0.3.0	NA (new)	Image
llm-d/llm-d-aws	v0.3.0	NA (new)	Image
llm-d/llm-d-xpu	v0.3.0	NA (new)	Image
llm-d/llm-d-inference-sim	v0.5.1	v0.3.0	Image
llm-d-incubation/llm-d-infra	v1.3.3	v1.1.1	Helm Chart
kubernetes-sig/gateway-api-inference-extension	v1.0.1	v0.5.1	Helm Chart
llm-d/llm-d-kv-cache-manager	v0.3.2	v0.2.0	Go Package (consumed in inference-scheduler)
llm-d/llm-d-benchmark	v0.3.0	v0.2.0	Tooling and Image

NOTE: In future we want to support compatibility matrixes. However as we are still getting off the ground, we cannot ensure that these components work with legacy versions.

---

🔹 llm-d/llm-d-inference-scheduler

• Description: This scheduler that makes optimized routing decisions for inference requests to the llm-d inference framework.
• Diff: v0.2.1 → v0.3.2

---

🔹 llm-d-incubation/llm-d-modelservice

• Description: modelservice is a Helm chart that simplifies LLM deployment on llm-d by declaratively managing Kubernetes resources for serving base models. It enables reproducible, scalable, and tunable model deployments through modular presets, and clean integration with llm-d ecosystem components (including vLLM, Gateway API Inference Extension, LeaderWorkerSet).
• Diff: v0.2.0 → v0.2.10

---

🔹 llm-d/llm-d-routing-sidecar

• Description: A reverse proxy redirecting incoming requests to the prefill worker specified in the x-prefiller-host-port HTTP request header.
• Diff: v0.2.0 → v0.3.0

---

🔹 vllm-project/vllm (upstream)

• Description: vLLM is a fast and easy-to-use library for LLM inference and serving. This project is the inferencing engine that forms the upstream of our llm-d/llm-d image.
• Diff: v0.10.0 → v0.11.0

---

🔹 llm-d/llm-d

• Description: A midstreamed image of vllm-project/vllm for inferencing, supporting features such as PD disaggregation, KV cache awareness and more.
• Diff: v0.2.0 → v0.3.0
• Image Variants: Different image variants of this component:
  • XPU: ghcr.io/llm-d/llm-d-xpu:v0.3.0
  • AWS: ghcr.io/llm-d/llm-d-aws:v0.3.0
    • Release v0.3.0 workaround for getting EFA to work
  • CUDA: ghcr.io/llm-d/llm-d-cuda:v0.3.0
  • GKE: ghcr.io/llm-d/llm-d-gke:v0.3.0
    • Release v0.3.0 workaround for running wide-ep on GKE with H200s

---

🔹 llm-d/llm-d-inference-sim

• Description: A light weight vLLM simulator emulates responses to the HTTP REST endpoints of vLLM.
• Diff: v0.3.0 → v0.5.1

---

🔹 llm-d-incubation/llm-d-infra

• Description: A helm chart for deploying gateway and gateway related infrastructure assets for llm-d.
• Diff: v1.1.1 → v1.3.3

---

🔹 kubernetes-sig/gateway-api-inference-extension

• Description: A Helm chart to deploy an InferencePool, a corresponding EndpointPicker (epp) deployment, and any other related assets.
• Diff: v0.5.1 → v1.0.1

---

🔹 llm-d/llm-d-kv-cache-manager

• Description: This repository contains the llm-d-kv-cache-manager, a pluggable service designed to enable KV-Cache Aware Routing and lay the foundation for advanced, cross-node cache coordination in vLLM-based serving platforms.
• Diff: v0.2.0 → v0.3.0

---

🔹 llm-d/llm-d-benchmark

• Description: This repository provides an automated workflow for benchmarking LLM inference using the llm-d stack. It includes tools for deployment, experiment execution, data collection, and teardown across multiple environments and deployment styles.
• Diff: v0.2.0 → v0.3.0

---

For more information on any of the component project or versions, please checkout their repos directly. For information on installing and using the new release refer to our guides. Thank you to all contributors who helped make this happen.