v1.6.0

Release Highlights

1. Component Migration to llm-d

• Full-featured Endpoint Picker (EPP), Body-Based Routing (BBR), and Latency Predictor components have moved to the llm-d repository to live in their dedicated ecosystem.
• GAIE now focuses on core Kubernetes Gateway API inference specifications, CRDs, and conformance definitions.

2. Introduction of Lightweight EPP (LWEPP)

• Introduced Lightweight EPP (lwepp) as a minimal, fast reference ext-proc implementation designed specifically for running and validating Gateway API conformance tests.
• Container image is officially promoted and available at registry.k8s.io/gateway-api-inference-extension/lwepp:v1.6.0.

3. API Updates

• Optional endpointPickerRef: spec.endpointPickerRef on InferencePool is now optional.
• Removed Alpha APIs: The following experimental APIs have been removed from GAIE (migrated to llm-d):
  • InferenceObjective (inference.networking.x-k8s.io/v1alpha2)
  • InferenceModelRewrite (inference.networking.x-k8s.io/v1alpha2)
  • EndpointPickerConfig (config.inference.networking.k8s.io/v1alpha1)

What's Changed

• fix: stage conformance/go.sum in release helper by @danehans in #2806
• fix: stop reseeding picker RNG on every pick by @danehans in #2805
• [pluggable parser] add AppProtocol validation in the InferencePool Reconcile loop by @zetxqx in #2761
• Fix GKE E2E Tests by @rlakhtakia in #2807
• Fix/remove prepare data plugins featuregate by @kaushikmitr in #2811
• Feature/prefix cache max tokens param by @kaushikmitr in #2812
• Fix/OpenAI parser vertex path support by @kaushikmitr in #2813
• Fix error messages in datagraph validation for mismatched types by @rahulgurnani in #2816
• docs(epp): refactor pickers and add READMEs by @LukeAVanDrie in #2814
• docs(epp): refactor fairness and add READMEs by @LukeAVanDrie in #2815
• docs: finalize flow control ordering policies docs by @LukeAVanDrie in #2821
• Generalize the request type passed down the framework plugins: rename LLM->Inference by @RyanRosario in #2673
• Generalize request type: Move RequestBody from scheduling to requesthandling by @RyanRosario in #2808
• cleanup: remove legacy predicted-latency scorer by @kaushikmitr in #2826
• Generalize request type: remove parser from director by @RyanRosario in #2810
• refactor(requestcontrol): rename several requestcontrol plugin interface by @zetxqx in #2830
• fix: use the most recent gcr.io/k8s-staging-test-infra/gcb-docker-gcloud image by @zetxqx in #2833
• Minimal refactoring to have a single shard by @shmuelk in #2658
• feat(flowcontrol): Add ImmediateResponse abort mechanism for evicting in-flight requests by @RishabhSaini in #2737
• Fixes for In-Flight Load DAG Validation and DataLayer Wiring by @brent-srg in #2819
• chore(deps): bump sigs.k8s.io/gateway-api/conformance from 0.0.0-20260406183130-ace5815870c6 to 1.5.1 in /conformance by @dependabot[bot] in #2841
• chore(deps): bump sigs.k8s.io/gateway-api from 0.0.0-20260406183130-ace5815870c6 to 1.5.1 in /conformance by @dependabot[bot] in #2840
• chore(deps): bump go.opentelemetry.io/otel/exporters/stdout/stdouttrace from 1.42.0 to 1.43.0 by @dependabot[bot] in #2838
• chore(deps): bump github.com/google/cel-go from 0.26.0 to 0.28.0 by @dependabot[bot] in #2839
• fix(requestcontrol): cancel PrepareData plugin context on timeout by @kaushikmitr in #2847
• Revert "Remove GW API CRD dependency in conformance tests" by @ericdbishop in #2850
• fix(predictedlatency): keep prefillTokensInFlight from drifting negative by @kaushikmitr in #2848
• feat(concurrencydetector): ledger support for disaggregated prefill/decode accounting by @brent-srg in #2491
• fix(latencypredictor): reload model in every sync cycle, not just when this worker downloaded by @kaushikmitr in #2851
• fix(dynamo integration): reorder mutate place so that jsonPayload can be mutated by preRequest plugins by @zetxqx in #2854
• Add multimodal support to vllmgrpc parser by @rockswe in #2849
• Add prefix cache benchmarking scenarios by @rlakhtakia in #2845
• Update benchmarking guides to handle multilora case by @rlakhtakia in #2844
• docs: update EPP protocol spec with streaming mode and health check requirements by @Neha-dot-Yadav in #2514
• cleanup: consolidate requestdataproducer into dataproducer by @kaushikmitr in #2828
• feat(epp): Implement generic request data producer auto-configuration by @rahulgurnani in #2846
• fix(prefixcache): hash trailing bytes to prevent multimodal URL hotspotting by @rahulgurnani in #2823
• fix: Run response path plugin hooks asynchronously to reduce streaming latency by @gyliu513 in #2717
• fix: support http token-ID inputs for v1/completions and v1/embeddings by @zetxqx in #2856
• Move elevran from reviewers to maintainers. by @elevran in #2861
• fix: fixed regression failure for standalone without inference api support by @capri-xiyue in #2862
• chore(deps): bump the kubernetes group in /conformance with 4 updates by @dependabot[bot] in #2867
• chore(deps): bump the kubernetes group with 6 updates by @dependabot[bot] in #2865
• Lightweight EPP by @RyanRosario in #2842
• fix: helm chart flags rendering for boolean values by @andyzhangx in #2871
• feat(helm/bbr): make ext_proc events configurable by @yehuditkerido in #2860
• [BBR] test: add request plugin header mutation tests by @noalimoy in #2544
• chore(deps): bump sigs.k8s.io/structured-merge-diff/v6 from 6.3.2 to 6.4.0 by @dependabot[bot] in #2866
• Add Standalone EPP e2e tests by @rlakhtakia in #2864
• test: add Trivy image vulnerability scan for EPP container image by @andyzhangx in #2872
• CI: Add CloudBuild step for LWEPP image by @RyanRosario in #2875
• [BBR] Verify EPP/BBR have no cross-imports by @asaadbalum in #2413
• feat(bbr): return client.Client from Handle instead of Reader by @yehuditkerido in #2885
• Pin github/codeql-action/upload-sarif to full-length commit SHA by @andyzhangx in #2886
• Add standalone agentgateway proxy support by @danehans in #2882
• Revert "Add standalone agentgateway proxy support" by @ahg-g in #2893
• Update broken links by @rlakhtakia in #2897
• docs: fix release process quickstart testing instructions by @EzgiTastan in #2892
• Restore standalone Agentgateway proxy support by @danehans in #2895
• Added the ability to set the namespace of the gateway to httproute and a guide-specific header by @ahg-g in #2888
• Update community meeting docs by @danehans in #2903
• Support agentgateway standalone sidecar by @danehans in #2905
• Added a note regarding the code move by @ahg-g in #2906
• Update multiple pool guide for Agentgateway by @danehans in #2896
• Bump aquasecurity/trivy-action to v0.36.0 by @yankay in #2927
• chore(deps): bump the kubernetes group in /conformance with 4 updates by @dependabot[bot] in #2943
• BUGFIX: Failing e2e test -- bypass llm-d render server by @RyanRosario in #2953
• Add v1.5.0 conformance report for Istio by @ericdbishop in #2951
• security(deps): upgrade prometheus to v0.311.3 to fix CVEs by @RyanRosario in #2955
• chore(deps): bump github.com/onsi/ginkgo/v2 from 2.28.1 to 2.29.0 by @dependabot[bot] in #2939
• chore(deps): bump google.golang.org/grpc from 1.80.0 to 1.81.1 by @dependabot[bot] in #2957
• lwepp: incorporate body content in routing by @RyanRosario in #2935
• chore: remove outdated istio conformance 'scripts' by @ericdbishop in #2947
• Cleanup BBR from main by @nicolexin in #2959
• chore(deps): bump golang.org/x/sync from 0.20.0 to 0.21.0 in /conformance by @dependabot[bot] in #2961
• chore(deps): bump go.opentelemetry.io/otel/exporters/stdout/stdouttrace from 1.43.0 to 1.44.0 by @dependabot[bot] in #2960
• chore(deps): bump go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc from 1.43.0 to 1.44.0 by @dependabot[bot] in #2956
• chore(deps): bump github.com/google/cel-go from 0.28.0 to 0.28.1 by @dependabot[bot] in #2942
• Make endpointPickerRef optional by @DamianSawicki in #2898
• Document conformance report deprecation policy by @danehans in #2932
• Fix GatewayFollowingEPPRoutingWithDataParallelism Test Failure by @RyanRosario in #2963
• chore(deps): bump the kubernetes group with 6 updates by @dependabot[bot] in #2968
• chore(deps): bump github.com/onsi/gomega from 1.40.0 to 1.42.0 by @dependabot[bot] in #2969
• chore(deps): bump github.com/prometheus/prometheus from 0.311.3 to 0.312.0 by @dependabot[bot] in #2971
• chore(deps): bump github.com/onsi/ginkgo/v2 from 2.29.0 to 2.31.0 by @dependabot[bot] in #2972
• Cleanup EPP and Latency Predictor by @nicolexin in #2967
• Remove inference objective, model rewrite and endpoint picker config APIs by @nicolexin in #2973
• deps: update to Gateway API v1.6-rc.1 by @howardjohn in #2964
• chore(deps): bump sigs.k8s.io/gateway-api from 1.6.0-rc.1 to 1.6.0 by @dependabot[bot] in #2981
• chore(deps): bump the kubernetes group with 6 updates by @dependabot[bot] in #2980
• chore(deps): bump sigs.k8s.io/gateway-api/conformance from 1.6.0-rc.1 to 1.6.0 in /conformance by @dependabot[bot] in #2983
• Cleanup Documentation and Miscellaneous Tools / Manifests by @nicolexin in #2978
• chore(deps): bump google.golang.org/grpc from 1.81.1 to 1.82.0 by @dependabot[bot] in #2987
• chore(deps): bump golang.org/x/sync from 0.21.0 to 0.22.0 by @dependabot[bot] in #2989
• chore(deps): bump sigs.k8s.io/structured-merge-diff/v6 from 6.4.0 to 6.4.2 by @dependabot[bot] in #2990
• docs: fix InferencePool YAML examples in api-types and implementers guide by @alexcoma in #2992
• chore(deps): bump google.golang.org/grpc from 1.82.0 to 1.82.1 by @dependabot[bot] in #2997
• chore(deps): bump github.com/go-logr/logr from 1.4.3 to 1.4.4 by @dependabot[bot] in #2998
• chore(deps): bump sigs.k8s.io/gateway-api/conformance from 1.6.0 to 1.6.1 in /conformance by @dependabot[bot] in #3000
• chore(deps): bump sigs.k8s.io/gateway-api from 1.6.0 to 1.6.1 in /conformance by @dependabot[bot] in #3001
• docs: prune stale gateway implementations by @naveenvenk17 in #2976
• Fix flaky GatewayWeightedAcrossTwoInferencePools by @howardjohn in #3002
• Add NGINX Gateway Fabric v2.6.0 conformance report by @ciarams87 in #2919
• Clarify EPP endpoint list backend scope by @danehans in #3003
• chore(deps): bump the kubernetes group in /conformance with 4 updates by @dependabot[bot] in #3006
• chore(deps): bump the kubernetes group across 1 directory with 5 updates by @dependabot[bot] in #3005
• fix(makefile): add missing yq target by @nicolexin in #3008
• cleanup: update release template, remove stale chart references and obsolete e2e workflows by @nicolexin in #3009
• fix: swap inverted pod add and update log messages by @UgaTheDev in #3013
• test: cover InferencePoolToEndpointPool by @UgaTheDev in #3011
• fix: NewDefaultExtProcServerRunner drops GrpcHealthPort and SecureServing defaults by @UgaTheDev in #3010
• chore(deps): bump google.golang.org/grpc from 1.82.1 to 1.83.0 by @dependabot[bot] in #3012

New Contributors

• @rockswe made their first contribution in #2849
• @andyzhangx made their first contribution in #2871
• @EzgiTastan made their first contribution in #2892
• @alexcoma made their first contribution in #2992
• @naveenvenk17 made their first contribution in #2976
• @ciarams87 made their first contribution in #2919
• @UgaTheDev made their first contribution in #3013

Full Changelog: v1.5.0...v1.6.0