feat(conformance): Add EPP conformance test for Gateway routing #961

zetxqx · 2025-06-11T03:58:33Z

This pull request introduces a new conformance test, GatewayFollowingEPPRouting, to validate that a Gateway correctly implements routing logic based on an EPP.

The new test suite ensures that the Gateway honors the EPP's endpoint selection. We used the testing-epp to control the EPP behavior using request header. The two test cases are

request header:[pod1IP, pod2IP, pod3IP]: EPP returns pod1IP, pod2IP, pod3IP, gateway should select the first one pod1IP and send traffic to
request header:[pod3IP, pod2IP, pod1IP] EPP returns ppod3IP, pod2IP, pod1IP, gateway should select the first one pod3IP and send traffic to

Verification

The new conformance test was run against the gke-l7-regional-external-managed GatewayClass and passed successfully.

against gke-l7

Test command

go test -v ./conformance -args -debug \
  -gateway-class gke-l7-regional-external-managed \
  -cleanup-base-resources=false \
  -run-test GatwayFollowingEPPRouting

Test results

=== RUN   TestConformance/HTTPRouteInvalidInferencePoolRef
    conformance.go:68: Skipping HTTPRouteInvalidInferencePoolRef: test explicitly skipped
=== RUN   TestConformance/InferencePoolAccepted
    conformance.go:68: Skipping InferencePoolAccepted: test explicitly skipped
=== RUN   TestConformance/InferencePoolResolvedRefsCondition
    conformance.go:68: Skipping InferencePoolResolvedRefsCondition: test explicitly skipped
--- PASS: TestConformance (53.87s)
    --- PASS: TestConformance/GatewayFollowingEPPRouting (51.41s)
        --- PASS: TestConformance/GatewayFollowingEPPRouting/should_route_traffic_to_a_single_designated_pod (0.08s)
        --- PASS: TestConformance/GatewayFollowingEPPRouting/should_route_traffic_to_two_designated_pods (0.09s)
        --- PASS: TestConformance/GatewayFollowingEPPRouting/should_route_traffic_to_all_available_pods (0.10s)
    --- SKIP: TestConformance/HTTPRouteInvalidInferencePoolRef (0.00s)
    --- SKIP: TestConformance/InferencePoolAccepted (0.00s)
    --- SKIP: TestConformance/InferencePoolResolvedRefsCondition (0.00s)
PASS
ok  	sigs.k8s.io/gateway-api-inference-extension/conformance	54.054s

against istio (FAILED)

The istio implementation may still send traffic to pods not selected by EPP.

go test -v ./conformance -args -debug -gateway-class istio -cleanup-base-resources=false -run-test GatewayFollowingEPPRouting

Test results

error log: gateway_following_epp_routing.go:144: Not all the requests are sent to the expectedPods successfully, err: request was handled by an unexpected pod "infra-backend-deployment-678ff64fb-txms2", Reached pods with request counts: map[infra-backend-deployment-678ff64fb-tnlhd:30 infra-backend-deployment-678ff64fb-txms2:32 infra-backend-deployment-678ff64fb-zjwk2:38]

=== RUN   TestConformance/HTTPRouteInvalidInferencePoolRef
    conformance.go:68: Skipping HTTPRouteInvalidInferencePoolRef: test explicitly skipped
=== RUN   TestConformance/InferencePoolAccepted
    conformance.go:68: Skipping InferencePoolAccepted: test explicitly skipped
=== RUN   TestConformance/InferencePoolResolvedRefsCondition
    conformance.go:68: Skipping InferencePoolResolvedRefsCondition: test explicitly skipped
--- FAIL: TestConformance (139.98s)
    --- FAIL: TestConformance/GatewayFollowingEPPRouting (137.31s)
        --- FAIL: TestConformance/GatewayFollowingEPPRouting/should_route_traffic_to_a_single_designated_pod (0.05s)
        --- FAIL: TestConformance/GatewayFollowingEPPRouting/should_route_traffic_to_two_designated_pods (0.05s)
        --- PASS: TestConformance/GatewayFollowingEPPRouting/should_route_traffic_to_all_available_pods (0.05s)
    --- SKIP: TestConformance/HTTPRouteInvalidInferencePoolRef (0.00s)
    --- SKIP: TestConformance/InferencePoolAccepted (0.00s)
    --- SKIP: TestConformance/InferencePoolResolvedRefsCondition (0.00s)
FAIL
FAIL	sigs.k8s.io/gateway-api-inference-extension/conformance	140.176s
FAIL

TODO(all done now)

(Done) Currently the Mock EPP can only accept one endpoint from the request header, I need to add multi-endpoints support to test the case when EPP return multiple endpoints. Extending feat(Conformance): Add a header based filter to make a controllable epp behavior determined by request header. #922
(Done) Need to refactor the test utils. The library our conformance test is depending on need some modification. Currently the library can only make request w/o body. I try to make some minimum change to make this PR.
(Done) Need to figure out a poll and check way to make sure the whole stack is ready to server request.

netlify · 2025-06-11T03:58:38Z

✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	`be9c779`
🔍 Latest deploy log	https://app.netlify.com/projects/gateway-api-inference-extension/deploys/6852d53b6e6ca700088914d8
😎 Deploy Preview	https://deploy-preview-961--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

k8s-ci-robot · 2025-06-11T03:58:39Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: zetxqx
Once this PR has been reviewed and has the lgtm label, please assign sergeykanzhelev for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2025-06-11T03:58:42Z

Hi @zetxqx. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

conformance/utils/traffic/traffic.go

conformance/tests/basic/gateway_following_epp_routing.go

robscott

Thanks @zetxqx!

conformance/tests/basic/gateway_following_epp_routing.go

conformance/tests/basic/gateway_following_epp_routing.yaml

conformance/utils/kubernetes/helpers.go

conformance/utils/traffic/traffic.go

conformance/tests/basic/gateway_following_epp_routing.yaml

… and small refactor.

conformance/utils/traffic/traffic.go

robscott

Thanks for all the work on this @zetxqx!

conformance/conformance.go

conformance/tests/basic/gateway_following_epp_routing.go

conformance/utils/traffic/traffic.go

robscott · 2025-06-13T20:05:21Z

/ok-to-test

zetxqx · 2025-06-13T23:32:59Z

/retest

robscott

Thanks @zetxqx! This is looking pretty close

conformance/tests/basic/gateway_following_epp_routing.go

conformance/utils/traffic/traffic.go

robscott · 2025-06-13T23:46:44Z

conformance/tests/basic/gateway_following_epp_routing.yaml

+# --- Backend Service ---
+# Service for the infra-backend-deployment.
+apiVersion: inference.networking.x-k8s.io/v1alpha2
+kind: InferenceModel
+metadata:
+  name: conformance-fake-model-server
+  namespace: gateway-conformance-app-backend
+spec:
+  modelName: conformance-fake-model
+  criticality: Critical # Mark it as critical to bypass the saturation check since the model server is fake and don't have such metrics. 
+  poolRef:
+    name: normal-gateway-pool


This isn't a Service + I don't think we need it

Rename the comment.

"I don't think we need it"
I think we still need the InferenceModel to direct the traffic to the correct POD hosting the model.

I didn't think we're testing EPP behavior? Instead I thought that every request would tell EPP which Pod IP(s) to return, and then our test would validate EPP behavior.

You're right about the test's intent. The need for the InferenceModel is an implementation detail of the testingEPP.

Even though we control the final endpoint list with request headers, the testingEPP still requires the InferenceModel resource to:

Discover the initial list of all possible pods from the referenced InferencePoo. The requestHeadBased filter(testingEPP used) then selects from this list.

Identify the modelName it is serving

So, while the test logic focuses on the gateway's reaction to the EPP's output, the testingEPP itself still needs the InferenceModel to know the universe of possible backends and the model being served

Can we modify EPP so it doesn't need InferenceModel when running in this context?

cc @ahg-g @kfswain

It's a little hard and needs more refactoring because currently the Pod lists is from the datastore(https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/pkg/epp/scheduling/scheduler.go#L108). And it's getting updated from some reconciler logic I believe. Hence there is not a easy way we can pass in a input parameters to EPP as what we did for custom filter.

Also, currently the EPP request control flow tries to find a inferenceModel(https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/pkg/epp/requestcontrol/director.go#L94-L97), so w/o inferenceModel the EPP will always return error.

Yeah, we can do that. @zetxqx will you cut an issue and assign it to me? Thanks!

Thanks both,

I created an issue to track the EPP refactoring: Refactor EPP to decouple core logic from InferenceModel resource #1001. @kfswain

I created another issue for remove inferenceModel dependency in the conformance tests: Conformance: refactor GatewayFollowingEPPRouting Conformance Test to Remove InferenceModel Dependency #1002

conformance/tests/basic/gateway_following_epp_routing.yaml

conformance/tests/basic/gateway_following_epp_routing.go

zetxqx added 3 commits June 9, 2025 23:32

Add gateway_following_epp_routing test.

8d6fb58

One working version.

f879558

Okay version of GatwayFollowingEPPRouting conformance test.

9f3e382

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jun 11, 2025

k8s-ci-robot requested review from ahg-g and danehans June 11, 2025 03:58

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jun 11, 2025

k8s-ci-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Jun 11, 2025

zetxqx commented Jun 11, 2025

View reviewed changes

conformance/utils/traffic/traffic.go Outdated Show resolved Hide resolved

zetxqx commented Jun 11, 2025

View reviewed changes

conformance/tests/basic/gateway_following_epp_routing.go Outdated Show resolved Hide resolved

robscott reviewed Jun 11, 2025

View reviewed changes

zetxqx added 3 commits June 11, 2025 18:28

fix typos and formats.

b83290c

Merge branch 'main' into epptest

a680c76

upgrader gateway-api versino to use the updated conformance testutils…

ff3086a

… and small refactor.

zetxqx mentioned this pull request Jun 11, 2025

Update Gateway API dependency to a stable release tag once available #969

Open

zetxqx commented Jun 11, 2025

View reviewed changes

conformance/utils/traffic/traffic.go Outdated Show resolved Hide resolved

zetxqx requested a review from robscott June 11, 2025 22:54

use AllowCRDsMismatch to bypass.

5b725d0

zetxqx mentioned this pull request Jun 12, 2025

Conformance Tests for v0.5 Release #977

Open

12 tasks

format.

838b535

robscott reviewed Jun 13, 2025

View reviewed changes

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jun 13, 2025

refactor more.

f313e58

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jun 13, 2025

wire up the flag.

bf95ea9

zetxqx mentioned this pull request Jun 13, 2025

Refactor conformance tests to use new test util function to replace MakeRequestAndExpectSuccess and MakeRequestAndExpectNotFound #983

Open

zetxqx requested a review from robscott June 13, 2025 23:33

robscott reviewed Jun 13, 2025

View reviewed changes

Refine test cases.

215b7ce

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jun 16, 2025

zetxqx requested a review from robscott June 16, 2025 07:04

Refine log error info.

417cb44

This was referenced Jun 17, 2025

feat(conformance): Add HTTPRouteMultipleRulesDifferentPools test #834

Merged

refactor:(conformance): combine methods for getting pods in helper method. #1003

Open

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 17, 2025

Merge branch 'main' into epptest

be9c779

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 18, 2025

feat(conformance): Add EPP conformance test for Gateway routing #961

Are you sure you want to change the base?

feat(conformance): Add EPP conformance test for Gateway routing #961

Conversation

zetxqx commented Jun 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Verification

against gke-l7

against istio (FAILED)

TODO(all done now)

Uh oh!

netlify bot commented Jun 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for gateway-api-inference-extension ready!

Uh oh!

k8s-ci-robot commented Jun 11, 2025

Uh oh!

k8s-ci-robot commented Jun 11, 2025

Uh oh!

Uh oh!

Uh oh!

robscott left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

robscott left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

robscott commented Jun 13, 2025

Uh oh!

zetxqx commented Jun 13, 2025

Uh oh!

robscott left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zetxqx commented Jun 11, 2025 •

edited

Loading

netlify bot commented Jun 11, 2025 •

edited

Loading