feat(conformance) add EPP unavailable fail-open test #999

zetxqx · 2025-06-17T01:03:30Z

This pull request introduces a new conformance test, epp_unavailable_fail_open, to validate the behavior of the extensionRef.failureMode==FailOpen setting.

A new conformance test, epp_unavailable_fail_open, has been added.
The test simulates an EPP becoming unavailable by deleting the EPP deployment and verifies that the gateway fails open and routes traffic to the backend service.
This PR is based on the work done in PR feat(conformance): Add EPP conformance test for Gateway routing #961, which added the initial EPP conformance test for Gateway routing.

Verification

gke-l7

go test -v ./conformance -args -debug -gateway-class gke-l7-regional-external-managed -cleanup-base-resources=false -allow-crds-mismatch=true -run-test EppUnAvailableFailOpen

=== NAME  TestConformance/EppUnAvailableFailOpen
    apply.go:283: 2025-06-17T01:19:39.311334464Z: Deleting epp-to-inference-model-reader RoleBinding
    apply.go:283: 2025-06-17T01:19:39.403450067Z: Deleting inference-model-reader Role
    apply.go:283: 2025-06-17T01:19:39.482466834Z: Deleting infra-backend-epp Deployment
    apply.go:283: 2025-06-17T01:19:39.494211326Z: Deleting infra-backend-endpoint-picker Service
    apply.go:283: 2025-06-17T01:19:39.61129453Z: Deleting httproute-for-primary-gw HTTPRoute
    apply.go:283: 2025-06-17T01:19:39.679690563Z: Deleting normal-gateway-pool InferencePool
    apply.go:283: 2025-06-17T01:19:39.746823836Z: Deleting conformance-fake-model-server InferenceModel
    apply.go:283: 2025-06-17T01:19:39.823244608Z: Deleting infra-backend-deployment Deployment
=== RUN   TestConformance/GatewayFollowingEPPRouting
    conformance.go:68: Skipping GatewayFollowingEPPRouting: test explicitly skipped
=== RUN   TestConformance/HTTPRouteInvalidInferencePoolRef
    conformance.go:68: Skipping HTTPRouteInvalidInferencePoolRef: test explicitly skipped
=== RUN   TestConformance/InferencePoolAccepted
    conformance.go:68: Skipping InferencePoolAccepted: test explicitly skipped
=== RUN   TestConformance/InferencePoolResolvedRefsCondition
    conformance.go:68: Skipping InferencePoolResolvedRefsCondition: test explicitly skipped
--- PASS: TestConformance (71.37s)
    --- PASS: TestConformance/EppUnAvailableFailOpen (68.24s)
        --- PASS: TestConformance/EppUnAvailableFailOpen/Phase_1:_Verify_baseline_connectivity_with_EPP_available (48.22s)
        --- PASS: TestConformance/EppUnAvailableFailOpen/Phase_2:_Verify_fail-open_behavior_after_EPP_becomes_unavailable (0.16s)
    --- SKIP: TestConformance/GatewayFollowingEPPRouting (0.00s)
    --- SKIP: TestConformance/HTTPRouteInvalidInferencePoolRef (0.00s)
    --- SKIP: TestConformance/InferencePoolAccepted (0.00s)
    --- SKIP: TestConformance/InferencePoolResolvedRefsCondition (0.00s)
PASS
ok  	sigs.k8s.io/gateway-api-inference-extension/conformance	71.575s

istio

go test -v ./conformance -args -debug -gateway-class istio -cleanup-base-resources=false -allow-crds-mismatch=true -run-test EppUnAvailableFailOpen

=== RUN   TestConformance/InferencePoolResolvedRefsCondition
    conformance.go:68: Skipping InferencePoolResolvedRefsCondition: test explicitly skipped
--- PASS: TestConformance (11.52s)
    --- PASS: TestConformance/EppUnAvailableFailOpen (8.80s)
        --- PASS: TestConformance/EppUnAvailableFailOpen/Phase_1:_Verify_baseline_connectivity_with_EPP_available (7.06s)
        --- PASS: TestConformance/EppUnAvailableFailOpen/Phase_2:_Verify_fail-open_behavior_after_EPP_becomes_unavailable (0.09s)
    --- SKIP: TestConformance/GatewayFollowingEPPRouting (0.00s)
    --- SKIP: TestConformance/HTTPRouteInvalidInferencePoolRef (0.00s)
    --- SKIP: TestConformance/InferencePoolAccepted (0.00s)
    --- SKIP: TestConformance/InferencePoolResolvedRefsCondition (0.00s)
PASS
ok  	sigs.k8s.io/gateway-api-inference-extension/conformance	11.715s

… and small refactor.

…e is set to failOpen.

netlify · 2025-06-17T01:03:35Z

✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	`32627c8`
🔍 Latest deploy log	https://app.netlify.com/projects/gateway-api-inference-extension/deploys/6850bee445ccc200083ddbce
😎 Deploy Preview	https://deploy-preview-999--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

k8s-ci-robot · 2025-06-17T01:03:36Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: zetxqx
Once this PR has been reviewed and has the lgtm label, please assign danehans for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2025-06-17T01:03:39Z

Hi @zetxqx. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

robscott

Thanks @zetxqx! A few small comments, otherwise LGTM.

robscott · 2025-06-17T04:41:14Z

conformance/tests/basic/epp_unavailable_fail_open.yaml

+        type: PathPrefix
+        value: /primary-gateway-test
+---
+# --- Conformance EPP service Definition ---


Would really like to see everything below this in base manifests + reused to simplify test cases.

robscott · 2025-06-17T04:41:48Z

conformance/tests/basic/epp_unavailable_fail_open.yaml

+# --- InferenceModel Definition ---
+# Service for the infra-backend-deployment.
+apiVersion: inference.networking.x-k8s.io/v1alpha2
+kind: InferenceModel
+metadata:
+  name: conformance-fake-model-server
+  namespace: gateway-conformance-app-backend
+spec:
+  modelName: conformance-fake-model
+  criticality: Critical # Mark it as critical to bypass the saturation check since the model server is fake and don't have such metrics. 
+  poolRef:
+    name: normal-gateway-pool


Recommend removing - follow up issue is fine since I know it requires corresponding EPP changes

robscott · 2025-06-17T04:43:49Z

conformance/tests/basic/epp_unavailable_fail_open.go

+			Headers:   map[string]string{eppSelectionHeaderName: targetPodIP},
+			Method:    http.MethodPost,
+			Body:      requestBody,
+			Backend:   appPodBackendPrefix,


In the first test don't we need to ensure that the Gateway routed to a specific Pod, while in the second test any Pod with this prefix is sufficient?

robscott · 2025-06-17T04:44:27Z

conformance/tests/basic/gateway_following_epp_routing.yaml

+            fieldRef:
+              fieldPath: status.podIP
+---
+# --- InferenceModel Definition ---


Same comments as above, would really like to minimize repetition here.

robscott · 2025-06-17T04:45:24Z

conformance/tests/basic/gateway_following_epp_routing.yaml

+  extensionRef:
+    name: infra-backend-endpoint-picker
+---
+# --- HTTPRoute for Primary Gateway (conformance-gateway) ---


It's possible that the only variation we need for most tests is on HTTPRoute, likely worth exploring how much we can share across tests. (Can be follow up issue)

robscott · 2025-06-17T04:46:28Z

conformance/utils/traffic/traffic.go

+
+// MakeRequestAndExpectEventuallyConsistentResponse makes a request using the parameters
+// from the Request struct and waits for the response to consistently match the expectations.
+func MakeRequestAndExpectEventuallyConsistentResponse(


Please make sure you're converging with @SinaChavoshi on these functions across PRs.

k8s-ci-robot · 2025-06-17T21:27:15Z

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

zetxqx added 13 commits June 9, 2025 23:32

Add gateway_following_epp_routing test.

8d6fb58

One working version.

f879558

Okay version of GatwayFollowingEPPRouting conformance test.

9f3e382

fix typos and formats.

b83290c

Merge branch 'main' into epptest

a680c76

upgrader gateway-api versino to use the updated conformance testutils…

ff3086a

… and small refactor.

use AllowCRDsMismatch to bypass.

5b725d0

format.

838b535

refactor more.

f313e58

wire up the flag.

bf95ea9

Refine test cases.

215b7ce

Refine log error info.

417cb44

Add test for epp becoming unavailable and the extensionRef.failureMod…

32627c8

…e is set to failOpen.

k8s-ci-robot requested review from danehans and robscott June 17, 2025 01:03

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jun 17, 2025

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jun 17, 2025

zetxqx changed the title ~~feat(conformance) add epp_unavailable_fail_open test to test the extensionRef.failureMode==FailOpen behavior~~ feat(conformance) add EPP unavailable fail-open test Jun 17, 2025

zetxqx mentioned this pull request Jun 17, 2025

Conformance Tests for v0.5 Release #977

Open

12 tasks

robscott reviewed Jun 17, 2025

View reviewed changes

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 17, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(conformance) add EPP unavailable fail-open test #999

feat(conformance) add EPP unavailable fail-open test #999

Uh oh!

zetxqx commented Jun 17, 2025 •

edited

Loading

Uh oh!

netlify bot commented Jun 17, 2025 •

edited

Loading

Uh oh!

k8s-ci-robot commented Jun 17, 2025

Uh oh!

k8s-ci-robot commented Jun 17, 2025

Uh oh!

robscott left a comment

Uh oh!

robscott Jun 17, 2025

Uh oh!

robscott Jun 17, 2025

Uh oh!

robscott Jun 17, 2025

Uh oh!

robscott Jun 17, 2025

Uh oh!

robscott Jun 17, 2025

Uh oh!

robscott Jun 17, 2025

Uh oh!

k8s-ci-robot commented Jun 17, 2025

Uh oh!

Uh oh!

feat(conformance) add EPP unavailable fail-open test #999

Are you sure you want to change the base?

feat(conformance) add EPP unavailable fail-open test #999

Uh oh!

Conversation

zetxqx commented Jun 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Verification

gke-l7

istio

Uh oh!

netlify bot commented Jun 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for gateway-api-inference-extension ready!

Uh oh!

k8s-ci-robot commented Jun 17, 2025

Uh oh!

k8s-ci-robot commented Jun 17, 2025

Uh oh!

robscott left a comment

Choose a reason for hiding this comment

Uh oh!

robscott Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

robscott Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

robscott Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

robscott Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

robscott Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

robscott Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

k8s-ci-robot commented Jun 17, 2025

Uh oh!

Uh oh!

zetxqx commented Jun 17, 2025 •

edited

Loading

netlify bot commented Jun 17, 2025 •

edited

Loading