Skip to content

feat(conformance): Add test execution instruction to the guide. #878

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 18 commits into
base: main
Choose a base branch
from
Open

feat(conformance): Add test execution instruction to the guide. #878

wants to merge 18 commits into from

Conversation

SinaChavoshi
Copy link
Contributor

This pr fixes a formatting issue in the report guide as well as adding instructions for test execution.

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label May 27, 2025
@netlify Netlify
Copy link

netlify bot commented May 27, 2025 •
edited
Loading

✅ Deploy Preview for gateway-api-inference-extension ready!

Name Link
🔨 Latest commit 6a13dc5
🔍 Latest deploy log https://app.netlify.com/projects/gateway-api-inference-extension/deploys/68472d109276fa00082d231e
😎 Deploy Preview https://deploy-preview-878--gateway-api-inference-extension.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@k8s-ci-robot k8s-ci-robot requested review from danehans and kfswain May 27, 2025 22:32
@k8s-ci-robot k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label May 27, 2025
@k8s-ci-robot
Copy link
Contributor

Hi @SinaChavoshi. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label May 27, 2025
danehans
danehans reviewed May 28, 2025
View reviewed changes
danehans
danehans reviewed May 28, 2025
View reviewed changes
danehans
danehans reviewed May 28, 2025
View reviewed changes
danehans
danehans reviewed May 28, 2025
View reviewed changes
danehans
danehans reviewed May 28, 2025
View reviewed changes
danehans
danehans reviewed May 28, 2025
View reviewed changes
danehans
danehans reviewed May 28, 2025
View reviewed changes
@danehans
Copy link
Contributor

@SinaChavoshi thanks for creating these conformance docs. The docs need to be in site-src to get published to the docs site.

Consider adding your readme here: https://gateway-api-inference-extension.sigs.k8s.io/guides/implementers/#conformance-tests.

See these Make targets for building and locally serving the docs to review your changes.

danehans
danehans reviewed May 30, 2025
View reviewed changes
danehans
danehans reviewed May 30, 2025
View reviewed changes
danehans
danehans reviewed May 30, 2025
View reviewed changes
danehans
danehans reviewed May 30, 2025
View reviewed changes
danehans
danehans reviewed May 30, 2025
View reviewed changes
danehans
danehans reviewed May 30, 2025
View reviewed changes
@danehans
Copy link
Contributor

danehans commented Jun 6, 2025

@SinaChavoshi after #922 merges, please update this guide to explain that the conformance EPP is configured with the HeaderBasedTestingFilter (ENABLE_REQ_HEADER_BASED_SCHEDULER_FOR_TESTING=true env var in the EPP deployment manifest).

@SinaChavoshi
Copy link
Contributor Author

SinaChavoshi commented Jun 9, 2025 •
edited
Loading

@SinaChavoshi after #922 merges, please update this guide to explain that the conformance EPP is configured with the HeaderBasedTestingFilter (ENABLE_REQ_HEADER_BASED_SCHEDULER_FOR_TESTING=true env var in the EPP deployment manifest).

ignore my earlier comment will fix this shortly .

@SinaChavoshi
Copy link
Contributor Author

SinaChavoshi commented Jun 9, 2025 •
edited
Loading

@danehans @nirrozenbaum turns out that the current implementation of the EPP requires the InferencePool name to be provided explicit as a flag to EPP This does not allow us to create a shared EPP. I have created a separate issue #946 to discuss / track if we want to change this behaviour.

robscott
robscott approved these changes Jun 13, 2025
View reviewed changes
Copy link
Member

@robscott robscott left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @SinaChavoshi!

site-src/guides/conformance-tests.md
2. Choose an Implementation -
Install an [existing implementation](https://gateway-api-inference-extension.sigs.k8s.io/implementations/gateways/). For setup instructions, refer to the [The Quickstart Guide](https://gateway-api-inference-extension.sigs.k8s.io/guides/). Alternatively run tests against your implementation after completing the [implementer's guide](https://gateway-api-inference-extension.sigs.k8s.io/guides/implementers/#implementers-guide).

Note: Since the EPP (End Point Picker) takes the `InferencePool` name as an environment variable, each conformance test creates a corresponding EPP deployment for each `InferencePool` it defines. For conformance testing, the EPP is configured with the `HeaderBasedTestingFilter`. This is enabled by setting the `ENABLE_REQ_HEADER_BASED_SCHEDULER_FOR_TESTING=true` environment variable in the EPP deployment manifest.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@SinaChavoshi @zetxqx Is this actually needed if all our test cases are just telling the EPP which endpoint to select via header? Couldn't we just have a single base EPP that all InferencePools pointed to?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I got a refactor PR #982 to move resources to basic common resources as suggested, please take a look.

The resource I moved to resources folder are:

  1. EPP service named primary-endpoint-picker-svc
  2. Primary Inference Pool named: primary-inference-pool
  3. Primary Inference model named: primary-inference-model-server

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @zetxqx, could i suggest we proceed with this PR and update it after #982 is merged ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, sounds good.

@robscott
Copy link
Member

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jun 13, 2025
zetxqx
zetxqx approved these changes Jun 17, 2025
View reviewed changes
Copy link
Contributor

@zetxqx zetxqx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

site-src/guides/conformance-tests.md
2. Choose an Implementation -
Install an [existing implementation](https://gateway-api-inference-extension.sigs.k8s.io/implementations/gateways/). For setup instructions, refer to the [The Quickstart Guide](https://gateway-api-inference-extension.sigs.k8s.io/guides/). Alternatively run tests against your implementation after completing the [implementer's guide](https://gateway-api-inference-extension.sigs.k8s.io/guides/implementers/#implementers-guide).

Note: Since the EPP (End Point Picker) takes the `InferencePool` name as an environment variable, each conformance test creates a corresponding EPP deployment for each `InferencePool` it defines. For conformance testing, the EPP is configured with the `HeaderBasedTestingFilter`. This is enabled by setting the `ENABLE_REQ_HEADER_BASED_SCHEDULER_FOR_TESTING=true` environment variable in the EPP deployment manifest.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, sounds good.

@k8s-ci-robot
Copy link
Contributor

@zetxqx: changing LGTM is restricted to collaborators

In response to this:

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: robscott, SinaChavoshi, zetxqx
Once this PR has been reviewed and has the lgtm label, please assign sergeykanzhelev for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@danehans danehans danehans left review comments

@robscott robscott robscott approved these changes

@zetxqx zetxqx zetxqx approved these changes

@kfswain kfswain Awaiting requested review from kfswain

Assignees
No one assigned
Labels
cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@SinaChavoshi @k8s-ci-robot @danehans @robscott @zetxqx