Skip to content

Add subsetting logic for epp #981

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Add subsetting logic for epp #981

wants to merge 1 commit into from
+455 −23

Conversation

rlakhtakia
Copy link
Contributor

Issue: #415
Proposal

Add subsetting filter to ensure EPP only selects from the list of endpoints passed in through request metadata.

Changes:

  • Filter logic + unit test + integration test
  • Update request type to pass in metadata context
  • Update scheduler profiles to use filter

@netlify Netlify
Copy link

netlify bot commented Jun 13, 2025 •
edited
Loading

✅ Deploy Preview for gateway-api-inference-extension ready!

Name Link
🔨 Latest commit d35c57d
🔍 Latest deploy log https://app.netlify.com/projects/gateway-api-inference-extension/deploys/6851af7990e4d500080f80d7
😎 Deploy Preview https://deploy-preview-981--gateway-api-inference-extension.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jun 13, 2025
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: rlakhtakia
Once this PR has been reviewed and has the lgtm label, please assign jeffwan for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jun 13, 2025
@k8s-ci-robot
Copy link
Contributor

Hi @rlakhtakia. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jun 13, 2025
nirrozenbaum
nirrozenbaum reviewed Jun 13, 2025
View reviewed changes
Copy link
Contributor

@nirrozenbaum nirrozenbaum left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

envoy specifics should be only in the server.go.
rest of the code should work with go general structs like maps

kfswain
kfswain reviewed Jun 13, 2025
View reviewed changes
@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 13, 2025
@k8s-ci-robot k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 13, 2025
nirrozenbaum
nirrozenbaum reviewed Jun 15, 2025
View reviewed changes
nirrozenbaum
nirrozenbaum reviewed Jun 15, 2025
View reviewed changes
@nirrozenbaum
Copy link
Contributor

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jun 16, 2025
kfswain
kfswain reviewed Jun 16, 2025
View reviewed changes
Copy link
Collaborator

@kfswain kfswain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just a couple comments to future proof us! otherwise lgtm

@kfswain
Copy link
Collaborator

kfswain commented Jun 17, 2025

/retest

@rlakhtakia
Copy link
Contributor Author

/retest

liu-cong
liu-cong reviewed Jun 17, 2025
View reviewed changes
Copy link
Contributor

@liu-cong liu-cong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just a few nits

pkg/epp/scheduling/framework/plugins/filter/subsetting_filter.go
// Create a map of endpoint addys for easy lookup
endpoints := make(map[string]bool)
for _, endpoint := range endpointSubsetList {
epStr := strings.Split(endpoint.(string), ":")[0]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Can you add a comment with an example endpoint value and explain why this spitting makes sense?

test/integration/util.go
@@ -86,7 +86,7 @@ func StreamedRequest(t *testing.T, client extProcPb.ExternalProcessor_ProcessCli
return responses, nil
}

func GenerateRequest(logger logr.Logger, prompt, model string) *extProcPb.ProcessingRequest {
func GenerateRequest(logger logr.Logger, prompt, model string, filterMetadata interface{}) *extProcPb.ProcessingRequest {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Can we make the filterMetadata a strongly typed structure for ease of use? Internally in the utils you can do any conversion needed.

test/integration/epp/hermetic_test.go
"test2",
modelSQLLora,
[]interface{}{"192.168.1.3:8000"}),
// Pod 3 will be picked because it is the only one matching subsetting target.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// Pod 3 will be picked because it is the only one matching subsetting target.
// Pod 3 has high queue and kv cache utilization, but it will still be picked because it is the only one matching subsetting target.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@danehans danehans danehans left review comments

@liu-cong liu-cong liu-cong left review comments

@nirrozenbaum nirrozenbaum nirrozenbaum left review comments

@kfswain kfswain kfswain left review comments

@ahg-g ahg-g Awaiting requested review from ahg-g

Assignees
No one assigned
Labels
cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@rlakhtakia @k8s-ci-robot @nirrozenbaum @kfswain @danehans @liu-cong