-
Notifications
You must be signed in to change notification settings - Fork 115
Add subsetting logic for epp #981
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
✅ Deploy Preview for gateway-api-inference-extension ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Hi @rlakhtakia. Thanks for your PR. I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
envoy specifics should be only in the server.go.
rest of the code should work with go general structs like maps
/ok-to-test |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
just a couple comments to future proof us! otherwise lgtm
/retest |
/retest |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
just a few nits
pkg/epp/scheduling/framework/plugins/filter/subsetting_filter.go
Outdated
Show resolved
Hide resolved
It is part of conformance. My point is that this doesn't need to be the deciding factor of how to implement a feature. Plugins is an architectural design pattern where features are implemented as callbacks on defined extension points. The configuration of the plugins is where we decide which ones are a must (part of conformance) and which are optional. The former can be done in code (or via validation on the config api), the latter via the config api. |
Yeah, I see your point, just making sure we are considering this fundamental. I think it's fine, so long as its only filters, and we don't add any extension points that run before filters. To me it makes sense that any immutable features run strictly before or after any user configured code. So that works for |
/retest |
++. agreed. |
pkg/epp/scheduling/framework/plugins/filter/subsetting_filter.go
Outdated
Show resolved
Hide resolved
@rlakhtakia pls rebase and run unit tests locally, this should allow you to find compile time problems earlier. |
I went over the PR. overall it LGTM if we decide to keep it as filter. one additional concern about the earlier discussion on this thread -
so in addition to the above comment @kfswain wrote, this works fine in current design as long as we have only one profile (not the case in llm-d). gateway-api-inference-extension/pkg/epp/requestcontrol/director.go Lines 141 to 142 in fe5dea3
and then candidate pods are filtered only once before scheduling, no matter how many profiles we have. @ahg-g @kfswain leaving the final stamp for you to decide if we want to merge or not. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks! One comment on how to deal with cases where the metadata exists but the list is empty, otherwise looks great.
@@ -304,6 +305,7 @@ func (r *Runner) initializeScheduler() (*scheduling.Scheduler, error) { | |||
kvCacheScorerWeight := envutil.GetEnvInt("KV_CACHE_SCORE_WEIGHT", scorer.DefaultKVCacheScorerWeight, setupLog) | |||
|
|||
schedulerProfile := framework.NewSchedulerProfile(). | |||
WithFilters(filter.NewSubsetFilter()). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is fine for now, but please open an issue to allow configuring this across profiles irrespective of the source of the configuration (see the discussion we had on the issue)
pkg/epp/scheduling/framework/plugins/filter/subsetting_filter.go
Outdated
Show resolved
Hide resolved
if !found { | ||
return pods | ||
} else if len(endpointSubsetList) == 0 { | ||
return pods |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Need to return an empty list here, not all pods, meaning all pods are filtered.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed now, and added a testcase to verify this.
@nirrozenbaum agree on the need for a followup, my thinking is to have a "framework" for configuring mandatory plugins via code (i.e., this is no a user facing api). An initial cut would be as simple as a list of filters that get prepended to all profiles. Since this is hardcoded, we can evolve it slowly and by reacting to new needs, it doesn't need to be fully fleshed out from the beginning since it is not user facing. |
Created #1068 as a follow up |
/lgtm Congrats @rlakhtakia on your first inference gateway PR! |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: ahg-g, rlakhtakia The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Issue: #415
Proposal
Add subsetting filter to ensure EPP only selects from the list of endpoints passed in through request metadata.
Changes: