Initial proposal for InferenceSchedulingObjective #1007
Conversation
k8s-ci-robot
added
the
cncf-cla: yes
✅ Deploy Preview for gateway-api-inference-extension ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: kfswain The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
approved
| @@ -0,0 +1,257 @@ | |||
| # Inference Scheduling Objective | |||
|
|
|||
| ```yaml | ||
| kind: InferenceModel | ||
| metadata: | ||
| name: llama4 |
There was a problem hiding this comment.
typo in the name, using same name for two inf models.
| // HTTPMatch is an http matching rule. The rules are ANDed. | ||
| type HTTPMatch struct { | ||
| // ModelName matches against the model name in the body as per OpenAI protocol | ||
| ModelName *string |
There was a problem hiding this comment.
if one wants to specify multiple models all with “app: prod”, he needs to define multiple HttpMatch entries, and repeat the same headers in all.
does it make sense to make this field an array of model names? (with a comment that list of models is Or’ed)
| // ModelName matches against the model name in the body as per OpenAI protocol | ||
| ModelName *string | ||
| // Headers specifies HTTP request header matchers. | ||
| Headers []HTTPHeaderMatch // mostly as defined in the gateway api |
There was a problem hiding this comment.
I’d be happy to see the differences between HttpHeaderMatch and GRPcHeaderMatch and see if we can consolidate to one.
personally I don’t like having two optional fields and documenting only one is allowed, although sometimes it is inevitable.
There was a problem hiding this comment.
I think we should start with http only and drop grpc for now. ext-proc only supports http anyways right now.
|
|
||
| #### After | ||
|
|
||
| Offload to httpRoute, EPP is now extended to override model name on: `X-Gateway-Model-Name`, added benefit of splitting on pools at the same place. |
There was a problem hiding this comment.
do we require this header always? is it mandatory?
Abdullah mentioned we might want to use bbr to achieve this, but as I stated in the past (while you were out on vacation), this should be verified in terms of performance. I have some concerns about making bbr a mandatory component, as it includes additional ext proc, adding another hop where we parse the request body. it may be expensive, especially in requests with big body (e.g., in a long and continuous chat).
the alternative is to require the sender to add the header, but then the sender is required to specify modelName both in header and body and this is not required by OpenAI schema.
none of the two options is good IMO. feels to me like we need additional thinking around this point in order to make it right.
There was a problem hiding this comment.
BBR is not a requirement for EPP. It is a requirement if one wants to do traffic splitting or model name redirection. The idea here is that EPP is now not concerned with those features per se and they are mostly pushed up to httproute and BBR.
Still some questions to hammer out. but mostly a lift and shift from: https://docs.google.com/document/d/1x6aI9pbTF5oOsaEQYc9n4pBBY3_AuEY2X51VKxmBSnU/edit?tab=t.0 with some corrections (Distinct objectives for the same target model is possible in the current API). Added open questions