support grayscale processes based on basic models

[zgqqiang]

gateway-api-inference-extension/pkg/epp/backend/metrics/metrics.go

Line 106 in 8326110

if p.MetricMapping.LoraRequestInfo != nil {

I think the code only supports grayscale processing between multiple LoRAs based on the same basic model. Is there a plan to support grayscale processing based on the basic model,

[ahg-g]

Can you elaborate more? I am not following the ask unfortunately.

[zgqqiang]

For example, if I deployed two different large models in the backend and did not enable LoRA (Low Rank Adaptation) adapter support, I would encounter a 404 error when accessing them. This is because when scheduling, the model to be scheduled is determined based on the query model metric vllm: lora_requests_info, which includes the label label label and value. However, if I did not use LoRA, this metric would be meaningless. When calling one of them randomly, I would find that the model does not match and return a 404 error

apiVersion: inference.networking.x-k8s.io/v1alpha1
kind: InferenceModel
metadata:
  name: inferencemodel-sample
spec:
  criticality: Critical
  modelName: qwq
  poolRef:
    group: inference.networking.x-k8s.io
    kind: InferencePool
    name: reasoning-pool
  targetModels:
  - name: Qwen2.5-72B-Instruct
    weight: 50
  - name: Qwen-72B-Chat
    weight: 50

[zgqqiang]

I find the latest master branch supports custom indicator names to obtain, but the tags do not support customization

[ahg-g]

The assumption is that you would deploy an InferencePool per base model, and have an HTTPRoute route rule to select between the pools based on the model name, one can use body-based-router extension to extract the modelName from the body and place it in the header to do that.

[kfswain]

Heya @zgqqiang did we get this issue resolved?

[zgqqiang]

@ahg-g The assumption is that you would deploy an InferencePool per base model, and have an HTTPRoute route rule to select between the pools based on the model name, one can use body-based-router extension to extract the modelName from the body and place it in the header to do that.

This situation will result in my routing not being dynamically scheduled based on vllm metrics. For example, if two services in the same model do not have enable lora enabled, and one service is currently processing 100 requests while the other is not processing any requests, my call will still randomly access one of them, because the current judgment is based on vllm: lora_requests_info metrics

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale