-
Notifications
You must be signed in to change notification settings - Fork 627
Open
Labels
Area: InferenceActivities related to Gateway API Inference Extension support.Activities related to Gateway API Inference Extension support.Priority: HighRequired in next 3 months to make progress, bugs that affect multiple users, or very bad UXRequired in next 3 months to make progress, bugs that affect multiple users, or very bad UXhelp wantedDenotes an issue that needs help from a contributor.Denotes an issue that needs help from a contributor.
Description
Add a plugin that utilizes Envoy External Processor (ExtProc) to act as an external Mixture-of-Models (MoM) router. The router should intelligently direct OpenAI API requests to the most suitable backend GenAI model from a defined pool based on the semantic understanding of the request's intent.
Metadata
Metadata
Assignees
Labels
Area: InferenceActivities related to Gateway API Inference Extension support.Activities related to Gateway API Inference Extension support.Priority: HighRequired in next 3 months to make progress, bugs that affect multiple users, or very bad UXRequired in next 3 months to make progress, bugs that affect multiple users, or very bad UXhelp wantedDenotes an issue that needs help from a contributor.Denotes an issue that needs help from a contributor.