Skip to content

[OEP] Adding OEP for Multi-instance serving.#584

Open
shenoyvvarun wants to merge 1 commit intoome-projects:mainfrom
shenoyvvarun:vasheno/mig-support-oep
Open

[OEP] Adding OEP for Multi-instance serving.#584
shenoyvvarun wants to merge 1 commit intoome-projects:mainfrom
shenoyvvarun:vasheno/mig-support-oep

Conversation

@shenoyvvarun
Copy link
Copy Markdown
Contributor

What this PR does

-Support for Multi-instance GPU serving in OME.

  • NOTE: This OEP supports only hardware isolated Multi instance GPU serving. Other way is to support multiple inference service is via KVCached which doesn't provide isolation but solves the case where a customer wants to serve multiple InferenceService via the same DAC.

Why we need it

  • Great opportunity to sell GPUs to customers who want predictable performance of DAC but, don't want to commit to full GPU.

Fixes #

How to test

Checklist

  • Tests added/updated (if applicable)
  • Docs updated (if applicable)
  • make test passes locally

@shenoyvvarun shenoyvvarun requested a review from slin1237 as a code owner April 23, 2026 19:42
@github-actions github-actions Bot added documentation Documentation changes oep OME Enhancement Proposal labels Apr 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Documentation changes oep OME Enhancement Proposal

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant