[Feature] [OIM] Make inference backend configurable

[Yu-amd]

Priority

P2-High

OS type

Ubuntu

Hardware type

GPU-AMD

Running nodes

Single Node

Description

Based on discussion with Zhiwei at Intel, I understand there are plans to create OIM for selecting inference backend based on the models.

I'm opening this feature request to add the requirement that OIM should allow user to configure and override the backend used. The reason is that not all inference backends will be available on all hardware and software platforms due to various reasons, so we want to ensure the user can manually configure the actual inference backend for the OPEA workload.

[poussa]

@Yu-amd thanks for the feature request. Just to clarify, by backend you mean the inference server such as tgi, vllm and others.

[mkbhanda]

Configurable at creation time?
Would this not need Helm charts and docker compose files to be created on the fly?

Or configurable at run time, so one call with vLLM and the next to the same pipeline endpoint with say TGI? Dynamic?
GMC had a switch statement for dynamic routing.

[Yu-amd]

@Yu-amd thanks for the feature request. Just to clarify, by backend you mean the inference server such as tgi, vllm and others.

Hi Poussa,

Sorry for the late response.

@Yu-amd thanks for the feature request. Just to clarify, by backend you mean the inference server such as tgi, vllm and others.

That is correct.

Thank you,
Yu

[Yu-amd]

Configurable at creation time? Would this not need Helm charts and docker compose files to be created on the fly?

Or configurable at run time, so one call with vLLM and the next to the same pipeline endpoint with say TGI? Dynamic? GMC had a switch statement for dynamic routing.

Hi Malini,

The feature request is based on the assumption that OIM will "pick" the best performing profile for the hardware detected at runtime. I'm requesting that OIM will allow the deployer to override the inference server regardless of what OIM considers is the most performant.

A real world example:

NVIDIA NIM Operator chooses TRTLLM as the inference server because it's optimized for NVIDIA hardware. For TRTLLM to work, it requires the UVM to be enabled. When UVM is enabled, vMotion is disabled. Any workload in a VM running with the TRTLLM inference server would not be vMotion-able. This is an issue for infra admins who need to migrate workloads to lifecycle the h/w and s/w stack in the data center. If the NIM operator allows the inference server to be configurable, then these workloads would be able to use a less performant inference server, for example, vLLM, and all these workloads would be able to vMotion.

Hope that helps.

Yu

[mkbhanda]

Thank you @Yu-amd ! The vMotion example is compelling. We could set the engine via a command line argument or environment variable and default to vLLM. @PeterYang12 and @poussa - please facilitate in design.

[yinghu5]

Hi @Yu-amd @mkbhanda @poussa @PeterYang12 , thank you a lot for addressing the issue. [RFC] OPEA Inference Microservices (OIM) opea-project/docs#327 merged, Just remind for next step