Able to set the toRender parameters dynamically

[kerthcet]

What would you like to be added:

Here's an example from Triton_RTLLM with lws, https://github.com/triton-inference-server/tutorials/blob/main/Deployment/Kubernetes/EKS_Multinode_Triton_TRTLLM/multinode_helm_chart/chart/templates/deployment.yaml,
it needs to set a bunch of parameters dynamically, see

          - python3
          - ./server.py
          - leader
          - --triton_model_repo_dir={{ $.Values.triton.triton_model_repo_path }}
          - --namespace={{ $.Release.Namespace }}
          - --pp={{ $.Values.tensorrtLLM.parallelism.pipeline }}
          - --tp={{ $.Values.tensorrtLLM.parallelism.tensor }}
          - --gpu_per_node={{ $.Values.gpuPerNode }}
          - --stateful_set_group_key=$(GROUP_KEY)

We should support this, basically, we can set the params in the model.spec.inferenceFlavors[x].params, with a prefix like Params_GPU_PER_NODE, when rending, we'll cut the Params_.

Why is this needed:

Completion requirements:

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

[googs1025]

  inferenceConfig:
    flavors:
      - name: a10 # gpu type
        params:
          Params_GPU_PER_NODE: 4
        limits:
          nvidia.com/gpu: 1

Could you explain more? I looked at the code and it seems that Param has been defined. here:

llmaz/api/core/v1alpha1/model_types.go

Line 124 in 14dde96

Params map[string]string `json:"params,omitempty"`

And Param is the parameter used in autoscaling. Do we need to modify the definition or other?

[kerthcet]

What I mean is right now, the render parameters is static, like the ModelName, ModelPath which is indispensable, see

llmaz/pkg/controller_helper/backendruntime/backendruntime.go

Lines 68 to 69 in 14dde96

	"ModelPath": source.ModelPath(),
	"ModelName": source.ModelName(),

.

However, some inference engines may need more parameters, like gpu_per_node here, so we may need a more dynamic way to recognize which parameters we should render, for example, it a parameter has Param_ in the head, we should render it. I'm still not sure how to implement this yet, because flavor.params seems not a proper place.

We can pend this until we have real case or the day we support Triton_RT.