Extend ServingRuntime API for Accelerator Support

[pallasathena92]

What type of PR is this?

/kind feature
/kind design

What this PR does / why we need it:

Extend the ServingRuntime and ClusterServingRuntime APIs to include accelerator requirements and configurations, enabling runtimes to specify which AcceleratorClasses they support.

ServingRuntimes need to declare their accelerator requirements so the runtime selection logic can match them with available AcceleratorClasses. This involves adding new fields to the existing ServingRuntimeSpec structure.

Special notes for your reviewer:

1. The json:"metadata" have be reverted back to json:"metadata,omitempty" to maintain compliance with Kubernetes API rules for streaming list types. The defaulter-gen annotation removal is acceptable since these types don't require custom defaulting behavior.

Does this PR introduce a user-facing change?

It introduced some new parameters in servingRuntimeSpec and SupportedModelFormat. All these parameters with optional marker, it won't impact current customer behavior.

[gemini-code-assist]

Summary of Changes

Hello @pallasathena92, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly extends the ServingRuntime and ClusterServingRuntime APIs to integrate comprehensive accelerator support. The core purpose is to enable machine learning runtimes to explicitly declare their hardware acceleration capabilities and requirements, facilitating more intelligent and efficient scheduling and resource allocation for inference services. This enhancement allows for fine-grained control over how models utilize accelerators, including specifying memory needs, parallelism configurations, and hardware features, without impacting existing customer behavior due to the optional nature of the new parameters.

Highlights

• Enhanced ServingRuntime API with Accelerator Requirements: The ServingRuntimeSpec now includes an AcceleratorRequirements field, allowing runtimes to declare which AcceleratorClasses they support, along with minimum memory, compute capability, required features, and preferred precisions.
• Granular Accelerator Configuration for Model Formats: The SupportedModelFormat now supports AcceleratorConfig, a map that allows specifying accelerator-specific overrides for model formats, such as minimum memory per billion parameters, tensor parallelism settings, runtime arguments, and environment variables.
• Introduction of New Accelerator-Specific Data Structures: New types like AcceleratorModelConfig and TensorParallelismConfig have been introduced to encapsulate detailed accelerator-specific settings for models and parallelism strategies.
• API Compliance Adjustments for InferenceServiceList: The InferenceServiceList definition has been updated to remove the +listType=set annotation, aligning with Kubernetes API rules for streaming list types and reverting json:"metadata" to json:"metadata,omitempty".

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request extends the ServingRuntime and related APIs to support accelerator-aware runtime selection. The changes introduce new structs like AcceleratorRequirements, AcceleratorModelConfig, and TensorParallelismConfig to specify accelerator capabilities and overrides. The addition of AcceleratorClassList and the registration with the scheme builder are correct. The removal of +listType=set from InferenceServiceList aligns with Kubernetes API best practices for list types.

My main feedback concerns the implementation of the GetAcceleratorConfig method, which currently returns a pointer to a copy of a map value, a pattern that can lead to subtle bugs. I've provided suggestions to refactor this to be more idiomatic and robust. The rest of the changes look good and are well-aligned with the goal of the PR.

[gemini-code-assist]

Storing structs by value in a map can lead to subtle issues, especially with helper methods that return pointers. The GetAcceleratorConfig method currently returns a pointer to a copy of the map value, not a pointer to the value in the map itself. This means any modifications made by the caller to the returned object will be lost.

To fix this and make the intent clear, I recommend storing pointers in the map. This avoids unnecessary copying and allows modifications to the stored objects if needed.

After this change, you'll also need to update GetAcceleratorConfig and regenerate the deepcopy and openapi files.

Suggested change

	AcceleratorConfig map[string]AcceleratorModelConfig `json:"acceleratorConfig,omitempty"`
	AcceleratorConfig map[string]*AcceleratorModelConfig `json:"acceleratorConfig,omitempty"`

[gemini-code-assist]

This function returns a pointer to a copy of the map value, which is misleading and bug-prone. If AcceleratorConfig stores pointers as suggested in my other comment, this function can be simplified to directly return the pointer from the map. This is more efficient and less error-prone.

Accessing a non-existent key in a map of pointers will correctly return nil, which is the desired behavior here.

func (f *SupportedModelFormat) GetAcceleratorConfig(acceleratorClass string) *AcceleratorModelConfig {
	if f.AcceleratorConfig == nil {
		return nil
	}
	return f.AcceleratorConfig[acceleratorClass]
}