[Core]Integrate the accelerator‑aware runtime into the InferenceService con…

[pallasathena92]

What type of PR is this?

/kind feature
/kind design

What this PR does / why we need it:

Integrates accelerator-aware runtime selection into the InferenceService controller to automatically match workloads with appropriate GPU/accelerator hardware.

1. add acceleratorclassselector in pkg/acceleratorclassselector
   Selector logic : Implements priority-based accelerator class selection for inference components (engine/decoder)
   Priority: Component override → InferenceService selector → Runtime requirements
   Fetcher: Queries cluster-scoped AcceleratorClass resources from Kubernetes API
   Type definitions: Interfaces and data structures for accelerator selection
   Error handling : Custom error types for not-found and compatibility issues

2. Controller Integration:
   Added AcceleratorClassSelector to InferenceServiceReconciler
   Selects accelerator classes for engine and decoder components during reconciliation
   Passes accelerator specs to component builders

3. Component Enhancements and Spec Merging Logic:
   Base/Engine/Decoder: Extended to accept accelerator class specs and names
   New utilities to merge runtime and InferenceService specs
   Handles accelerator requirements, resource requests, and node selectors
   The main merging logic is:
   args: customer's args + runtime
   env: customer's env + runtime
   resource: choice the max number from customer's setting, runtime, AC
   nodeSelector: customer's + AC

4. Remove default value for AcceleratorSelectionPolicy
   the current main AC select logic is:
   Implements priority-based accelerator class selection for inference components (engine/decoder)
   if no AC specified, ac selection based on policy.
   here is a following improvment for policy selection.

Does this PR introduce a user-facing change?

None, all field is optional. If customer doesn't set it, they won't meet any different behavior.

[gemini-code-assist]

Summary of Changes

Hello @pallasathena92, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This PR introduces accelerator-aware runtime selection for InferenceServices, enabling automatic matching of workloads to GPUs. It adds a new AcceleratorClassSelector package, integrates its logic into the InferenceService controller, and enhances component builders to apply accelerator-specific configurations (node selectors, resources, environment variables, and command-line arguments). The selection prioritizes explicit overrides over policy-based defaults. Unit tests for the new selector and merging utilities are also included.

Highlights

• Overall Intent: This pull request integrates accelerator-aware runtime selection into the InferenceService controller. The goal is to automatically match inference workloads with appropriate GPU/accelerator hardware by introducing a new AcceleratorClassSelector and enhancing the reconciliation logic to consider accelerator requirements and policies.
• Key Changes: A new pkg/acceleratorclassselector package is introduced, containing logic for fetching AcceleratorClass resources, defining custom errors, and implementing a priority-based selection mechanism. This selection prioritizes component-specific overrides, then InferenceService-level selectors, and finally falls back to runtime requirements (currently defaulting to the first available accelerator class). The InferenceServiceReconciler is updated to use this new selector, passing chosen accelerator class specifications and names to the engine and decoder component builders. Component builders and their respective BaseComponentFields are extended to accept these accelerator details. New utility functions are added in pkg/controller/v1beta1/inferenceservice/utils/merging.go to merge node selectors, resource requests/limits (taking maximums), and container arguments (including handling multi-line arguments and overriding integer parameters like --tp-size and --pp-size) based on runtime and accelerator class specifications. The AcceleratorSelectionPolicy in the InferenceService API (pkg/apis/ome/v1beta1/inference_service.go) removes its default value, and the runtimeselector package is updated to expose CalculateFormatScore and GetSupportedModelFormat for better integration. Minor fixes include removing a caBundle placeholder in a webhook config and correcting a typo in OpenAPI descriptions.
• Reviewer Activity: No specific reviewer activity was provided in the context.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces significant new functionality for accelerator-aware runtime selection, which is a great addition. The code is generally well-structured, but I've identified several critical issues related to error handling and potential nil pointer dereferences that could cause the controller to panic. There are also some areas where code clarity and maintainability could be improved. Please review the detailed comments, especially the ones marked as critical and high severity.

[gemini-code-assist]

The AcceleratorCompatibilityError struct is defined but doesn't implement the error interface because it's missing an Error() method. This prevents it from being used as a standard Go error, which seems to be the intention. To ensure it can be properly handled as an error throughout the codebase, you should add an Error() method.

[gemini-code-assist]

The ResourceRequests field is defined as map[string]string. For Kubernetes resource management, it's standard and safer to use corev1.ResourceList, which is map[corev1.ResourceName]resource.Quantity. Using map[string]string bypasses the validation and parsing that resource.Quantity provides, making it prone to errors and inconsistencies. Please change the type to corev1.ResourceList and add the necessary imports (k8s.io/api/core/v1 and k8s.io/apimachinery/pkg/api/resource).