[misc] fix external service being cleaned up and kimi runtime update

[slin1237]

What type of PR is this?

/kind bug
/kind cleanup

What this PR does / why we need it:

• fix external service being cleaned up
• update latest kimi runtime

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?

[gemini-code-assist]

Summary of Changes

Hello @slin1237, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request primarily focuses on enhancing the stability and configuration of the SGLang runtime within the system. It introduces critical updates to the kimi-k2-pd-rt runtime, including revised environment variables, volume mounts, and security contexts, alongside adjustments to resource limits and command-line arguments for improved performance and compatibility. Concurrently, it refines the controller's cleanup mechanism to ensure that external services are not inadvertently removed when ingress creation is explicitly disabled, thereby preventing service disruptions. The router image and its resource allocations have also been updated, and a new utility pod for SGLang benchmarking has been added.

Highlights

• Runtime Configuration Updates: The kimi-k2-pd-rt.yaml runtime configuration has been significantly updated. This includes adding necessary volume mounts (/dev/shm, /dev/infiniband, SGLang recorder/profiler directories) and security contexts (IPC_LOCK, CAP_SYS_ADMIN, privileged mode) for both leader and worker components. Various SGLang environment variables have been adjusted (some removed, new ones added for metrics, unbuffered Python, and expert distribution). Command-line arguments for the SGLang server have also been updated, including changing the InfiniBand device, adjusting CUDA graph batch sizes, and increasing max running requests.
• Router Image and Resource Updates: The sgl-router image has been updated to dev for both deepseek-rdma-pd-rt.yaml and kimi-k2-pd-rt.yaml runtimes. Additionally, the router's CPU and memory limits in kimi-k2-pd-rt.yaml have been significantly increased (CPU from 1 to 10, memory from 2Gi to 20Gi), and its policy changed from cache_aware to random. New arguments for max-payload-size and worker-startup-timeout-secs were also added.
• External Service Cleanup Logic Enhancement: The controller's cleanup logic (pkg/controller/v1beta1/inferenceservice/cleanup.go) has been modified to prevent the premature deletion of external services. If an InferenceService has the ome.io/ingress-disable-creation: "true" annotation and there are active traffic-serving components (router, engine, or predictor), the external service will now be preserved. A new annotation ome.io/ingress-disable-creation: "true" was also added to a sample InferenceService (llama-4-maverick-17b-128e-instruct-fp8.yaml) to demonstrate this.
• New SGLang Benchmark Pod: A new Kubernetes Pod definition (sgl-benchmark-pod.yaml) has been added. This pod uses the sglang:dev image and mounts a shared models directory, providing a utility for SGLang benchmarking or development.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request updates runtime configurations for Kimi and Deepseek, and fixes an issue with external service cleanup. My review focuses on several critical and high-severity issues in the YAML configurations:

• The use of mutable dev image tags, which impacts reproducibility.
• The removal of livenessProbes, which is critical for service reliability.
• The introduction of privileged: true security contexts, which poses a significant security risk.

I've also included suggestions for improving the Go code's maintainability by using constants for annotation keys, and a minor fix for the new benchmark pod YAML.

Please address the critical issues before merging.