feat(ci): add HPA pod autoscaling validation to inference workflow#163
Merged
dims merged 1 commit intoNVIDIA:mainfrom Feb 20, 2026
Merged
feat(ci): add HPA pod autoscaling validation to inference workflow#163dims merged 1 commit intoNVIDIA:mainfrom
dims merged 1 commit intoNVIDIA:mainfrom
Conversation
dims
force-pushed
the
add-hpa-gpu-metrics-test
branch
from
9beb15f to
8fbe5ab
Compare
Validate the custom metrics pipeline (DCGM → Prometheus → prometheus-adapter → custom metrics API) that HPA consumes for GPU-aware pod autoscaling. Queries the custom.metrics.k8s.io API directly for gpu_utilization, gpu_memory_used, and gpu_power_usage metrics. DCGM exporter runs as a DaemonSet in gpu-operator namespace, so Prometheus labels GPU metrics with namespace=gpu-operator. We query that namespace to validate the full metrics pipeline. Dynamo uses PodCliqueSets (not Deployments), so we validate the metrics API availability rather than creating an HPA object. This covers CNCF AI Conformance requirement #8b (pod autoscaling). Signed-off-by: Davanum Srinivas <dsrinivas@nvidia.com>
dims
force-pushed
the
add-hpa-gpu-metrics-test
branch
from
8fbe5ab to
46148df
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
pod_autoscalinggpu_utilizationcustom metric from prometheus-adaptermaxReplicas=1to validate the metrics pipeline (DCGM exporter → Prometheus → prometheus-adapter → HPA) without triggering actual scalingTest plan
gpu_utilizationfrom prometheus-adapter (AbleToScale=True,currentMetricsnon-empty)