external.metrics API returns empty resource list when no scaledObject or scaledJob defined using external.metrics #1797
Labels
bug
Something isn't working
Comments
|
Thanks, although it is not clear to me what should be returned in this specific case? |
|
Can we compare to metrics-server when there are no pods? |
|
Any update? The workaround is to create a dummy object? I have the same issue when using with Kyverno. Same as : kyverno/kyverno#2267 |
|
We would need to know what to return instead of empty list. @R1V3N do you think you can investigate and compare to k8s metrics-server as @coderanger suggested? |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions. |
|
This issue has been automatically closed due to inactivity. |
|
This is still an issue, haven't managed to research what the k8s metric server returns. Will try and research and post back here. |
|
confirmed, the issue is still present. kyverno v2.3.3, keda v2.6.2. And some dummy Prometheus scaled object is fixing this as a workaround. |
Closed
2 tasks
|
@zroubalik, @JorTurFer I think we shoud return like this below. when execute "kubectl api-resources", it return "configmaps" for example，not return every configmap object name. It return "configmaps" for example，not return every configmap object name.But "external.metrics.k8s.io/v1beta1" return every object now. |
This is interesting @penghuazhou , you are right and we are exposing all the metrics there, let me go depeer in this |
|
@penghuazhou could you please open a PR with a proposed solution from your case? We can try to test from your solution from that branch. |
|
@zroubalik I have commit a PR: #3825. |
|
I can confirm we are also using Keda, 2.7.1, and have this exact same issue as well on ALL of our 190+ EKS clusters ranging from 1.21 to 1.23. |
|
Upstream issue: kubernetes-sigs/custom-metrics-apiserver#146 |
|
I close this as the upstream has solved the issue. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Report
The Keda 2 external.metrics api returns an empty resources list, if no scalers using external.metrics are defined. The Go kubernetes client will error on an empty resource list https://github.com/kubernetes/client-go/blob/release-11.0/discovery/cached/memory/memcache.go#L215 .
In our specific case Flux CD is using the Go client and logs the following error if we have no ScaledObjects or ScaledJobs defined using external metrics:
We defined a dummy Prometheus scaledObject and the error log went away.
Confirmed an available apiservice
Then confirmed the empty resource list
Once dummy Prometheus scaled object was created, resource list was non empty
Expected Behavior
apiservice should not return an empty resource list for external.metrics as this will cause the K8s Go client to error.
Actual Behavior
Until a ScaledObject or ScaledJob is defined using external.metric the apiservice returns an empty resources list
Steps to Reproduce the Problem
Logs from KEDA operator
No response
KEDA Version
No response
Kubernetes Version
No response
Platform
No response
Scaler Details
No response
Anything else?
I originally asked about this in #1698, and asked to file an issue. Finally circling back to file an actual issue.
The text was updated successfully, but these errors were encountered: