Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Couldn't get resource list for external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1 #4154

Closed
besha100 opened this issue Jan 23, 2023 · 5 comments
Closed

Couldn't get resource list for external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1 #4154

besha100 opened this issue Jan 23, 2023 · 5 comments

Comments

@besha100
Copy link

I started to get this error on every Helm operation after upgrading to latest keda helm chart 2.9.3
"couldn't get resource list for external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1".

I'm using using latest helm 3.11

I checked the api services and all checks passed
kubectl describe apiservices v1beta1.external.metrics.k8s.io
Conditions:
Last Transition Time: 2023-01-23T10:01:47Z
Message: all checks passed
Reason: Passed
Status: True
Type: Available
Events:

@JorTurFer
Copy link
Member

What is your Kubernetes version? Could you share KEDA logs?

@besha100
Copy link
Author

besha100 commented Jan 23, 2023 •
edited by JorTurFer
Loading

kubernetes version v1.24.3
keda-operator pod log

2023-01-23T17:58:01Z    INFO    controller-runtime.metrics      Metrics server is starting to listen    {"addr": ":8080"}
2023-01-23T17:58:01Z    INFO    setup   Starting manager
2023-01-23T17:58:01Z    INFO    setup   KEDA Version: 2.9.2
2023-01-23T17:58:01Z    INFO    setup   Git Commit: 9bc3f66578a08cdfe084468ea3ef998fa6bf3bb0
2023-01-23T17:58:01Z    INFO    setup   Go Version: go1.18.8
2023-01-23T17:58:01Z    INFO    setup   Go OS/Arch: linux/amd64
2023-01-23T17:58:01Z    INFO    setup   Running on Kubernetes 1.24      {"version": "v1.24.3"}
2023-01-23T17:58:01Z    INFO    Starting server {"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
2023-01-23T17:58:01Z    INFO    Starting server {"kind": "health probe", "addr": "[::]:8081"}
I0123 17:58:01.932262       1 leaderelection.go:248] attempting to acquire leader lease monitoring/operator.keda.sh...
I0123 17:58:17.450522       1 leaderelection.go:258] successfully acquired lease monitoring/operator.keda.sh
2023-01-23T17:58:17Z    INFO    grpc_server     Starting Metrics Service gRPC Server    {"address": ":9666"}
2023-01-23T17:58:17Z    INFO    Starting EventSource    {"controller": "triggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "TriggerAuthentication", "source": "kind source: *v1alpha1.TriggerAuthentication"}
2023-01-23T17:58:17Z    INFO    Starting EventSource    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "source": "kind source: *v1alpha1.ScaledObject"}
2023-01-23T17:58:17Z    INFO    Starting Controller     {"controller": "triggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "TriggerAuthentication"}
2023-01-23T17:58:17Z    INFO    Starting EventSource    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "source": "kind source: *v2.HorizontalPodAutoscaler"}        
2023-01-23T17:58:17Z    INFO    Starting Controller     {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject"}
2023-01-23T17:58:17Z    INFO    Starting EventSource    {"controller": "clustertriggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "ClusterTriggerAuthentication", "source": "kind source: *v1alpha1.ClusterTriggerAuthentication"}
2023-01-23T17:58:17Z    INFO    Starting Controller     {"controller": "clustertriggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "ClusterTriggerAuthentication"}
2023-01-23T17:58:17Z    INFO    Starting EventSource    {"controller": "scaledjob", "controllerGroup": "keda.sh", "controllerKind": "ScaledJob", "source": "kind source: *v1alpha1.ScaledJob"}
2023-01-23T17:58:17Z    INFO    Starting Controller     {"controller": "scaledjob", "controllerGroup": "keda.sh", "controllerKind": "ScaledJob"}
2023-01-23T17:58:17Z    INFO    Starting workers        {"controller": "scaledjob", "controllerGroup": "keda.sh", "controllerKind": "ScaledJob", "worker count": 1}
2023-01-23T17:58:17Z    INFO    Starting workers        {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "worker count": 5}
2023-01-23T17:58:17Z    INFO    Starting workers        {"controller": "triggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "TriggerAuthentication", "worker count": 1}
2023-01-23T17:58:17Z    INFO    Starting workers        {"controller": "clustertriggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "ClusterTriggerAuthentication", "worker count": 1}

keda-operator-metrics-apiserver pod log

I0123 17:57:57.565915       1 main.go:214] keda_metrics_adapter "msg"="KEDA Version: 2.9.2" 
I0123 17:57:57.565968       1 main.go:215] keda_metrics_adapter "msg"="KEDA Commit: 9bc3f66578a08cdfe084468ea3ef998fa6bf3bb0"
I0123 17:57:57.565974       1 main.go:216] keda_metrics_adapter "msg"="Go Version: go1.18.8"
I0123 17:57:57.565980       1 main.go:217] keda_metrics_adapter "msg"="Go OS/Arch: linux/amd64"
I0123 17:58:00.743958       1 request.go:682] Waited for 1.04660934s due to client-side throttling, not priority and fairness, request: GET:https://10.0.0.1:443/apis/autoscaling/v2beta2?timeout=32s
I0123 17:58:02.147512       1 listener.go:44] keda_metrics_adapter/controller-runtime/metrics "msg"="Metrics server is starting to listen" "addr"=":8080"
I0123 17:58:02.147990       1 main.go:162] keda_metrics_adapter "msg"="Connecting Metrics Service gRPC client to the server" "address"="keda-operator.monitoring.svc.cluster.local:9666"
2023/01/23 17:58:02 Starting metrics server at :9022
I0123 17:58:02.148375       1 internal.go:366] keda_metrics_adapter "msg"="Starting server" "addr"={"IP":"::","Port":8080,"Zone":""} "kind"="metrics" "path"="/metrics"
I0123 17:58:02.148489       1 controller.go:185] keda_metrics_adapter "msg"="Starting EventSource" "controller"="scaledobject" "controllerGroup"="keda.sh" "controllerKind"="ScaledObject" "source"="kind source: *v1alpha1.ScaledObject"
I0123 17:58:02.148514       1 controller.go:185] keda_metrics_adapter "msg"="Starting EventSource" "controller"="scaledobject" "controllerGroup"="keda.sh" "controllerKind"="ScaledObject" "source"="kind source: *v1alpha1.ScaledObject"
I0123 17:58:02.148529       1 controller.go:193] keda_metrics_adapter "msg"="Starting Controller" "controller"="scaledobject" "controllerGroup"="keda.sh" "controllerKind"="ScaledObject"
I0123 17:58:02.249245       1 provider.go:74] keda_metrics_adapter/provider "msg"="starting"
I0123 17:58:02.249269       1 main.go:282] keda_metrics_adapter "msg"="starting adapter..."
I0123 17:58:02.449518       1 controller.go:227] keda_metrics_adapter "msg"="Starting workers" "controller"="scaledobject" "controllerGroup"="keda.sh" "controllerKind"="ScaledObject" "worker count"=1
I0123 17:58:02.470440       1 serving.go:342] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
I0123 17:58:02.987229       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0123 17:58:02.987251       1 shared_informer.go:255] Waiting for caches to sync for RequestHeaderAuthRequestController
I0123 17:58:02.987259       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I0123 17:58:02.987279       1 shared_informer.go:255] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0123 17:58:02.987288       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I0123 17:58:02.987300       1 shared_informer.go:255] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0123 17:58:02.987705       1 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::apiserver.local.config/certificates/apiserver.crt::apiserver.local.config/certificates/apiserver.key"       
I0123 17:58:02.988414       1 secure_serving.go:210] Serving securely on [::]:6443
I0123 17:58:02.988578       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0123 17:58:03.087397       1 shared_informer.go:262] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0123 17:58:03.087427       1 shared_informer.go:262] Caches are synced for RequestHeaderAuthRequestController
I0123 17:58:03.087445       1 shared_informer.go:262] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
W0123 17:58:03.175393       1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "keda-operator.monitoring.svc.cluster.local:9666",
  "ServerName": "keda-operator.monitoring.svc.cluster.local:9666",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 10.0.242.61:9666: connect: connection refused"

@JorTurFer
Copy link
Member

JorTurFer commented Jan 23, 2023 •
edited
Loading

oh, wait...
Could this be related with #4008 ?
I mean, you see that issue on every helm action, not in KEDA itself, right? Something has changed in kubectl v1.26 and if you are using latest version of kubectl (helm uses it under the hood), you can face with this issue. We are still investigating it
AFAIK, this is only a warning but everything works

@JorTurFer
Copy link
Member

Upstream issue: kubernetes-sigs/custom-metrics-apiserver#146

@besha100
Copy link
Author

@JorTurFer yes exactly the same issue! So we can close this ticket as duplicate!
Thank you:)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
Roadmap - KEDA Core
Status: Proposed
Milestone
No milestone
Development

No branches or pull requests
2 participants
@besha100 @JorTurFer