Couldn't get resource list for metrics error #11772

wyardley · 2023-01-26T23:04:33Z

With the latest version of helm, I'm getting warnings like this when running operations. I'm wondering if it has to do with deprecated API versions, or if it's something else.

% helm upgrade --dry-run -i xxx somedir -f helm-charts/values-default.yaml
E0126 14:24:31.061339    6338 memcache.go:255] couldn't get resource list for external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1
E0126 14:24:31.366546    6338 memcache.go:255] couldn't get resource list for external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1
E0126 14:24:31.493404    6338 memcache.go:255] couldn't get resource list for external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1
E0126 14:24:31.698458    6338 memcache.go:255] couldn't get resource list for external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1
E0126 14:24:31.980491    6338 memcache.go:255] couldn't get resource list for external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1
E0126 14:24:32.227059    6338 memcache.go:255] couldn't get resource list for external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1
E0126 14:24:32.369217    6338 memcache.go:255] couldn't get resource list for external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1
E0126 14:24:32.477016    6338 memcache.go:255] couldn't get resource list for external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1
E0126 14:24:32.605685    6338 memcache.go:255] couldn't get resource list for external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1
E0126 14:24:32.789270    6338 memcache.go:255] couldn't get resource list for external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1
E0126 14:24:33.063156    6338 memcache.go:255] couldn't get resource list for external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1
E0126 14:24:33.291941    6338 memcache.go:255] couldn't get resource list for external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1
E0126 14:24:33.426387    6338 memcache.go:255] couldn't get resource list for external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1
[....]

(The command still succeeds, but it's messy / unsightly)

Output of helm version: version.BuildInfo{Version:"v3.11.0", GitCommit:"472c5736ab01133de504a826bd9ee12cbe4e7904", GitTreeState:"clean", GoVersion:"go1.19.5"}

Output of kubectl version:

Client Version: v1.25.2
Kustomize Version: v4.5.7
Server Version: v1.25.4-gke.2100

Cloud Provider/Platform (AKS, GKE, Minikube etc.): GKE

The text was updated successfully, but these errors were encountered:

joejulian · 2023-01-27T01:20:16Z

I started getting that with kubectl commands, too, with kubernetes 1.25.6. Do you get that with kubectl?

wyardley · 2023-01-27T01:22:45Z

No - I get this from helm 3.11 in CI and locally (I did not with helm 3.7; haven't tested different versions).
I do not see that with any kubectl commands (e.g., kubectl get pods), at least with kubectl v 1.25 (see version output above); if there's a specific command I should try, let me know.

wyardley · 2023-01-27T01:27:38Z

Also, here's the output of helm with --debug, though don't think that helps?:

upgrade.go:142: [debug] preparing upgrade for xx
E0126 17:26:46.472558   29623 memcache.go:255] couldn't get resource list for external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1

gothrek22 · 2023-01-27T09:42:22Z

EKS 1.24, kubectl 1.25.4, getting the same error.

totogtr · 2023-01-27T10:08:28Z

Same error, EKS 1.24, kubectl 1.24.4

CR-volodymyrroshko · 2023-01-27T10:19:59Z

Same issue, GKE 1.22.15-gke.1000, kubectl 1.24.0

DmitryMihailov · 2023-01-27T15:16:42Z

Same error, Kubectl: 1.26.1

gjenkins8 · 2023-01-27T17:08:56Z

Two notes:

memcache.go is part of kubernetes/client-go -- https://github.com/kubernetes/client-go/blob/master/discovery/cached/memory/memcache.go#L255
external.metrics.k8s.io isn't part of kubernetes upstream: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-metrics-apis

I don't think (as @joejulian mentions) this is a helm issue. If I had to guess, a new version of a common metrics provider has removed its external.metrics.k8s.io/v1beta1 api.

wyardley · 2023-01-27T17:14:04Z

I don't think (as @joejulian mentions) this is a helm issue

I get the error when running helm; are you saying that helm is shelling out to a third party program, or just that this error is coming from a library that helm includes? It does appear like helm imports k8s.io/client-go in multiple places....

joejulian · 2023-01-28T00:06:15Z

Yes, using the k8s.io/client-go library for interfacing with the kubernetes API is the standard way of doing it.

I was just asking the question to see if I could find any commonality between what you're seeing with helm and what I'm seeing with kubectl. They both use client-go.

CR-volodymyrroshko · 2023-01-30T14:13:58Z

Issue fixed.
In my prevoius configuration for deploy application i used helm v3.10.3. My current helm version was updated to v3.11.0
It is cause why i am gettings the issue. When i rollback my client to previous version v3.10.3 - issue fixed.

MBingazy · 2023-02-02T11:14:08Z

I am running into a likewise issue, the helm version is v3.11.0 and the installation process was completed successfully after throwing these warnings.

E0202 12:51:02.217710 99829 memcache.go:255] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized
E0202 12:51:03.197970 99829 memcache.go:255] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized
E0202 12:51:04.851745 99829 memcache.go:255] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized
E0202 12:51:06.071295 99829 memcache.go:255] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized
E0202 12:51:09.482704 99829 memcache.go:255] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized
E0202 12:51:10.459653 99829 memcache.go:255] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized
E0202 12:51:12.063372 99829 memcache.go:255] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized
E0202 12:51:13.374725 99829 memcache.go:255] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized
E0202 12:51:14.701756 99829 memcache.go:255] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized
E0202 12:51:16.460971 99829 memcache.go:255] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized
E0202 12:51:17.431306 99829 memcache.go:255] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized
E0202 12:51:18.396043 99829 memcache.go:255] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized
E0202 12:51:19.380456 99829 memcache.go:255] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized
E0202 12:51:20.363210 99829 memcache.go:255] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized
E0202 12:51:21.333310 99829 memcache.go:255] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized
E0202 12:51:22.318098 99829 memcache.go:255] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized

Will this cause any problems in the future, so I have to roll back to helm version v3.10.3 and re-install?

gillesdouaire · 2023-02-02T14:46:36Z

We are running in the same issues - a lot of warnings, no real impact in the end.

Is there a context where these warnings are meaningful?

Is there a way to keep using 3.11.0 but not get these warnings?

Down-pining will solve the issue, but is there anything less temporary we can do?

sandy-wang · 2023-02-03T01:23:35Z

Those errors are from Stderr, if you look closely, the helm chart did get installed successfully. You can set to ignore Stderr on your pipeline as a workaround.
For example: on DevOps pipeline, if you have failOnStderr: true, then remove it or change it to false (by default it's false.)

wyardley · 2023-02-03T01:31:52Z

Ignoring stderr doesn’t seem like a good fix.
It would prevent one from seeing important and / or fatal errors as well as this noise.

Presumably, either helm should downgrade the version of that lib, look for an upstream fix of that lib, or swallow the errors itself in a more limited way.

wyardley · 2023-02-03T01:48:44Z

seems like this is probably the change that introduced it:
#11622

I posted an issue upstream at:
kubernetes/client-go#1223

sandy-wang · 2023-02-03T02:21:51Z

@wyardley I had the same conern "It would prevent one from seeing important and / or fatal errors as well as this noise" when other enginee proposed this workaround. Apparently, in DevOps pipeline fatal error will fail the step as by default failOnStderr is false.
Good to see there is a fix brought up.

wyardley · 2023-02-03T02:32:37Z

In almost any type of CI pipeline that's configured properly, the job will fail based on the process's exit code (success on exit code 0, failure on any other exit code), so suppressing stderr should not make it "succeed" on failure. However, suppressing all stderr would potentially make debugging a failure more challenging in some cases IMO.

joejulian · 2023-02-04T00:34:55Z

I finally had some time to dig in to this on my own cluster.

E0126 14:24:31.061339    6338 memcache.go:255] couldn't get resource list for external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1

is client-go trying to fill its cache. When an APIService is defined that no longer has a controller servicing it, those messages are displayed by client-go.

In my case, I upgraded prometheus-adapter which seems to have changed from custom.metrics.k8s.io to metrics.k8s.io. To fix this I just had to delete the unwanted api service:

kubectl delete apiservices v1beta1.custom.metrics.k8s.io

This is not a helm problem.

joejulian · 2023-02-04T00:40:31Z

WRT the argument over stderr vs stdout: it's very common to have stdout reserved for necessary output and unrelated data sent to stderr. If you're using ArgoCD, for example, and you had all those client-go errors cluttering up the yaml output, it would make it unusable. For this reason, it's sent to stderr. Messages on stderr are not necessarily errors and "any type of CI pipeline that's configured properly" will accept the exit code of the application as the indication of failure, not output from any terminal channel. We only have two and leaving stdout uncluttered will satisfy the needs of the majority.

sandy-wang · 2023-02-06T05:25:49Z

I finally had some time to dig in to this on my own cluster.
E0126 14:24:31.061339    6338 memcache.go:255] couldn't get resource list for external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1
is client-go trying to fill its cache. When an APIService is defined that no longer has a controller servicing it, those messages are displayed by client-go.

In my case, I upgraded prometheus-adapter which seems to have changed from custom.metrics.k8s.io to metrics.k8s.io. To fix this I just had to delete the unwanted api service:
kubectl delete apiservices v1beta1.custom.metrics.k8s.io
This is not a helm problem.

Strange enough, our AKS cluster still have the controller servicing the apiservice in place. Can you shine a light on why we are getting the error on helm upgrade?

➜ k get apiservice v1beta1.external.metrics.k8s.io -o yaml
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
annotations:
meta.helm.sh/release-name: keda
meta.helm.sh/release-namespace: keda
creationTimestamp: "2022-11-10T02:06:29Z"
labels:
app.kubernetes.io/component: operator
app.kubernetes.io/instance: keda
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: v1beta1.external.metrics.k8s.io
app.kubernetes.io/part-of: keda-operator
app.kubernetes.io/version: 2.6.1
helm.sh/chart: keda-2.6.2
name: v1beta1.external.metrics.k8s.io
resourceVersion: "40571550"
uid: 86acce8a-7c89-4b5d-a286-9602e6f0d9d6
spec:
group: external.metrics.k8s.io
groupPriorityMinimum: 100
insecureSkipTLSVerify: true
service:
name: keda-operator-metrics-apiserver
namespace: keda
port: 443
version: v1beta1
versionPriority: 100
status:
conditions:

lastTransitionTime: "2023-02-03T23:56:22Z"
message: all checks passed
reason: Passed
status: "True"
type: Available

➜ k get svc -n keda
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
keda-operator-metrics-apiserver ClusterIP 10.1.207.28 443/TCP,80/TCP 88d

➜ k get ep -n keda
NAME ENDPOINTS AGE
keda-operator-metrics-apiserver 10.2.32.11:6443,10.2.32.11:8080 88d

and the Pod
keda-operator-metrics-apiserver-66995cfc7b-78xm7 1/1 Running 0 2d23h 10.2.32.11

joejulian · 2023-02-06T18:29:09Z

No. Please check with the metrics provider.

sandy-wang · 2023-02-07T03:14:22Z

found it kubernetes-sigs/custom-metrics-apiserver#146

JorTurFer · 2023-02-11T12:52:49Z

Hello,
If you are affected by this issue and you are using KEDA, there is a workaround you can use: kedacore/keda#4224 (comment)

stevensshi · 2023-05-11T07:39:13Z

Re-install kubectl fixed the warning message for me. I first remove originally homebrew installed kubectl in WSL, which throws this warning in every call. And reinstall using apt-get kubuctl. installation guide.
memcache.go:255] couldn't get resource list for external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1

JorTurFer · 2023-05-11T08:07:01Z

The problems is that tooling needs to be update to fixed version of the kubernetes client. Probably the brew source isn't using latest versions...

In parallel, a fix has been released as part of custom-metrics-api service (the library that projects like prometheus-adapter or KEDA use), so future releases of metrics servers should work properly even though you use affected versions (for example, in KEDA we will solve it in next release)

mmorejon · 2023-05-18T10:04:43Z

In parallel, a fix has been released as part of custom-metrics-api service (the library that projects like prometheus-adapter or KEDA use), so future releases of metrics servers should work properly even though you use affected versions (for example, in KEDA we will solve it in next release)

Does Keda have an issue open to follow? @JorTurFer

JorTurFer · 2023-05-18T11:33:58Z

hi @mmorejon
This is the PR in the custom-metrics-apiserver (the upstream) and this is the PR in KEDA.
As I said, the next (KEDA) release will have this problem fixed independently of the tooling version that you are using 😄

joejulian added the question/support label Jan 27, 2023

joejulian closed this as completed Feb 4, 2023

This was referenced Feb 11, 2023

Discovery memCacheClient has inconsistent error behavior with other DiscoveryInterfaces kubernetes/kubernetes#111292

Closed

Empty response warning with latest kube-client (v0.26) kubernetes-sigs/custom-metrics-apiserver#146

Closed

vihangm mentioned this issue Mar 15, 2023

Explicitly install a controlled version of helm and kubectl newrelic/newrelic-k8s-metrics-adapter#105

Merged

edify42 mentioned this issue Mar 22, 2023

Deprecate external.metrics.k8s.io APIService creation DataDog/helm-charts#962

Closed

nocodebackend mentioned this issue Jul 4, 2023

"couldn't get resource list" error kubernetes/client-go#1223

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Couldn't get resource list for metrics error #11772

Couldn't get resource list for metrics error #11772

wyardley commented Jan 26, 2023 •

edited

joejulian commented Jan 27, 2023

wyardley commented Jan 27, 2023

wyardley commented Jan 27, 2023

gothrek22 commented Jan 27, 2023

totogtr commented Jan 27, 2023

CR-volodymyrroshko commented Jan 27, 2023

DmitryMihailov commented Jan 27, 2023

gjenkins8 commented Jan 27, 2023

wyardley commented Jan 27, 2023

joejulian commented Jan 28, 2023

CR-volodymyrroshko commented Jan 30, 2023 •

edited

MBingazy commented Feb 2, 2023

gillesdouaire commented Feb 2, 2023

sandy-wang commented Feb 3, 2023 •

edited

wyardley commented Feb 3, 2023

wyardley commented Feb 3, 2023

sandy-wang commented Feb 3, 2023

wyardley commented Feb 3, 2023

joejulian commented Feb 4, 2023

joejulian commented Feb 4, 2023

sandy-wang commented Feb 6, 2023

joejulian commented Feb 6, 2023

sandy-wang commented Feb 7, 2023

JorTurFer commented Feb 11, 2023

stevensshi commented May 11, 2023 •

edited

JorTurFer commented May 11, 2023 •

edited

mmorejon commented May 18, 2023

JorTurFer commented May 18, 2023 •

edited

Couldn't get resource list for metrics error #11772

Couldn't get resource list for metrics error #11772

Comments

wyardley commented Jan 26, 2023 • edited

joejulian commented Jan 27, 2023

wyardley commented Jan 27, 2023

wyardley commented Jan 27, 2023

gothrek22 commented Jan 27, 2023

totogtr commented Jan 27, 2023

CR-volodymyrroshko commented Jan 27, 2023

DmitryMihailov commented Jan 27, 2023

gjenkins8 commented Jan 27, 2023

wyardley commented Jan 27, 2023

joejulian commented Jan 28, 2023

CR-volodymyrroshko commented Jan 30, 2023 • edited

MBingazy commented Feb 2, 2023

gillesdouaire commented Feb 2, 2023

sandy-wang commented Feb 3, 2023 • edited

wyardley commented Feb 3, 2023

wyardley commented Feb 3, 2023

sandy-wang commented Feb 3, 2023

wyardley commented Feb 3, 2023

joejulian commented Feb 4, 2023

joejulian commented Feb 4, 2023

sandy-wang commented Feb 6, 2023

joejulian commented Feb 6, 2023

sandy-wang commented Feb 7, 2023

JorTurFer commented Feb 11, 2023

stevensshi commented May 11, 2023 • edited

JorTurFer commented May 11, 2023 • edited

mmorejon commented May 18, 2023

JorTurFer commented May 18, 2023 • edited

wyardley commented Jan 26, 2023 •

edited

CR-volodymyrroshko commented Jan 30, 2023 •

edited

sandy-wang commented Feb 3, 2023 •

edited

stevensshi commented May 11, 2023 •

edited

JorTurFer commented May 11, 2023 •

edited

JorTurFer commented May 18, 2023 •

edited