Metrics server not working on 4.7.0

[zamu87]

• I have tried with the latest version of Docker Desktop
• I have tried disabling enabled experimental features
• I have uploaded Diagnostics
• Diagnostics ID:

Expected behavior

Metrics server running

Actual behavior

Metrics server cannot start and logs the following error:

E0411 13:39:34.578401       1 scraper.go:140] "Failed to scrape node" err="Get \"https://192.168.65.4:10250/metrics/resource\": context deadline exceeded" node="docker-desktop"
I0411 13:39:40.241772       1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve"

Information

Metrics server works correctly on 4.6.1 and on 4.7.0 preview (76176)

• macOS Version: 12.3.1
• Intel chip or Apple chip: M1 Pro
• Docker Desktop Version: 4.7.0 (77141)

Output of /Applications/Docker.app/Contents/MacOS/com.docker.diagnose check

Starting diagnostics

[PASS] DD0027: is there available disk space on the host?
[PASS] DD0028: is there available VM disk space?
[PASS] DD0031: does the Docker API work?
[PASS] DD0004: is the Docker engine running?
[PASS] DD0011: are the LinuxKit services running?
[PASS] DD0016: is the LinuxKit VM running?
[PASS] DD0001: is the application running?
[PASS] DD0018: does the host support virtualization?
[PASS] DD0017: can a VM be started?
[PASS] DD0015: are the binary symlinks installed?
[PASS] DD0003: is the Docker CLI working?
[PASS] DD0013: is the $PATH ok?
[PASS] DD0007: is the backend responding?
[PASS] DD0014: are the backend processes running?
[PASS] DD0008: is the native API responding?
[PASS] DD0009: is the vpnkit API responding?
[PASS] DD0010: is the Docker API proxy responding?
[PASS] DD0012: is the VM networking working?
[PASS] DD0032: do Docker networks overlap with host IPs?
[SKIP] DD0030: is the image access management authorized?
[PASS] DD0019: is the com.docker.vmnetd process responding?
[PASS] DD0033: does the host have Internet access?
No fatal errors detected.

Steps to reproduce the behavior

1. Install Docker Desktop 4.7.0
2. Install metrics server kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
3. Add "--kubelet-insecure-tls" under the args section of metrics server deployment

[ncresswell]

i am seeing the same issue.. following..

[wang-mask]

i have the same problem

[antoinedray]

same here

[raulpuente]

same here

[wang-mask]

I have solved this problem, you can try to set the --metric-resolution flag of metrics-server to 30s or longer.
To see the detail kubernetes-sigs/metrics-server#907

[VtG242]

In general scraping of kubelet /metrics takes more than 50 seconds using kubernetes in docker desktop on Apple M1.
In my case really --metric-resolution=90s solved the problem but the time is terrible.

[alanbrent]

I am having the same issue. So far I can get it to work if I increase the scrape interval all the way to 120s, but that's a workaround rather than a fix.

Can anyone from the Docker team(s) responsible for this product speak to way the kubelet takes so long to respond to the metrics endpoint?

[eldada]

I tried the suggested increase of resolution to --metric-resolution=120s and still fails.
The ...context deadline exceeded is gone, but I still get

I0602 07:25:09.859415       1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve"

[vkeeps]

same here，and how to solve it?

[gsy567]

same here，any update?

[bconfiden2]

@eldada In my case with --metric-resolution=90s, the metrics server worked after waiting 90 seconds.
It might waits for scraping the metrics

I0617 12:27:19.023258       1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve"
I0617 12:27:29.015066       1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve"
I0617 12:27:39.021942       1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve"
I0617 12:27:49.021978       1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve"
I0617 12:27:59.025582       1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve"
I0617 12:28:09.022221       1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve"
I0617 12:28:19.025656       1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve"
I0617 12:28:29.026233       1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve"
I0617 12:28:39.026624       1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve"
I0617 12:28:49.027680       1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve"

[phxism]

same here on 4.9.1
I0618 02:40:32.115542 1 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController I0618 02:40:32.115590 1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0618 02:40:59.510857 1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve" I0618 02:41:09.509626 1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve" I0618 02:41:19.510075 1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve" I0618 02:41:29.510229 1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve" I0618 02:41:39.510356 1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve" I0618 02:41:49.510741 1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve" I0618 02:41:52.173745 1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve" I0618 02:41:59.512561 1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve" I0618 02:42:09.512245 1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve" I0618 02:42:19.510408 1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve" I0618 02:42:29.510170 1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve" I0618 02:42:39.510117 1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve" I0618 02:42:49.511055 1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve" I0618 02:42:59.510729 1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve" I0618 02:43:09.511510 1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve" I0618 02:43:18.174487 1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve" I0618 02:43:19.512251 1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve"

[appellod]

I'm having the same issue on Docker Desktop for Windows v4.9.1. I'll open another issue in that repository, but wanted to chime in that it doesn't seem platform specific. I'm going to try downgrading to 4.6.1 in the meantime.

Edit: Metrics server is working on Docker Desktop for Windows v4.6.1.

[danielsollondon]

same here