[BUG] metrics server not reporting metrics for hpa

[fragolinux]

What did you do

• How was the cluster created?

  • k3d cluster create test

• What did you do afterwards?

  • installed my pods, i get in events errors about metrics, in particular:
  • FailedComputeMetricsReplicas invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)
  • FailedGetResourceMetric failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)
  • or failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

What did you expect to happen

hpa getting metrics so it can do its job

Screenshots or terminal output

If applicable, add screenshots or terminal output (code block) to help explain your problem.

Which OS & Architecture

• output of k3d runtime-info

arch: x86_64
cgroupdriver: cgroupfs
cgroupversion: "2"
endpoint: /var/run/docker.sock
filesystem: extfs
name: docker
os: Docker Desktop
ostype: linux
version: 20.10.17

Which version of k3d

• output of k3d version

k3d version v5.4.4
k3s version v1.23.8-k3s1 (default)

Which version of docker

• output of docker version and docker info

Client:
Cloud integration: v1.0.24
Version: 20.10.17
API version: 1.41
Go version: go1.17.11
Git commit: 100c701
Built: Mon Jun 6 23:04:45 2022
OS/Arch: darwin/amd64
Context: default
Experimental: true

Server: Docker Desktop 4.10.1 (82475)
Engine:
Version: 20.10.17
API version: 1.41 (minimum version 1.12)
Go version: go1.17.11
Git commit: a89b842
Built: Mon Jun 6 23:01:23 2022
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.6
GitCommit: 10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
runc:
Version: 1.1.2
GitCommit: v1.1.2-0-ga916309
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Client:
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc., v0.8.2)
compose: Docker Compose (Docker Inc., v2.6.1)
extension: Manages Docker extensions (Docker Inc., v0.2.7)
sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc., 0.6.0)
scan: Docker Scan (Docker Inc., v0.17.0)

Server:
Containers: 44
Running: 9
Paused: 0
Stopped: 35
Images: 66
Server Version: 20.10.17
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
runc version: v1.1.2-0-ga916309
init version: de40ad0
Security Options:
seccomp
Profile: default
cgroupns
Kernel Version: 5.10.104-linuxkit
Operating System: Docker Desktop
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 2.921GiB
Name: docker-desktop
ID: IO7B:SU45:NMT4:WQV4:4SFS:NNEJ:SKLF:RA7I:YLWZ:JA7J:EN3U:TSS3
Docker Root Dir: /var/lib/docker
Debug Mode: false
HTTP Proxy: http.docker.internal:3128
HTTPS Proxy: http.docker.internal:3128
No Proxy: hubproxy.docker.internal
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
e4t.local:5555
hubproxy.docker.internal:5000
127.0.0.0/8
Live Restore Enabled: false

p.s. adding --kubelet-insecure-tls to metrics server deployment didn't help...