[Question] Unable to get metrics from metrics server (AKS 1.23.3)

[Kapsztajn]

Describe scenario
I'm currently unavailable to get any metrics from AKS. Cluster version 1.23.3
k top nodes give me
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get nodes.metrics.k8s.io)
Pods from metrics server are running fine, I even restarted it:

kube-system     metrics-server-5955646df-nmkkw     1/1     Running   0       67m     10.244.1.230   aks-poolagent-36239600-vmss000000   <none>           <none>
kube-system     metrics-server-5955646df-qwtcd     1/1     Running   0       67m     10.244.1.229   aks-poolagent-36239600-vmss000000   <none>           <none>

Service is working too:

kube-system     metrics-server         ClusterIP      10.0.59.233    <none>           443/TCP         2y168d

If I curl the pod from inside another pod I got this response:

curl --insecure -iL https://10.244.1.230:4443
HTTP/2 403 
cache-control: no-cache, private
content-type: application/json
x-content-type-options: nosniff
content-length: 233
date: Thu, 04 Aug 2022 12:10:09 GMT

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
    
  },
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {
    
  },
  "code": 403

Metric server pods deployment configuration:

Name:                   metrics-server
Namespace:              kube-system
CreationTimestamp:      Mon, 17 Feb 2020 16:38:15 +0000
Labels:                 addonmanager.kubernetes.io/mode=Reconcile
                        k8s-app=metrics-server
                        kubernetes.io/cluster-service=true
Annotations:            deployment.kubernetes.io/revision: 17
Selector:               k8s-app=metrics-server
Replicas:               2 desired | 2 updated | 2 total | 2 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  1 max unavailable, 1 max surge
Pod Template:
  Labels:           k8s-app=metrics-server
  Annotations:      kubectl.kubernetes.io/restartedAt: 2022-08-04T11:01:36Z
  Service Account:  metrics-server
  Containers:
   metrics-server:
    Image:      mcr.microsoft.com/oss/kubernetes/metrics-server:v0.5.2
    Port:       4443/TCP
    Host Port:  0/TCP
    Command:
      /metrics-server
      --kubelet-insecure-tls
      --kubelet-preferred-address-types=InternalIP
      --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
      --cert-dir=/tmp
      --secure-port=4443
    Limits:
      cpu:     1
      memory:  2000Mi
    Requests:
      cpu:        44m
      memory:     55Mi
    Liveness:     http-get https://:ms-https/livez delay=20s timeout=10s period=10s #success=1 #failure=3
    Readiness:    http-get https://:ms-https/readyz delay=60s timeout=10s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /tmp from tmp-dir (rw)
  Volumes:
   tmp-dir:
    Type:               EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:             
    SizeLimit:          <unset>
  Priority Class Name:  system-node-critical
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
OldReplicaSets:  <none>
NewReplicaSet:   metrics-server-5955646df (2/2 replicas created)
Events:          <none>

And metric service config:

Name:              metrics-server
Namespace:         kube-system
Labels:            addonmanager.kubernetes.io/mode=Reconcile
                   kubernetes.io/cluster-service=true
                   kubernetes.io/name=Metrics-server
Annotations:       Selector:  k8s-app=metrics-server
Type:              ClusterIP
IP:                10.0.59.233
Port:              <unset>  443/TCP
TargetPort:        443/TCP
Endpoints:         10.244.1.229:443,10.244.1.230:443
Session Affinity:  None
Events:            <none>

Question
How can I fix this issue? Do I need to contact MS? I don't have premium support.

[ghost]

Action required from @Azure/aks-pm

[ghost]

Issue needing attention of @Azure/aks-leads

[Kapsztajn]

Ok I managed to fix the issue. I removed metrics server via:
kubectl delete -f https://raw.githubusercontent.com/x1wins/CW-OVP/master/k8s-manifests/components.yaml

and after that installed it again via:
kubectl create -f https://raw.githubusercontent.com/x1wins/CW-OVP/master/k8s-manifests/components.yaml

After that problem was resolved. Issue which helped me: kubernetes-sigs/metrics-server#188