kubernetes.client.rest.ApiException: (503) error with list_cluster_custom_object

[moluzhui]

What happened (please include outputs or screenshots):
I wanted to use client to get node data collected by metric-server and set it to be collected every 30 seconds, but the following error occurred after several times of collection

Error is as follows

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/test.py", line 51, in get_host_info
    for device_resource_info in api.list_cluster_custom_object("metrics.k8s.io", "v1beta1", "nodes")['items']:
  File "/usr/local/lib/python3.6/site-packages/kubernetes/client/api/custom_objects_api.py", line 1344, in list_cluster_custom_object
    (data) = self.list_cluster_custom_object_with_http_info(group, version, plural, **kwargs)  # noqa: E501
  File "/usr/local/lib/python3.6/site-packages/kubernetes/client/api/custom_objects_api.py", line 1457, in list_cluster_custom_object_with_http_info
    collection_formats=collection_formats)
  File "/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 345, in call_api
    _preload_content, _request_timeout)
  File "/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 176, in __call_api
    _request_timeout=_request_timeout)
  File "/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 366, in request
    headers=headers)
  File "/usr/local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 241, in GET
    query_params=query_params)
  File "/usr/local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 231, in request
    raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (503)
Reason: Service Unavailable
HTTP response headers: HTTPHeaderDict({'Content-Type': 'text/plain; charset=utf-8', 'X-Content-Type-Options': 'nosniff', 'Date': 'Wed, 27 May 2020 14:03:55 GMT', 'Content-Length': '20'})
HTTP response body: service unavailable

What you expected to happen:
I want to be able to get node's resource status every time.

How to reproduce it (as minimally and precisely as possible):
The client part of the code is as follows

config.kube_config.load_kube_config(config_file="~/.kube/config")
v1 = client.CoreV1Api()
api = client.CustomObjectsApi()
while(true):
    for node_resource_info in api.list_cluster_custom_object("metrics.k8s.io", "v1beta1", "nodes")['items']:
    ...
    ...
    time.sleep(30)

Anything else we need to know?:

Environment:

• Kubernetes version (kubectl version): v1.17.3
• OS (e.g., MacOS 10.13.6): The container OS is Alpine Linux 3.10 and the host OS is Ubuntu 16.04.6 LTS
• Python version (python --version): python3.6.9
• Python client version (pip list | grep kubernetes)：kubernetes==11.0.0

[moshevayner]

Hey @moluzhui
Since the API server failed to process the request, I'm not sure it's a client error.
To start with- did you check the api server and metrics server logs (they should both be in the kube-system namespace.

[moluzhui]

@MoShitrit The results seem to be all right

$ kubectl get svc,deployment,pod -n kube-system
NAME                     TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                  AGE
service/kube-dns         ClusterIP   10.96.0.10     <none>        53/UDP,53/TCP,9153/TCP   10d
service/metrics-server   ClusterIP   10.97.221.48   <none>        443/TCP                  36h

NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/coredns          2/2     2            2           10d
deployment.apps/metrics-server   1/1     1            1           36h

NAME                                  READY   STATUS    RESTARTS   AGE
pod/coredns-6955765f44-4nmz4          1/1     Running   0          10d
pod/coredns-6955765f44-w8985          1/1     Running   0          10d
pod/etcd-node-2                       1/1     Running   0          10d
pod/kube-apiserver-node-2             1/1     Running   0          10d
pod/kube-controller-manager-node-2    1/1     Running   12         10d
pod/kube-flannel-ds-amd64-ld2nq       1/1     Running   31         10d
pod/kube-flannel-ds-amd64-rk28k       1/1     Running   17         10d
pod/kube-proxy-fc48p                  1/1     Running   0          10d
pod/kube-proxy-h4mkw                  1/1     Running   0          10d
pod/kube-scheduler-node-2             1/1     Running   14         10d
pod/metrics-server-5dc6f58b6f-xx9vg   1/1     Running   17         36h

[moshevayner]

Well, I'm not really sure @moluzhui
Your metrics-server pod has had 17 restarts in the past 36 hours

pod/metrics-server-5dc6f58b6f-xx9vg   1/1     Running   17         36h

There's a chance that it had a restart during one of the requests you made to the metrics api, which would result in a 500 error.

Can you run kubectl -n kube-system logs metrics-server-5dc6f58b6f-xx9vg -p
And also kubectl -n kube-system describe pod metrics-server-5dc6f58b6f-xx9vg?

They could shed more light on why is this pod restarting so much.

[moluzhui]

$ kubectl -n kube-system logs pod/metrics-server-5dc6f58b6f-xx9vg -p
I0528 10:11:47.703016       1 serving.go:312] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
I0528 10:11:48.733255       1 secure_serving.go:116] Serving securely on [::]:443
E0528 10:11:54.568428       1 reststorage.go:160] unable to fetch pod metrics for pod default/result-time-78df75wth9d: no metrics known for pod
E0528 10:11:54.568504       1 reststorage.go:160] unable to fetch pod metrics for pod default/result-time-78df75wth9d: no metrics known for pod
E0528 10:11:54.579375       1 reststorage.go:135] unable to fetch node metrics for node "node-2": no metrics known for node
E0528 10:11:54.579393       1 reststorage.go:135] unable to fetch node metrics for node "node-5": no metrics known for node
E0528 10:11:54.583128       1 reststorage.go:135] unable to fetch node metrics for node "node-2": no metrics known for node
E0528 10:11:54.583141       1 reststorage.go:135] unable to fetch node metrics for node "node-5": no metrics known for node

$ kubectl -n kube-system describe pod metrics-server-5dc6f58b6f-xx9vg
Name:         metrics-server-5dc6f58b6f-xx9vg
Namespace:    kube-system
Priority:     0
Node:         node-5/10.0.0.5
Start Time:   Tue, 26 May 2020 21:16:20 -0400
Labels:       k8s-app=metrics-server
              pod-template-hash=5dc6f58b6f
Annotations:  <none>
Status:       Running
IP:           10.244.1.58
IPs:
  IP:           10.244.1.58
Controlled By:  ReplicaSet/metrics-server-5dc6f58b6f
Containers:
  metrics-server:
    Container ID:  docker://42671c614ff8e0af3ad5cb78908f410b1514503b856882bbcd5643803953290e
    Image:         k8s.gcr.io/metrics-server-amd64:v0.3.6
    Image ID:      docker-pullable://registry.cn-hangzhou.aliyuncs.com/google_containers/metrics-server-amd64@sha256:c9c4e95068b51d6b33a9dccc61875df07dc650abbf4ac1a19d58b4628f89288b
    Port:          <none>
    Host Port:     <none>
    Args:
      --kubelet-insecure-tls
      --kubelet-preferred-address-types=InternalIP
    State:          Running
      Started:      Thu, 28 May 2020 07:04:44 -0400
    Last State:     Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Thu, 28 May 2020 06:11:46 -0400
      Finished:     Thu, 28 May 2020 07:04:41 -0400
    Ready:          True
    Restart Count:  17
    Environment:    <none>
    Mounts:
      /tmp from tmp-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from metrics-server-token-kpszn (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  tmp-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  metrics-server-token-kpszn:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  metrics-server-token-kpszn
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>

Don't seem to see anything wrong with that

[moshevayner]

Well, actually it does seem like a problem with the metrics server.
First of all- as I noted earlier- the fact that it restarted 17 times over a relatively short period- indicates something isn't right. From the output of describe you can see that it terminated due to an error:

    Last State:     Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Thu, 28 May 2020 06:11:46 -0400
      Finished:     Thu, 28 May 2020 07:04:41 -0400
    Ready:          True

Secondly- it seems to be failing to pull metrics from several nodes/pods.
Also, to make sure- I verified that this function call works for me locally.

I'm pretty sure it's not a bug in the client, but rather an issue with the metrics server not running properly.

I see on the issue description that you're running k8s ver 1.17.3 - what's the version of your metrics server?
It could also be helpful if you can post the output of kubectl -n kube-system get deploy metrics-server -o yaml to give some additional context about the running config.

[moluzhui]

@MoShitrit The metric-server version is 0.3.6. The final result appears to be that metric-server is unavailable.

$ kubectl -n kube-system get deploy metrics-server -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"k8s-app":"metrics-server"},"name":"metrics-server","namespace":"kube-system"},"spec":{"selector":{"matchLabels":{"k8s-app":"metrics-server"}},"template":{"metadata":{"labels":{"k8s-app":"metrics-server"},"name":"metrics-server"},"spec":{"containers":[{"args":["--kubelet-insecure-tls","--kubelet-preferred-address-types=InternalIP"],"image":"k8s.gcr.io/metrics-server-amd64:v0.3.6","name":"metrics-server","volumeMounts":[{"mountPath":"/tmp","name":"tmp-dir"}]}],"serviceAccountName":"metrics-server","volumes":[{"emptyDir":{},"name":"tmp-dir"}]}}}}
  creationTimestamp: "2020-05-27T01:16:20Z"
  generation: 1
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
  resourceVersion: "2162530"
  selfLink: /apis/apps/v1/namespaces/kube-system/deployments/metrics-server
  uid: 90b0f169-91a3-4af5-83a5-3d649039f638
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: metrics-server
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        k8s-app: metrics-server
      name: metrics-server
    spec:
      containers:
      - args:
        - --kubelet-insecure-tls
        - --kubelet-preferred-address-types=InternalIP
        image: k8s.gcr.io/metrics-server-amd64:v0.3.6
        imagePullPolicy: IfNotPresent
        name: metrics-server
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /tmp
          name: tmp-dir
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: metrics-server
      serviceAccountName: metrics-server
      terminationGracePeriodSeconds: 30
      volumes:
      - emptyDir: {}
        name: tmp-dir
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2020-05-27T01:16:20Z"
    lastUpdateTime: "2020-05-27T01:16:22Z"
    message: ReplicaSet "metrics-server-5dc6f58b6f" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  - lastTransitionTime: "2020-05-29T11:06:31Z"
    lastUpdateTime: "2020-05-29T11:06:31Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  observedGeneration: 1
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

[moshevayner]

So yeah I would definitely change direction here towards investigating why is metrics-server failing.
I found a couple of issues on their repo with several suggestions on how to fix the metrics pulling, examples:
kubernetes-sigs/metrics-server#167 (comment)
kubernetes-sigs/metrics-server#143 (comment)
Worth looking into those and figure out what's missing.
It might be - --kubelet-insecure-tls, but I'm not 100% sure.

[roycaihw]

Yes this looks more like a server-side issue than a python client one. You could enable the debug logging to see if the server gives any more meaningful response, or try monitoring the same endpoint with kubectl kubectl get --raw '/apis/metrics.k8s.io/v1beta1/nodes'

[moshevayner]

/remove-kind bug

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[moshevayner]

/close