Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

unable to fully collect metrics #130

Open
vvkkhjt opened this issue Sep 3, 2018 · 30 comments

Comments

@vvkkhjt
Copy link

@vvkkhjt vvkkhjt commented Sep 3, 2018

use k8s v1.11.2 && metrics-server v0.3.0

Did the following:

  • Add apiserver args like this:
    KUBE_API_ARGS="--authorization-mode=Node,RBAC --runtime-config=rbac.authorization.k8s.io/v1beta1 --kubelet-https=true --enable-bootstrap-token-auth --token-auth-file=/etc/kubernetes/token.csv --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem --tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem --client-ca-file=/etc/kubernetes/ssl/ca.pem --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem --etcd-cafile=/etc/kubernetes/ssl/ca.pem --etcd-certfile=/etc/kubernetes/ssl/kubernetes.pem --etcd-keyfile=/etc/kubernetes/ssl/kubernetes-key.pem --enable-swagger-ui=true --apiserver-count=3 --audit-log-maxage=30 --audit-log-maxbackup=3 --audit-log-maxsize=100 --audit-log-path=/var/lib/audit.log --event-ttl=1h --requestheader-client-ca-file=/etc/kubernetes/ssl/ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --proxy-client-cert-file=/etc/kubernetes/ssl/kube-proxy.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-proxy-key.pem"
  • Add kubelet args like this:
    KUBELET_ARGS="--network-plugin=cni --cgroup-driver=cgroupfs --cluster-dns=10.254.0.2 --bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --cert-dir=/etc/kubernetes/ssl --cluster-domain=cluster.local --hairpin-mode promiscuous-bridge --serialize-image-pulls=false --authentication-token-webhook=true --read-only-port=10255"
  • Ran kubectl create -f deploy/1.8+/
  • look metrics-server logs;

then metrics-server logs show this:

unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:192.168.66.66: unable to fetch metrics from Kubelet 192.168.66.66 (192.168.66.66): Get https://192.168.66.66:10250/stats/summary/: x509: cannot validate certificate for 192.168.66.66 because it doesn't contain any IP SANs, unable to fully scrape metrics from source kubelet_summary:192.168.66.64: unable to fetch metrics from Kubelet 192.168.66.64 (192.168.66.64): Get https://192.168.66.64:10250/stats/summary/: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:192.168.220.50: unable to fetch metrics from Kubelet 192.168.220.50 (192.168.220.50): Get https://192.168.220.50:10250/stats/summary/: x509: cannot validate certificate for 192.168.220.50 because it doesn't contain any IP SANs]

when I use metrics-server v0.2.1,add metrcs-server args like this:

command:
   - /metrics-server
   - --source=kubernetes.summary_api:https://kubernetes.default.svc?kubeletHttps=true&kubeletPort=10250&useServiceAccount=true&insecure=true

It seems to work,but when i use v0.3.0 and add args --source it's says Error: unknown flag: --source,
So I used the help instruction to find the corresponding parameter, but I didn't find it. So, how can I fix this problem?

This is my first time to mention issue, which is not so good. Sorry.

@MIBc

This comment has been minimized.

Copy link
Contributor

@MIBc MIBc commented Sep 3, 2018

@vvkkhjt try this flag --kubelet-insecure-tls

@DirectXMan12

This comment has been minimized.

Copy link
Contributor

@DirectXMan12 DirectXMan12 commented Sep 5, 2018

the --kubelet-insecure-tls flag is the equivalent of the insecure=true flag in the old metrics server release. However, if you're going to connect via internal IP addresses, I'd recommended just setting up your certificates so that they list the internal IP as an alternate name.

@vvkkhjt

This comment has been minimized.

Copy link
Author

@vvkkhjt vvkkhjt commented Sep 6, 2018

@MIBc when i use that flag:

streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=15, ErrCode=NO_ERROR, debug=""
E0906 06:28:39.309417       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=15, ErrCode=NO_ERROR, debug=""
E0906 06:28:39.312965       1 reflector.go:322] github.com/kubernetes-incubator/metrics-server/vendor/k8s.io/client-go/informers/factory.go:130: Failed to watch *v1.Pod: Get https://10.254.0.1:443/api/v1/pods?resourceVersion=856148&timeoutSeconds=395&watch=true: dial tcp 10.254.0.1:443: connect: connection refused
E0906 06:28:39.313303       1 reflector.go:322] github.com/kubernetes-incubator/metrics-server/vendor/k8s.io/client-go/informers/factory.go:130: Failed to watch *v1.Node: Get https://10.254.0.1:443/api/v1/nodes?resourceVersion=858077&timeoutSeconds=440&watch=true: dial tcp 10.254.0.1:443: connect: connection refused
@DirectXMan12

This comment has been minimized.

Copy link
Contributor

@DirectXMan12 DirectXMan12 commented Sep 6, 2018

Get https://10.254.0.1:443/api/v1/pods?resourceVersion=856148&timeoutSeconds=395&watch=true: dial tcp 10.254.0.1:443: connect: connection refused

That implies that metrics-server isn't able to contact your main kubernetes API server.

@vvkkhjt

This comment has been minimized.

Copy link
Author

@vvkkhjt vvkkhjt commented Sep 7, 2018

@DirectXMan12 but the other services are working good。。when i use helm install heapster,it works good either。

helm install stable/heapster --name heapster --namespace kube-system --set "rbac.create=true,command[0]=/heapster,command[1]=--source=kubernetes.summary_api:https://kubernetes.default.svc?kubeletHttps=true&kubeletPort=10250&useServiceAccount=true&insecure=true" --debug

then

k top node

NAME             CPU(cores)   CPU%      MEMORY(bytes)   MEMORY%   
192.168.220.50   254m         3%        3606Mi          11%       
192.168.66.64    226m         5%        2096Mi          57%       
192.168.66.66    99m          2%        1388Mi          62% 
@brunovlucena

This comment has been minimized.

Copy link

@brunovlucena brunovlucena commented Sep 24, 2018

Hi all,

k8s: v1.11
metrics-server: 0.3.1

I am not using TLS, However, I fall into similar problem(not being able to fully collect metrics). The logs of metrics-server show that it receives a - Raw (json) response - from kubelet, but i can't retrieve the metrics via k top node/pod.

After receiving the kubelet's sumary(json) there is such a line:
ScrapeMetrics: time: 59.513576ms, nodes: 0, pods: 0

Another message says it cannot parse some init containers. Is this the reason it is failing?
unable to get a valid timestamp for metric point for container

Best!

E0924 12:45:08.398855 1 manager.go:102] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:k8s1: unable to get a valid timestamp for metric point for container "install-cni" in pod kube-system/kube-router-2l6f5 on node "10.36.10.1", discarding data: no non-zero timestamp on either CPU or memory, unable to fully scrape metrics from source kubelet_summary:k8s4: unable to get a valid timestamp for metric point for container "install-cni" in pod kube-system/kube-router-6zvxd on node "10.36.10.4", discarding data: no non-zero timestamp on either CPU or memory, unable to fully scrape
metrics from source kubelet_summary:k8s5: [unable to get a valid timestamp for metric point for container "compile" in pod default/hello-6786bd6b44-t6z5z on node "10.36.10.5", discarding data: no non-zero timestamp on either CPU or memory, unable to get a valid timestamp for metric point for container "prepare" in pod default/hello-6786bd6b44-t6z5z on node "10.36.10.5", discarding data: no non-zero timestamp on either
CPU or memory, unable to get a valid timestamp for metric point for container "install-cni" in pod kube-system/kube-router-gntmq on node "10.36.10.5", discarding data: no non-zero timestamp on either CPU or memory]]

@DirectXMan12

This comment has been minimized.

Copy link
Contributor

@DirectXMan12 DirectXMan12 commented Sep 24, 2018

@brunovlucena what happens when you run k top node? It's expected that right after pod start there'll be missing points, but you should still be able to get information for other pods/nodes.

@gabrielfsousa

This comment has been minimized.

Copy link

@gabrielfsousa gabrielfsousa commented Sep 25, 2018

i have the same problem

@brunovlucena

This comment has been minimized.

Copy link

@brunovlucena brunovlucena commented Sep 26, 2018

@brunovlucena what happens when you run k top node? It's expected that right after pod start there'll be missing points, but you should still be able to get information for other pods/nodes.

Error from server (NotFound): nodemetrics.metrics.k8s.io "k8s1" not found

@DirectXMan12

This comment has been minimized.

Copy link
Contributor

@DirectXMan12 DirectXMan12 commented Sep 27, 2018

is there only one node? kubectl top is pretty finnicky about partial data, so it'd be good to see if there's partial data being returned (try kubectl top node --v=10).

@brunovlucena

This comment has been minimized.

Copy link

@brunovlucena brunovlucena commented Oct 2, 2018

kubectl top node k8s1 --heapster-service=metrics-server --heapster-port= --heapster-namespace=kube-system --heapster-scheme= --v=10

I1002 11:53:59.648863    8979 request.go:897] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"nodemetrics.metrics.k8s.io \"k8s1\" not found","reason":"NotFound","details":{"name":"k8s1","group":"metrics.k8s.io","kind":"nodemetrics"},"code":404}
I1002 11:53:59.649584    8979 helpers.go:201] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "nodemetrics.metrics.k8s.io \"k8s1\" not found",
  "reason": "NotFound",
  "details": {
    "name": "k8s1",
    "group": "metrics.k8s.io",
    "kind": "nodemetrics"
  },
  "code": 404
}]

kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes" | jq

{
  "kind": "NodeMetricsList",
  "apiVersion": "metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes"
  },
  "items": []
}

kubectl get nodes

NAME      STATUS    ROLES     AGE       VERSION
k8s1      Ready     <none>    20m       v1.11.2
k8s4      Ready     <none>    20m       v1.11.2
k8s5      Ready     <none>    20m       v1.11.2

kubectl get ep --all-namespaces -o wide

NAMESPACE     NAME                      ENDPOINTS                                               AGE
default       kubernetes                10.36.10.1:6443,10.36.10.4:6443,10.36.10.5:6443         21m
kube-system   kube-controller-manager   <none>                                                  21m
kube-system   kube-dns                  10.200.0.2:53,10.200.1.3:53,10.200.0.2:53 + 1 more...   9s
kube-system   kube-scheduler            <none>                                                  21m
kube-system   metrics-server            10.200.1.2:443                                          6m
@DirectXMan12

This comment has been minimized.

Copy link
Contributor

@DirectXMan12 DirectXMan12 commented Oct 2, 2018

That kubectl top node output looks truncated, and seems to have extra heapster arguments. Is that the full output? If not, can you post the full output, and if so, can you try without the extra heapster arguments?

@lbudai74

This comment has been minimized.

Copy link

@lbudai74 lbudai74 commented Oct 5, 2018

@DirectXMan12 I have tried the --kubelet-insecure-tls flag, but heapster fails to start and in the logs I can see a "usage" screen telling me "unknown flag: --kubelet-insecure-tls".

I have also tried to add that on the line where the source is defined, like this:
- --source=kubernetes.summary_api:https://kubernetes.default?kubeletHttps=true&kubeletPort=10250&kubelet-insecure-tls=true
In this case heapster will start, but will still fail to retrieve metrics due to lack of IP SAN in the kubelet certs.

I've tried with both v1.4.0 and v1.5.4 of heapster.

Is there a way to tell heapster to accept the cert without that IP SAN setting?

Thank you,
Laszlo

@tengzhexiao

This comment has been minimized.

Copy link

@tengzhexiao tengzhexiao commented Oct 12, 2018

i have the same problem...

@DirectXMan12

This comment has been minimized.

Copy link
Contributor

@DirectXMan12 DirectXMan12 commented Nov 8, 2018

@lbudai74 this is the repository for metrics-server, not heapster. Heapster's repository can be found at kubernetes/heapster.

@DirectXMan12

This comment has been minimized.

Copy link
Contributor

@DirectXMan12 DirectXMan12 commented Nov 8, 2018

@tengzhexiao which problem?

@Rajat-0

This comment has been minimized.

Copy link
Contributor

@Rajat-0 Rajat-0 commented Nov 29, 2018

I am also facing the same issue
metrics-server logs ::

I1129 06:48:29.089117       1 serving.go:273] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
[restful] 2018/11/29 06:48:30 log.go:33: [restful/swagger] listing is available at https://:443/swaggerapi
[restful] 2018/11/29 06:48:30 log.go:33: [restful/swagger] https://:443/swaggerui/ is mapped to folder /swagger-ui/
I1129 06:48:30.183062       1 serve.go:96] Serving securely on [::]:443
E1129 06:49:30.145670       1 manager.go:102] unable to fully collect metrics: unable to fully scrape metrics from source kubelet_summary:127.0.0.1: unable to fetch metrics from Kubelet 127.0.0.1 (127.0.0.1): Get https://127.0.0.1:10250/stats/summary/: dial tcp 127.0.0.1:10250: connect: connection refused
E1129 06:50:30.139404       1 manager.go:102] unable to fully collect metrics: unable to fully scrape metrics from source kubelet_summary:127.0.0.1: unable to fetch metrics from Kubelet 127.0.0.1 (127.0.0.1): Get https://127.0.0.1:10250/stats/summary/: dial tcp 127.0.0.1:10250: connect: connection refused

@DirectXMan12

This comment has been minimized.

Copy link
Contributor

@DirectXMan12 DirectXMan12 commented Nov 30, 2018

@Rajat-0 I think you're facing a different issue. If your kubelet lists the preferred address as 127.0.0.1, metrics-server can't collect from it, because it's the 127.0.0.1 is the loopback adapter (i.e. localhost in the pod is not localhost in the node).

@lolspider

This comment has been minimized.

Copy link

@lolspider lolspider commented Dec 3, 2018

i have the same problem

@kfox1111

This comment has been minimized.

Copy link

@kfox1111 kfox1111 commented Feb 6, 2019

what about a feature that has watches a namespace for a configmap per node that contains the ca to use to verify that node? Nodes can self register self signed ca's then.

@shahaf600

This comment has been minimized.

Copy link

@shahaf600 shahaf600 commented Feb 7, 2019

well I have the same issue, opened a ticket here: #214 tried to be as much as informative as possible.
so this is what happens with v0.2.1 - when i run watch 'kubectl top pod <pod_name>' , it runs the command every 1 second , so some of the the queries work and some of them return: "Error from server (NotFound): podmetrics.metrics.k8s.io "POD_NAME**" not found"

in v0.3.1 it ignores completely from pods with init containers. and why is that ? because init containers returns empty stats in stats summary api call (unless you are using docker on the kubernetes nodes).
and I think it's because when parsing the json in the go code of metrics-server it drops the pods with init containers entirely .

@stafot

This comment has been minimized.

Copy link

@stafot stafot commented Feb 11, 2019

@DirectXMan12
Provider: AWS EKS
K8s version: 1.11.5
metrics-server: 0.3.1
Seems having similar issue with @brunovlucena but doesn't look that is related with heapster, even If I run also heapster apart from metrics-server because without heapster my kubernetes dashboard graphs were not working.
The verbose outpout you ask from @brunovlucena in my case is the following.

I0211 10:34:04.021131   11131 round_trippers.go:411] Response Headers:
I0211 10:34:04.021143   11131 round_trippers.go:414]     Audit-Id: 52cf9a16-77ce-48ca-a6c9-474626e3ca79
I0211 10:34:04.021168   11131 round_trippers.go:414]     Content-Type: text/plain; charset=utf-8
I0211 10:34:04.021179   11131 round_trippers.go:414]     X-Content-Type-Options: nosniff
I0211 10:34:04.021189   11131 round_trippers.go:414]     Content-Length: 20
I0211 10:34:04.021198   11131 round_trippers.go:414]     Date: Mon, 11 Feb 2019 08:34:03 GMT
I0211 10:34:04.021234   11131 request.go:942] Response Body: service unavailable
I0211 10:34:04.021434   11131 helpers.go:198] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "the server is currently unable to handle the request (get pods.metrics.k8s.io)",
  "reason": "ServiceUnavailable",
  "details": {
    "group": "metrics.k8s.io",
    "kind": "pods",
    "causes": [
      {
        "reason": "UnexpectedServerResponse",
        "message": "service unavailable"
      }
    ]
  },
  "code": 503
}]
F0211 10:34:04.021487   11131 helpers.go:116] Error from server (ServiceUnavailable): the server is currently unable to handle the request (get pods.metrics.k8s.io)

Any ideas would be appreciated.

@Kingchan230

This comment has been minimized.

Copy link

@Kingchan230 Kingchan230 commented Feb 28, 2019

@DirectXMan12
Provider: AWS EKS
K8s version: 1.11.5
metrics-server: 0.3.1
Seems having similar issue with @brunovlucena but doesn't look that is related with heapster, even If I run also heapster apart from metrics-server because without heapster my kubernetes dashboard graphs were not working.
The verbose outpout you ask from @brunovlucena in my case is the following.

I0211 10:34:04.021131   11131 round_trippers.go:411] Response Headers:
I0211 10:34:04.021143   11131 round_trippers.go:414]     Audit-Id: 52cf9a16-77ce-48ca-a6c9-474626e3ca79
I0211 10:34:04.021168   11131 round_trippers.go:414]     Content-Type: text/plain; charset=utf-8
I0211 10:34:04.021179   11131 round_trippers.go:414]     X-Content-Type-Options: nosniff
I0211 10:34:04.021189   11131 round_trippers.go:414]     Content-Length: 20
I0211 10:34:04.021198   11131 round_trippers.go:414]     Date: Mon, 11 Feb 2019 08:34:03 GMT
I0211 10:34:04.021234   11131 request.go:942] Response Body: service unavailable
I0211 10:34:04.021434   11131 helpers.go:198] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "the server is currently unable to handle the request (get pods.metrics.k8s.io)",
  "reason": "ServiceUnavailable",
  "details": {
    "group": "metrics.k8s.io",
    "kind": "pods",
    "causes": [
      {
        "reason": "UnexpectedServerResponse",
        "message": "service unavailable"
      }
    ]
  },
  "code": 503
}]
F0211 10:34:04.021487   11131 helpers.go:116] Error from server (ServiceUnavailable): the server is currently unable to handle the request (get pods.metrics.k8s.io)

Any ideas would be appreciated.

when i use kubectl top node ,I get the same error like you ,have you fix it ?

@stafot

This comment has been minimized.

Copy link

@stafot stafot commented Feb 28, 2019

@Kingchan230 Yes I resolved my issue. In my case the problem was that I applied some resources from kube-prometheus that messed my infra. At this implementation prometheus-adapter instead of using v1beta1.custom.metrics.k8s.io uses v1beta1.metrics.k8s.io and I did not applied a prometheus-adapter service. I don't know if you are in the same situation. But to help you debug it, in my case the command that lead me to understand my misconfiguration was the below one. kubectl describe apiservice v1beta1.metrics.k8s.io which returned the following, inter alla

    Message:               service/prometheus-adapter in "monitoring" is not present
    Reason:                ServiceNotFound
    Status:                False
    Type:                  Available

If something is unclear feel free to reach me back.

@tonglv

This comment has been minimized.

Copy link

@tonglv tonglv commented Apr 8, 2019

i have the same problem
E0404 07:03:00.095957 1 manager.go:102] unable to fully collect metrics: unable to fully scrape metrics from source kubelet_summary:kubernetes-113: unable to fetch metrics from Kubelet kubernetes-113 (10.180.160.113): request failed - "500 Internal Server Error", response: "Authorization error (user=system:serviceaccount:kube-system:metrics-server, verb=get, resource=nodes, subresource=stats)"

@ksylvan

This comment has been minimized.

Copy link

@ksylvan ksylvan commented Jun 18, 2019

I had the same issue and re-applying the Prometheus adapter API service file fixed it:

kubectl apply -f prometheus-adapter-apiService.yaml
@urvineet

This comment has been minimized.

Copy link

@urvineet urvineet commented Jul 5, 2019

FIxed for me on EKS
helm install stable/metrics-server --name metrics-server --namespace metrics --set args[0]=--kubelet-insecure-tls,args[1]=--kubelet-preferred-address-types=InternalIP

@fejta-bot

This comment has been minimized.

Copy link

@fejta-bot fejta-bot commented Oct 3, 2019

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@vvkkhjt

This comment has been minimized.

Copy link
Author

@vvkkhjt vvkkhjt commented Oct 10, 2019

/remove-lifecycle stale

@JokerDevops

This comment has been minimized.

Copy link

@JokerDevops JokerDevops commented Oct 28, 2019

@DirectXMan12 I use kubectl top node --v=10 command output information is as follows:

[root@node40 ~]# kubectl top node --v=10
I1028 17:02:10.710211    2914 loader.go:359] Config loaded from file /root/.kube/config
I1028 17:02:10.714789    2914 loader.go:359] Config loaded from file /root/.kube/config
I1028 17:02:10.715216    2914 round_trippers.go:419] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.13.6 (linux/amd64) kubernetes/abdda3f" 'https://192.168.0.50:8443/api?timeout=32s'
I1028 17:02:10.728855    2914 round_trippers.go:438] GET https://192.168.0.50:8443/api?timeout=32s 200 OK in 13 milliseconds
I1028 17:02:10.728981    2914 round_trippers.go:444] Response Headers:
I1028 17:02:10.729031    2914 round_trippers.go:447]     Content-Length: 135
I1028 17:02:10.729076    2914 round_trippers.go:447]     Date: Mon, 28 Oct 2019 09:02:10 GMT
I1028 17:02:10.729140    2914 round_trippers.go:447]     Content-Type: application/json
I1028 17:02:10.729241    2914 request.go:942] Response Body: {"kind":"APIVersions","versions":["v1"],"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"192.168.0.41:6443"}]}
I1028 17:02:10.729562    2914 round_trippers.go:419] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.13.6 (linux/amd64) kubernetes/abdda3f" 'https://192.168.0.50:8443/apis?timeout=32s'
I1028 17:02:10.730976    2914 round_trippers.go:438] GET https://192.168.0.50:8443/apis?timeout=32s 200 OK in 1 milliseconds
I1028 17:02:10.731137    2914 round_trippers.go:444] Response Headers:
I1028 17:02:10.731203    2914 round_trippers.go:447]     Date: Mon, 28 Oct 2019 09:02:10 GMT
I1028 17:02:10.731299    2914 round_trippers.go:447]     Content-Type: application/json
I1028 17:02:10.731361    2914 round_trippers.go:447]     Content-Length: 4054
I1028 17:02:10.731480    2914 request.go:942] Response Body: {"kind":"APIGroupList","apiVersion":"v1","groups":[{"name":"apiregistration.k8s.io","versions":[{"groupVersion":"apiregistration.k8s.io/v1","version":"v1"},{"groupVersion":"apiregistration.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"apiregistration.k8s.io/v1","version":"v1"}},{"name":"extensions","versions":[{"groupVersion":"extensions/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"extensions/v1beta1","version":"v1beta1"}},{"name":"apps","versions":[{"groupVersion":"apps/v1","version":"v1"},{"groupVersion":"apps/v1beta2","version":"v1beta2"},{"groupVersion":"apps/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"apps/v1","version":"v1"}},{"name":"events.k8s.io","versions":[{"groupVersion":"events.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"events.k8s.io/v1beta1","version":"v1beta1"}},{"name":"authentication.k8s.io","versions":[{"groupVersion":"authentication.k8s.io/v1","version":"v1"},{"groupVersion":"authentication.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"authentication.k8s.io/v1","version":"v1"}},{"name":"authorization.k8s.io","versions":[{"groupVersion":"authorization.k8s.io/v1","version":"v1"},{"groupVersion":"authorization.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"authorization.k8s.io/v1","version":"v1"}},{"name":"autoscaling","versions":[{"groupVersion":"autoscaling/v1","version":"v1"},{"groupVersion":"autoscaling/v2beta1","version":"v2beta1"},{"groupVersion":"autoscaling/v2beta2","version":"v2beta2"}],"preferredVersion":{"groupVersion":"autoscaling/v1","version":"v1"}},{"name":"batch","versions":[{"groupVersion":"batch/v1","version":"v1"},{"groupVersion":"batch/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"batch/v1","version":"v1"}},{"name":"certificates.k8s.io","versions":[{"groupVersion":"certificates.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"certificates.k8s.io/v1beta1","version":"v1beta1"}},{"name":"networking.k8s.io","versions":[{"groupVersion":"networking.k8s.io/v1","version":"v1"}],"preferredVersion":{"groupVersion":"networking.k8s.io/v1","version":"v1"}},{"name":"policy","versions":[{"groupVersion":"policy/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"policy/v1beta1","version":"v1beta1"}},{"name":"rbac.authorization.k8s.io","versions":[{"groupVersion":"rbac.authorization.k8s.io/v1","version":"v1"},{"groupVersion":"rbac.authorization.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"rbac.authorization.k8s.io/v1","version":"v1"}},{"name":"settings.k8s.io","versions":[{"groupVersion":"settings.k8s.io/v1alpha1","version":"v1alpha1"}],"preferredVersion":{"groupVersion":"settings.k8s.io/v1alpha1","version":"v1alpha1"}},{"name":"storage.k8s.io","versions":[{"groupVersion":"storage.k8s.io/v1","version":"v1"},{"groupVersion":"storage.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"storage.k8s.io/v1","version":"v1"}},{"name":"admissionregistration.k8s.io","versions":[{"groupVersion":"admissionregistration.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"admissionregistration.k8s.io/v1beta1","version":"v1beta1"}},{"name":"apiextensions.k8s.io","versions":[{"groupVersion":"apiextensions.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"apiextensions.k8s.io/v1beta1","version":"v1beta1"}},{"name":"scheduling.k8s.io","versions":[{"groupVersion":"scheduling.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"scheduling.k8s.io/v1beta1","version":"v1beta1"}},{"name":"coordination.k8s.io","versions":[{"groupVersion":"coordination.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"coordination.k8s.io/v1beta1","version":"v1beta1"}},{"name":"metrics.k8s.io","versions":[{"groupVersion":"metrics.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"metrics.k8s.io/v1beta1","version":"v1beta1"}}]}
I1028 17:02:10.732013    2914 round_trippers.go:419] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.13.6 (linux/amd64) kubernetes/abdda3f" 'https://192.168.0.50:8443/apis/metrics.k8s.io/v1beta1/nodes'
I1028 17:02:10.735511    2914 round_trippers.go:438] GET https://192.168.0.50:8443/apis/metrics.k8s.io/v1beta1/nodes 200 OK in 3 milliseconds
I1028 17:02:10.735538    2914 round_trippers.go:444] Response Headers:
I1028 17:02:10.735547    2914 round_trippers.go:447]     Date: Mon, 28 Oct 2019 09:02:10 GMT
I1028 17:02:10.735553    2914 round_trippers.go:447]     Content-Length: 375
I1028 17:02:10.735559    2914 round_trippers.go:447]     Content-Type: application/json
I1028 17:02:10.735588    2914 request.go:942] Response Body: {"kind":"NodeMetricsList","apiVersion":"metrics.k8s.io/v1beta1","metadata":{"selfLink":"/apis/metrics.k8s.io/v1beta1/nodes"},"items":[{"metadata":{"name":"192.168.0.47","selfLink":"/apis/metrics.k8s.io/v1beta1/nodes/192.168.0.47","creationTimestamp":"2019-10-28T09:02:10Z"},"timestamp":"2019-10-28T09:01:13Z","window":"30s","usage":{"cpu":"63913024n","memory":"577908Ki"}}]}
I1028 17:02:10.736558    2914 round_trippers.go:419] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.13.6 (linux/amd64) kubernetes/abdda3f" 'https://192.168.0.50:8443/api/v1/nodes'
I1028 17:02:10.741531    2914 round_trippers.go:438] GET https://192.168.0.50:8443/api/v1/nodes 200 OK in 4 milliseconds
I1028 17:02:10.741556    2914 round_trippers.go:444] Response Headers:
I1028 17:02:10.741565    2914 round_trippers.go:447]     Content-Type: application/json
I1028 17:02:10.741572    2914 round_trippers.go:447]     Date: Mon, 28 Oct 2019 09:02:10 GMT
I1028 17:02:10.743772    2914 request.go:942] Response Body: {"kind":"NodeList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/nodes","resourceVersion":"1940342"},"items":[{"metadata":{"name":"192.168.0.40","selfLink":"/api/v1/nodes/192.168.0.40","uid":"5e1f1f10-f182-11e9-84fc-902b34d396c8","resourceVersion":"1940340","creationTimestamp":"2019-10-18T08:36:23Z","labels":{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/os":"linux","kubernetes.io/hostname":"192.168.0.40","node-role.kubernetes.io/master":"node40"},"annotations":{"node.alpha.kubernetes.io/ttl":"0","volumes.kubernetes.io/controller-managed-attach-detach":"true"}},"spec":{"podCIDR":"10.20.0.0/24","taints":[{"key":"node-role.kubernetes.io/master","value":"node40","effect":"NoSchedule"}]},"status":{"capacity":{"cpu":"4","ephemeral-storage":"79833Mi","hugepages-2Mi":"0","memory":"16296408Ki","pods":"200"},"allocatable":{"cpu":"4","ephemeral-storage":"75339870903","hugepages-2Mi":"0","memory":"16194008Ki","pods":"200"},"conditions":[{"type":"MemoryPressure","status":"False","lastHeartbeatTime":"2019-10-28T09:02:10Z","lastTransitionTime":"2019-10-28T08:48:09Z","reason":"KubeletHasSufficientMemory","message":"kubelet has sufficient memory available"},{"type":"DiskPressure","status":"False","lastHeartbeatTime":"2019-10-28T09:02:10Z","lastTransitionTime":"2019-10-28T08:48:09Z","reason":"KubeletHasNoDiskPressure","message":"kubelet has no disk pressure"},{"type":"PIDPressure","status":"False","lastHeartbeatTime":"2019-10-28T09:02:10Z","lastTransitionTime":"2019-10-28T08:48:09Z","reason":"KubeletHasSufficientPID","message":"kubelet has sufficient PID available"},{"type":"Ready","status":"True","lastHeartbeatTime":"2019-10-28T09:02:10Z","lastTransitionTime":"2019-10-28T08:48:09Z","reason":"KubeletReady","message":"kubelet is posting ready status"}],"addresses":[{"type":"InternalIP","address":"192.168.0.40"},{"type":"Hostname","address":"192.168.0.40"}],"daemonEndpoints":{"kubeletEndpoint":{"Port":10250}},"nodeInfo":{"machineID":"8f57bb01b7084c389d564b784f8061c9","systemUUID":"032b0290-0434-05d3-9606-c80700080009","bootID":"9e985074-6ec7-4b83-bb10-abfe3949c0fe","kernelVersion":"4.19.12-1.el7.elrepo.x86_64","osImage":"CentOS Linux 7 (Core)","containerRuntimeVersion":"docker://18.6.3","kubeletVersion":"v1.13.6","kubeProxyVersion":"v1.13.6","operatingSystem":"linux","architecture":"amd64"},"images":[{"names":["quay.io/external_storage/rbd-provisioner@sha256:94fd36b8625141b62ff1addfa914d45f7b39619e55891bad0294263ecd2ce09a","quay.io/external_storage/rbd-provisioner:latest"],"sizeBytes":404591002},{"names":["registry.cn-hangzhou.aliyuncs.com/dnsjia/k8s@sha256:1d2e1229a918f4bc38b5a3f9f5f11302b3e71f8397b492afac7f273a0008776a","registry.cn-hangzhou.aliyuncs.com/dnsjia/k8s:kubernetes-dashboard-amd64_v1.10.0"],"sizeBytes":122460923}]}},{"metadata":{"name":"192.168.0.41","selfLink":"/api/v1/nodes/192.168.0.41","uid":"5e3a5adf-f182-11e9-84fc-902b34d396c8","resourceVersion":"1940341","creationTimestamp":"2019-10-18T08:36:24Z","labels":{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/os":"linux","kubernetes.io/hostname":"192.168.0.41","node-role.kubernetes.io/master":"node41"},"annotations":{"node.alpha.kubernetes.io/ttl":"0","volumes.kubernetes.io/controller-managed-attach-detach":"true"}},"spec":{"podCIDR":"10.20.2.0/24","taints":[{"key":"node-role.kubernetes.io/master","value":"node41","effect":"NoSchedule"}]},"status":{"capacity":{"cpu":"4","ephemeral-storage":"79833Mi","hugepages-2Mi":"0","memory":"16296404Ki","pods":"200"},"allocatable":{"cpu":"4","ephemeral-storage":"75339870903","hugepages-2Mi":"0","memory":"16194004Ki","pods":"200"},"conditions":[{"type":"MemoryPressure","status":"False","lastHeartbeatTime":"2019-10-28T09:02:10Z","lastTransitionTime":"2019-10-28T08:29:26Z","reason":"KubeletHasSufficientMemory","message":"kubelet has sufficient memory available"},{"type":"DiskPressure","status":"False","lastHeartbeatTime":"2019-10-28T09:02:10Z","lastTransitionTime":"2019-10-28T08:29:26Z","reason":"KubeletHasNoDiskPressure","message":"kubelet has no disk pressure"},{"type":"PIDPressure","status":"False","lastHeartbeatTime":"2019-10-28T09:02:10Z","lastTransitionTime":"2019-10-28T08:29:26Z","reason":"KubeletHasSufficientPID","message":"kubelet has sufficient PID available"},{"type":"Ready","status":"True","lastHeartbeatTime":"2019-10-28T09:02:10Z","lastTransitionTime":"2019-10-28T08:29:26Z","reason":"KubeletReady","message":"kubelet is posting ready status"}],"addresses":[{"type":"InternalIP","address":"192.168.0.41"},{"type":"Hostname","address":"192.168.0.41"}],"daemonEndpoints":{"kubeletEndpoint":{"Port":10250}},"nodeInfo":{"machineID":"7069d7465cb74ff59781bec038157837","systemUUID":"032b0290-0434-05d7-9c06-490700080009","bootID":"c03c0982-3180-4acf-bdeb-03584787bf41","kernelVersion":"4.19.12-1.el7.elrepo.x86_64","osImage":"CentOS Linux 7 (Core)","containerRuntimeVersion":"docker://18.6.3","kubeletVersion":"v1.13.6","kubeProxyVersion":"v1.13.6","operatingSystem":"linux","architecture":"amd64"}}},{"metadata":{"name":"192.168.0.43","selfLink":"/api/v1/nodes/192.168.0.43","uid":"5e5422a9-f182-11e9-84fc-902b34d396c8","resourceVersion":"1940342","creationTimestamp":"2019-10-18T08:36:24Z","labels":{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/os":"linux","kubernetes.io/hostname":"192.168.0.43","node-role.kubernetes.io/master":"node43"},"annotations":{"node.alpha.kubernetes.io/ttl":"0","volumes.kubernetes.io/controller-managed-attach-detach":"true"}},"spec":{"podCIDR":"10.20.3.0/24","taints":[{"key":"node-role.kubernetes.io/master","value":"node43","effect":"NoSchedule"}]},"status":{"capacity":{"cpu":"4","ephemeral-storage":"79833Mi","hugepages-2Mi":"0","memory":"32602520Ki","pods":"200"},"allocatable":{"cpu":"4","ephemeral-storage":"75339870903","hugepages-2Mi":"0","memory":"32500120Ki","pods":"200"},"conditions":[{"type":"MemoryPressure","status":"False","lastHeartbeatTime":"2019-10-28T09:02:10Z","lastTransitionTime":"2019-10-28T08:44:03Z","reason":"KubeletHasSufficientMemory","message":"kubelet has sufficient memory available"},{"type":"DiskPressure","status":"False","lastHeartbeatTime":"2019-10-28T09:02:10Z","lastTransitionTime":"2019-10-28T08:44:03Z","reason":"KubeletHasNoDiskPressure","message":"kubelet has no disk pressure"},{"type":"PIDPressure","status":"False","lastHeartbeatTime":"2019-10-28T09:02:10Z","lastTransitionTime":"2019-10-28T08:44:03Z","reason":"KubeletHasSufficientPID","message":"kubelet has sufficient PID available"},{"type":"Ready","status":"True","lastHeartbeatTime":"2019-10-28T09:02:10Z","lastTransitionTime":"2019-10-28T08:44:03Z","reason":"KubeletReady","message":"kubelet is posting ready status"}],"addresses":[{"type":"InternalIP","address":"192.168.0.43"},{"type":"Hostname","address":"192.168.0.43"}],"daemonEndpoints":{"kubeletEndpoint":{"Port":10250}},"nodeInfo":{"machineID":"fca78b295dc246ca8644f6fed6b145b0","systemUUID":"29c7c1c0-d7da-11dd-8c43-74d02bc70855","bootID":"9e5c2b9e-9cd5-4fc1-b197-3e83bcecbe4f","kernelVersion":"4.19.12-1.el7.elrepo.x86_64","osImage":"CentOS Linux 7 (Core)","containerRuntimeVersion":"docker://18.6.3","kubeletVersion":"v1.13.6","kubeProxyVersion":"v1.13.6","operatingSystem":"linux","architecture":"amd64"}}},{"metadata":{"name":"192.168.0.44","selfLink":"/api/v1/nodes/192.168.0.44","uid":"5e55144b-f182-11e9-84fc-902b34d396c8","resourceVersion":"1940332","creationTimestamp":"2019-10-18T08:36:24Z","labels":{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/os":"linux","kubernetes.io/hostname":"192.168.0.44","node-role.kubernetes.io/node":"node44"},"annotations":{"node.alpha.kubernetes.io/ttl":"0","volumes.kubernetes.io/controller-managed-attach-detach":"true"}},"spec":{"podCIDR":"10.20.4.0/24"},"status":{"capacity":{"cpu":"4","ephemeral-storage":"79833Mi","hugepages-2Mi":"0","memory":"32809284Ki","pods":"200"},"allocatable":{"cpu":"4","ephemeral-storage":"75339870903","hugepages-2Mi":"0","memory":"32706884Ki","pods":"200"},"conditions":[{"type":"MemoryPressure","status":"False","lastHeartbeatTime":"2019-10-28T09:02:05Z","lastTransitionTime":"2019-10-28T08:34:28Z","reason":"KubeletHasSufficientMemory","message":"kubelet has sufficient memory available"},{"type":"DiskPressure","status":"False","lastHeartbeatTime":"2019-10-28T09:02:05Z","lastTransitionTime":"2019-10-28T08:34:28Z","reason":"KubeletHasNoDiskPressure","message":"kubelet has no disk pressure"},{"type":"PIDPressure","status":"False","lastHeartbeatTime":"2019-10-28T09:02:05Z","lastTransitionTime":"2019-10-28T08:34:28Z","reason":"KubeletHasSufficientPID","message":"kubelet has sufficient PID available"},{"type":"Ready","status":"True","lastHeartbeatTime":"2019-10-28T09:02:05Z","lastTransitionTime":"2019-10-28T08:34:28Z","reason":"KubeletReady","message":"kubelet is posting ready status"}],"addresses":[{"type":"InternalIP","address":"192.168.0.44"},{"type":"Hostname","address":"192.168.0.44"}],"daemonEndpoints":{"kubeletEndpoint":{"Port":10250}},"nodeInfo":{"machineID":"3f08673aa85c4fbbbac93b9e7ff56d74","systemUUID":"03de0294-0480-0521-1506-470700080009","bootID":"d3a83738-9f77-453f-94bd-5cc42d3a001e","kernelVersion":"4.19.12-1.el7.elrepo.x86_64","osImage":"CentOS Linux 7 (Core)","containerRuntimeVersion":"docker://18.6.3","kubeletVersion":"v1.13.6","kubeProxyVersion":"v1.13.6","operatingSystem":"linux","architecture":"amd64"},"images":[{"names":["quay.io/external_storage/rbd-provisioner@sha256:94fd36b8625141b62ff1addfa914d45f7b39619e55891bad0294263ecd2ce09a","quay.io/external_storage/rbd-provisioner:latest"],"sizeBytes":404591002},{"names":["coredns/coredns@sha256:12eb885b8685b1b13a04ecf5c23bc809c2e57917252fd7b0be9e9c00644e8ee5","coredns/coredns:1.6.2"],"sizeBytes":44100963},{"names":["radial/busyboxplus@sha256:a68c05ab1112fd90ad7b14985a48520e9d26dbbe00cb9c09aa79fdc0ef46b372","radial/busyboxplus:curl"],"sizeBytes":4233788},{"names":["registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64@sha256:759c3f0f6493093a9043cc813092290af69029699ade0e3dbe024e968fcb7cca","registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:3.1"],"sizeBytes":742472}]}},{"metadata":{"name":"192.168.0.45","selfLink":"/api/v1/nodes/192.168.0.45","uid":"94b73d60-f182-11e9-a4e9-902b34d79c49","resourceVersion":"1940324","creationTimestamp":"2019-10-18T08:37:55Z","labels":{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/os":"linux","kubernetes.io/hostname":"192.168.0.45","node-role.kubernetes.io/node":"node45"},"annotations":{"node.alpha.kubernetes.io/ttl":"0","volumes.kubernetes.io/controller-managed-attach-detach":"true"}},"spec":{"podCIDR":"10.20.5.0/24"},"status":{"capacity":{"cpu":"4","ephemeral-storage":"79833Mi","hugepages-2Mi":"0","memory":"16296404Ki","pods":"200"},"allocatable":{"cpu":"4","ephemeral-storage":"75339870903","hugepages-2Mi":"0","memory":"16194004Ki","pods":"200"},"conditions":[{"type":"MemoryPressure","status":"False","lastHeartbeatTime":"2019-10-28T09:02:02Z","lastTransitionTime":"2019-10-28T08:30:42Z","reason":"KubeletHasSufficientMemory","message":"kubelet has sufficient memory available"},{"type":"DiskPressure","status":"False","lastHeartbeatTime":"2019-10-28T09:02:02Z","lastTransitionTime":"2019-10-28T08:30:42Z","reason":"KubeletHasNoDiskPressure","message":"kubelet has no disk pressure"},{"type":"PIDPressure","status":"False","lastHeartbeatTime":"2019-10-28T09:02:02Z","lastTransitionTime":"2019-10-28T08:30:42Z","reason":"KubeletHasSufficientPID","message":"kubelet has sufficient PID available"},{"type":"Ready","status":"True","lastHeartbeatTime":"2019-10-28T09:02:02Z","lastTransitionTime":"2019-10-28T08:30:42Z","reason":"KubeletReady","message":"kubelet is posting ready status"}],"addresses":[{"type":"InternalIP","address":"192.168.0.45"},{"type":"Hostname","address":"192.168.0.45"}],"daemonEndpoints":{"kubeletEndpoint":{"Port":10250}},"nodeInfo":{"machineID":"07a79cac7dcb4123a25f13c938dfc6f3","systemUUID":"032b0290-0434-05d8-3206-970700080009","bootID":"020b4ba4-6a0e-4968-a31e-837d03a1d71a","kernelVersion":"4.19.12-1.el7.elrepo.x86_64","osImage":"CentOS Linux 7 (Core)","containerRuntimeVersion":"docker://18.6.3","kubeletVersion":"v1.13.6","kubeProxyVersion":"v1.13.6","operatingSystem":"linux","architecture":"amd64"},"images":[{"names":["harbor.docker.zhiweireach.com/kubernetes/quay.io/external_storage/rbd-provisioner@sha256:b8ad79759bcd2dd289546bc350e3cf4cb2d7567ba9166e2ac57750a2277df4e9","quay.io/external_storage/rbd-provisioner@sha256:94fd36b8625141b62ff1addfa914d45f7b39619e55891bad0294263ecd2ce09a","harbor.docker.zhiweireach.com/kubernetes/quay.io/external_storage/rbd-provisioner:latest","quay.io/external_storage/rbd-provisioner:latest"],"sizeBytes":404591002},{"names":["registry.cn-hangzhou.aliyuncs.com/dnsjia/k8s@sha256:1d2e1229a918f4bc38b5a3f9f5f11302b3e71f8397b492afac7f273a0008776a","registry.cn-hangzhou.aliyuncs.com/dnsjia/k8s:kubernetes-dashboard-amd64_v1.10.0"],"sizeBytes":122460923},{"names":["coredns/coredns@sha256:12eb885b8685b1b13a04ecf5c23bc809c2e57917252fd7b0be9e9c00644e8ee5","coredns/coredns:1.6.2"],"sizeBytes":44100963},{"names":["gcr.azk8s.cn/google_containers/metrics-server-amd64@sha256:78938f933822856f443e6827fe5b37d6cc2f74ae888ac8b33d06fdbe5f8c658b","gcr.azk8s.cn/google_containers/metrics-server-amd64:v0.3.1"],"sizeBytes":40767713},{"names":["radial/busyboxplus@sha256:a68c05ab1112fd90ad7b14985a48520e9d26dbbe00cb9c09aa79fdc0ef46b372","radial/busyboxplus:curl"],"sizeBytes":4233788},{"names":["registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64@sha256:759c3f0f6493093a9043cc813092290af69029699ade0e3dbe024e968fcb7cca","registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:3.1"],"sizeBytes":742472}]}},{"metadata":{"name":"192.168.0.46","selfLink":"/api/v1/nodes/192.168.0.46","uid":"0c957804-f3d9-11e9-a4a3-74d02bc70855","resourceVersion":"1940337","creationTimestamp":"2019-10-21T08:01:55Z","labels":{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/os":"linux","edgenode":"true","kubernetes.io/hostname":"192.168.0.46","node-role.kubernetes.io/LB":"node46"},"annotations":{"node.alpha.kubernetes.io/ttl":"0","volumes.kubernetes.io/controller-managed-attach-detach":"true"}},"spec":{"podCIDR":"10.20.7.0/24","taints":[{"key":"edgenode","value":"true","effect":"NoSchedule"}]},"status":{"capacity":{"cpu":"4","ephemeral-storage":"84878856Ki","hugepages-2Mi":"0","memory":"32807696Ki","pods":"200"},"allocatable":{"cpu":"4","ephemeral-storage":"78224353561","hugepages-2Mi":"0","memory":"32705296Ki","pods":"200"},"conditions":[{"type":"MemoryPressure","status":"False","lastHeartbeatTime":"2019-10-28T09:02:09Z","lastTransitionTime":"2019-10-28T09:01:08Z","reason":"KubeletHasSufficientMemory","message":"kubelet has sufficient memory available"},{"type":"DiskPressure","status":"False","lastHeartbeatTime":"2019-10-28T09:02:09Z","lastTransitionTime":"2019-10-28T09:01:08Z","reason":"KubeletHasNoDiskPressure","message":"kubelet has no disk pressure"},{"type":"PIDPressure","status":"False","lastHeartbeatTime":"2019-10-28T09:02:09Z","lastTransitionTime":"2019-10-28T09:01:08Z","reason":"KubeletHasSufficientPID","message":"kubelet has sufficient PID available"},{"type":"Ready","status":"True","lastHeartbeatTime":"2019-10-28T09:02:09Z","lastTransitionTime":"2019-10-28T09:01:08Z","reason":"KubeletReady","message":"kubelet is posting ready status"}],"addresses":[{"type":"InternalIP","address":"192.168.0.46"},{"type":"Hostname","address":"192.168.0.46"}],"daemonEndpoints":{"kubeletEndpoint":{"Port":10250}},"nodeInfo":{"machineID":"ad0f2368b9b744b9b8c7e60e24e4d64a","systemUUID":"03de0294-0480-0502-a706-f30700080009","bootID":"1e7be4cb-ef6b-4be0-af94-d12c648b370d","kernelVersion":"4.19.12-1.el7.elrepo.x86_64","osImage":"CentOS Linux 7 (Core)","containerRuntimeVersion":"docker://18.6.3","kubeletVersion":"v1.13.6","kubeProxyVersion":"v1.13.6","operatingSystem":"linux","architecture":"amd64"},"images":[{"names":["traefik@sha256:20fd074599968976f5e06105a99b15de6d74d0b636ae5a8b760d81ee4c98f217","traefik:v1.7"],"sizeBytes":85586633},{"names":["registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64@sha256:759c3f0f6493093a9043cc813092290af69029699ade0e3dbe024e968fcb7cca","registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:3.1"],"sizeBytes":742472}]}},{"metadata":{"name":"192.168.0.47","selfLink":"/api/v1/nodes/192.168.0.47","uid":"5e39cc7b-f182-11e9-84fc-902b34d396c8","resourceVersion":"1940326","creationTimestamp":"2019-10-18T08:36:24Z","labels":{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/os":"linux","kubernetes.io/hostname":"192.168.0.47","node-role.kubernetes.io/node":"node47"},"annotations":{"node.alpha.kubernetes.io/ttl":"0","volumes.kubernetes.io/controller-managed-attach-detach":"true"}},"spec":{"podCIDR":"10.20.1.0/24"},"status":{"capacity":{"cpu":"4","ephemeral-storage":"92689184Ki","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"32810168Ki","pods":"200"},"allocatable":{"cpu":"4","ephemeral-storage":"85422351833","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"32707768Ki","pods":"200"},"conditions":[{"type":"MemoryPressure","status":"False","lastHeartbeatTime":"2019-10-28T09:02:03Z","lastTransitionTime":"2019-10-28T08:14:09Z","reason":"KubeletHasSufficientMemory","message":"kubelet has sufficient memory available"},{"type":"DiskPressure","status":"False","lastHeartbeatTime":"2019-10-28T09:02:03Z","lastTransitionTime":"2019-10-28T08:14:09Z","reason":"KubeletHasNoDiskPressure","message":"kubelet has no disk pressure"},{"type":"PIDPressure","status":"False","lastHeartbeatTime":"2019-10-28T09:02:03Z","lastTransitionTime":"2019-10-28T08:14:09Z","reason":"KubeletHasSufficientPID","message":"kubelet has sufficient PID available"},{"type":"Ready","status":"True","lastHeartbeatTime":"2019-10-28T09:02:03Z","lastTransitionTime":"2019-10-28T08:14:09Z","reason":"KubeletReady","message":"kubelet is posting ready status"}],"addresses":[{"type":"InternalIP","address":"192.168.0.47"},{"type":"Hostname","address":"192.168.0.47"}],"daemonEndpoints":{"kubeletEndpoint":{"Port":10250}},"nodeInfo":{"machineID":"fadf2695e72543ee939ec989564b0419","systemUUID":"f42d1b60-2302-11e6-9e84-3497f658b7c9","bootID":"1759ee65-42a5-4a8f-8f75-681995a0b6c9","kernelVersion":"4.19.12-1.el7.elrepo.x86_64","osImage":"CentOS Linux 7 (Core)","containerRuntimeVersion":"docker://18.6.3","kubeletVersion":"v1.13.6","kubeProxyVersion":"v1.13.6","operatingSystem":"linux","architecture":"amd64"},"images":[{"names":["coredns/coredns@sha256:12eb885b8685b1b13a04ecf5c23bc809c2e57917252fd7b0be9e9c00644e8ee5","coredns/coredns:1.6.2"],"sizeBytes":44100963},{"names":["gcr.azk8s.cn/google_containers/metrics-server-amd64@sha256:78938f933822856f443e6827fe5b37d6cc2f74ae888ac8b33d06fdbe5f8c658b","gcr.azk8s.cn/google_containers/metrics-server-amd64:v0.3.1"],"sizeBytes":40767713},{"names":["radial/busyboxplus@sha256:a68c05ab1112fd90ad7b14985a48520e9d26dbbe00cb9c09aa79fdc0ef46b372","radial/busyboxplus:curl"],"sizeBytes":4233788},{"names":["registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64@sha256:759c3f0f6493093a9043cc813092290af69029699ade0e3dbe024e968fcb7cca","registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:3.1"],"sizeBytes":742472}]}},{"metadata":{"name":"192.168.0.49","selfLink":"/api/v1/nodes/192.168.0.49","uid":"b99a9542-f224-11e9-a4a3-74d02bc70855","resourceVersion":"1940329","creationTimestamp":"2019-10-19T03:58:35Z","labels":{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/os":"linux","edgenode":"true","kubernetes.io/hostname":"192.168.0.49","node-role.kubernetes.io/LB":"node49"},"annotations":{"node.alpha.kubernetes.io/ttl":"0","volumes.kubernetes.io/controller-managed-attach-detach":"true"}},"spec":{"podCIDR":"10.20.6.0/24","taints":[{"key":"edgenode","value":"true","effect":"NoSchedule"}]},"status":{"capacity":{"cpu":"2","ephemeral-storage":"79833Mi","hugepages-2Mi":"0","memory":"16315300Ki","pods":"200"},"allocatable":{"cpu":"2","ephemeral-storage":"75339870903","hugepages-2Mi":"0","memory":"16212900Ki","pods":"200"},"conditions":[{"type":"MemoryPressure","status":"False","lastHeartbeatTime":"2019-10-28T09:02:04Z","lastTransitionTime":"2019-10-28T08:45:43Z","reason":"KubeletHasSufficientMemory","message":"kubelet has sufficient memory available"},{"type":"DiskPressure","status":"False","lastHeartbeatTime":"2019-10-28T09:02:04Z","lastTransitionTime":"2019-10-28T08:45:43Z","reason":"KubeletHasNoDiskPressure","message":"kubelet has no disk pressure"},{"type":"PIDPressure","status":"False","lastHeartbeatTime":"2019-10-28T09:02:04Z","lastTransitionTime":"2019-10-28T08:45:43Z","reason":"KubeletHasSufficientPID","message":"kubelet has sufficient PID available"},{"type":"Ready","status":"True","lastHeartbeatTime":"2019-10-28T09:02:04Z","lastTransitionTime":"2019-10-28T08:45:43Z","reason":"KubeletReady","message":"kubelet is posting ready status"}],"addresses":[{"type":"InternalIP","address":"192.168.0.49"},{"type":"Hostname","address":"192.168.0.49"}],"daemonEndpoints":{"kubeletEndpoint":{"Port":10250}},"nodeInfo":{"machineID":"c371d64c8cd64876b5ea8d919604a317","systemUUID":"03000200-0400-0500-0006-000700080009","bootID":"1ce6b758-e882-40a1-a79c-348b74b8a7d1","kernelVersion":"4.19.12-1.el7.elrepo.x86_64","osImage":"CentOS Linux 7 (Core)","containerRuntimeVersion":"docker://18.6.3","kubeletVersion":"v1.13.6","kubeProxyVersion":"v1.13.6","operatingSystem":"linux","architecture":"amd64"},"images":[{"names":["traefik@sha256:20fd074599968976f5e06105a99b15de6d74d0b636ae5a8b760d81ee4c98f217","traefik:v1.7"],"sizeBytes":85586633},{"names":["registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64@sha256:759c3f0f6493093a9043cc813092290af69029699ade0e3dbe024e968fcb7cca","registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:3.1"],"sizeBytes":742472}]}},{"metadata":{"name":"192.168.0.51","selfLink":"/api/v1/nodes/192.168.0.51","uid":"e8ed1a4c-f3da-11e9-84fc-902b34d396c8","resourceVersion":"1940325","creationTimestamp":"2019-10-21T08:15:14Z","labels":{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/os":"linux","edgenode":"true","kubernetes.io/hostname":"192.168.0.51","node-role.kubernetes.io/LB":"node51"},"annotations":{"node.alpha.kubernetes.io/ttl":"0","volumes.kubernetes.io/controller-managed-attach-detach":"true"}},"spec":{"podCIDR":"10.20.8.0/24","taints":[{"key":"edgenode","value":"true","effect":"NoSchedule"}]},"status":{"capacity":{"cpu":"4","ephemeral-storage":"82586216Ki","hugepages-2Mi":"0","memory":"32602700Ki","pods":"200"},"allocatable":{"cpu":"4","ephemeral-storage":"76111456540","hugepages-2Mi":"0","memory":"32500300Ki","pods":"200"},"conditions":[{"type":"MemoryPressure","status":"False","lastHeartbeatTime":"2019-10-28T09:02:02Z","lastTransitionTime":"2019-10-28T05:24:42Z","reason":"KubeletHasSufficientMemory","message":"kubelet has sufficient memory available"},{"type":"DiskPressure","status":"False","lastHeartbeatTime":"2019-10-28T09:02:02Z","lastTransitionTime":"2019-10-28T05:24:42Z","reason":"KubeletHasNoDiskPressure","message":"kubelet has no disk pressure"},{"type":"PIDPressure","status":"False","lastHeartbeatTime":"2019-10-28T09:02:02Z","lastTransitionTime":"2019-10-28T05:24:42Z","reason":"KubeletHasSufficientPID","message":"kubelet has sufficient PID available"},{"type":"Ready","status":"True","lastHeartbeatTime":"2019-10-28T09:02:02Z","lastTransitionTime":"2019-10-28T05:24:42Z","reason":"KubeletReady","message":"kubelet is posting ready status"}],"addresses":[{"type":"InternalIP","address":"192.168.0.51"},{"type":"Hostname","address":"192.168.0.51"}],"daemonEndpoints":{"kubeletEndpoint":{"Port":10250}},"nodeInfo":{"machineID":"0a7981f33ce6487da53328e0b39bc7aa","systemUUID":"4569c760-d83f-11dd-8dab-74d02bc70858","bootID":"4907b93c-41dc-42a4-a069-0a1f59b7d9e7","kernelVersion":"4.19.12-1.el7.elrepo.x86_64","osImage":"CentOS Linux 7 (Core)","containerRuntimeVersion":"docker://18.6.3","kubeletVersion":"v1.13.6","kubeProxyVersion":"v1.13.6","operatingSystem":"linux","architecture":"amd64"},"images":[{"names":["traefik@sha256:20fd074599968976f5e06105a99b15de6d74d0b636ae5a8b760d81ee4c98f217","traefik:v1.7"],"sizeBytes":85586633},{"names":["registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64@sha256:759c3f0f6493093a9043cc813092290af69029699ade0e3dbe024e968fcb7cca","registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:3.1"],"sizeBytes":742472}]}}]}
NAME           CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
192.168.0.47   64m          1%     564Mi           1%
192.168.0.41   <unknown>                           <unknown>               <unknown>               <unknown>
192.168.0.44   <unknown>                           <unknown>               <unknown>               <unknown>
192.168.0.49   <unknown>                           <unknown>               <unknown>               <unknown>
192.168.0.40   <unknown>                           <unknown>               <unknown>               <unknown>
192.168.0.43   <unknown>                           <unknown>               <unknown>               <unknown>
192.168.0.45   <unknown>                           <unknown>               <unknown>               <unknown>
192.168.0.46   <unknown>                           <unknown>               <unknown>               <unknown>
192.168.0.51   <unknown>                           <unknown>               <unknown>               <unknown>

The result shows that I got CPU, memory and other information of all nodes, but only node47 node is displayed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.