New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kubelet /stats/summary returns empty pods list - disabling HPA #69658

Open
aalubin opened this Issue Oct 11, 2018 · 1 comment

Comments

Projects
None yet
3 participants
@aalubin

aalubin commented Oct 11, 2018

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

/sig node
/area kubelet
/area kubelet-api

What happened:
We are trying to migrate from docker to cri-o for container runtime

When querying the kubelet for /stats/summary (curl localhost:10255/stats/summary)
the pods information is an empty list although pods are running on the node, the general server information is available.
This is an example of the end of the output:
.
.
.
},
"rlimit": {
"time": "2018-10-11T06:38:33Z",
"maxpid": 40960,
"curproc": 1972
}
},
"pods": []
}

What you expected to happen:
statics information on pod

How to reproduce it (as minimally and precisely as possible):
not sure, here is our kubelet configuration:
xecStart=/usr/bin/kubelet --allow-privileged=true
--cadvisor-port=4194
--cgroup-driver=cgroupfs
--client-ca-file=/etc/pki/kubernetes_certs/ca.pem
--cluster-dns=10.10.10.2
--cluster-domain=customdomain.com
--cni-bin-dir=/opt/cni/bin
--cni-conf-dir=/etc/cni/net.d
--container-runtime-endpoint=unix:///var/run/crio/crio.sock
--container-runtime=remote
--cpu-manager-policy=none
--feature-gates=TaintBasedEvictions=true
--image-gc-high-threshold=70
--image-gc-low-threshold=50
--kube-reserved-cgroup=podruntime.slice
--kube-reserved=cpu=1,memory=3Gi
--kubeconfig=/etc/kubernetes/kubelet_kubeconfig
--kubelet-cgroups=/podruntime.slice
--network-plugin=cni
--node-labels=configure_bgp_peers=true
--pod-manifest-path=/etc/kubernetes/manifests
--register-node=true
--register-with-taints=new_node=true:NoSchedule
--runtime-cgroups=/podruntime.slice
--runtime-request-timeout=10m
--system-reserved-cgroup=system.slice
--system-reserved=cpu=1,memory=768Mi
--tls-cert-file=/etc/pki/kubernetes_certs/k24.customdomain.com_cert.pem
--tls-private-key-file=/etc/pki/kubernetes_certs/k24.customdomain.com_key.pem

Anything else we need to know?:
We've got the same results with mix of the following configuration:

  • kubelet 1.10.7 and 1.11.3
  • cri-o 1.10.5 and 1.11.6

Another issue that we have that I think is the result of this is that crictl stats does not return any information

Environment:

  • Kubernetes version (use kubectl version): 1.11.3
  • Cloud provider or hardware configuration: on premise Dell 630/640 servers
  • OS (e.g. from /etc/os-release): centos 7
  • Kernel (e.g. uname -a): Linux k24.customdomain.com 4.14.35-1818.2.1.el7uek.x86_64 #2 SMP Mon Aug 27 21:16:31 PDT 2018 x86_64 x86_64 x8 6_64 GNU/Linux
  • Install tools: puppet automation
  • Others:

@aalubin aalubin changed the title from /stats/summary return empty pods list - disabling HPA to kubelet /stats/summary returns empty pods list - disabling HPA Oct 11, 2018

@k8s-ci-robot k8s-ci-robot added sig/node and removed needs-sig labels Oct 11, 2018

@WanLinghao

This comment has been minimized.

Show comment
Hide comment
@WanLinghao
Contributor

WanLinghao commented Oct 15, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment