no metrics known for pod #1149

ffxgamer · 2019-11-28T11:04:52Z

Version:

k3s version v1.0.0 (18bd921)

Describe the bug

pods metrics shows nothing.

root@k3s-1:/home/vagrant/Downloads/k3s/1.0_docker# kubectl top pods -A
root@k3s-1:/home/vagrant/Downloads/k3s/1.0_docker# kubectl top nodes
NAME    CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
k3s-1   61m          3%     944Mi           23%

metrics-server pod's logs showed:

1128 10:46:08.336882       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/svclb-traefik-qzs58: no metrics known for pod                                                        │
│ E1128 10:46:08.336886       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/coredns-d798c9dd-8qzvr: no metrics known for pod                                                     │
│ E1128 10:46:08.336888       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/metrics-server-6d684c7b5-mqrnm: no metrics known for pod                                             │
│ E1128 10:46:08.336891       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/local-path-provisioner-58fb86bdfd-nxtp5: no metrics known for pod                                    │
│ E1128 10:46:23.342457       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/svclb-traefik-qzs58: no metrics known for pod                                                        │
│ E1128 10:46:23.342646       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/coredns-d798c9dd-8qzvr: no metrics known for pod                                                     │
│ E1128 10:46:23.342738       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/metrics-server-6d684c7b5-mqrnm: no metrics known for pod                                             │
│ E1128 10:46:23.342817       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/local-path-provisioner-58fb86bdfd-nxtp5: no metrics known for pod                                    │
│ E1128 10:46:23.342872       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/traefik-65bccdc4bd-8t87c: no metrics known for pod                                                   │
│ E1128 10:46:38.347310       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/local-path-provisioner-58fb86bdfd-nxtp5: no metrics known for pod                                    │
│ E1128 10:46:38.347328       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/traefik-65bccdc4bd-8t87c: no metrics known for pod                                                   │
│ E1128 10:46:38.347332       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/svclb-traefik-qzs58: no metrics known for pod                                                        │
│ E1128 10:46:38.347335       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/coredns-d798c9dd-8qzvr: no metrics known for pod                                                     │
│ E1128 10:46:38.347337       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/metrics-server-6d684c7b5-mqrnm: no metrics known for pod                                             │

To Reproduce

Install k3s

# download install-k3s script
curl -sfL https://get.k3s.io > ./rancher/install-k3s.sh

# install dockerd
cd install_docker
./install_docker.sh
cd -

cp -rp ./bin/k3s /usr/local/bin/

export INSTALL_K3S_SKIP_DOWNLOAD=true
export INSTALL_K3S_EXEC="--docker"
./rancher/install-k3s.sh

Display pods' cpu and memory

kubectl top pods -A
<nothing>

Expected behavior

Display pods' cpu and memory values.

Like:

root@k3s-2:/home/vagrant/Downloads/k3s/0.9_docker# k3s -v
k3s version v0.9.0 (65d87648)
root@k3s-2:/home/vagrant/Downloads/k3s/0.9_docker# kubectl top pods -A
NAMESPACE     NAME                               CPU(cores)   MEMORY(bytes)
kube-system   coredns-66f496764-fwg9w            2m           6Mi
kube-system   docker-registry-59c76676fd-brj4n   1m           5Mi
kube-system   host-path-provisioner-n9xkw        1m           6Mi
kube-system   metrics-server-5d9bd54785-clrwb    1m           12Mi
kube-system   minio-54b65b5dc7-s7dbl             1m           8Mi
kube-system   postgres-8549f56f76-ntbbp          1m           37Mi
kube-system   svclb-traefik-6c8w7                0m           1Mi
kube-system   traefik-d869575c8-6dg44            3m           15Mi

Actual behavior

kubectl top pods -A showed nothing

Additional context

environment
I started a virtualbox with vagrant

cat Vagrantfile
# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
	(1..1).each do |i|
		config.vm.define "k3s-#{i}" do |node|
		# 设置虚拟机的Box
                node.vm.box = "bento/ubuntu-18.04"
		# 设置虚拟机的主机名
		node.vm.hostname="k3s-#{i}"
		# 设置虚拟机的IP
		node.vm.network "private_network", ip: "192.168.33.2#{i}"
                #node.vm.network "public_network", ip: "10.0.23.24#{i}", bridge: "en0: Wi-Fi (Wireless)"
		# 设置主机与虚拟机的共享目录
		#node.vm.synced_folder "~/Desktop/share", "/home/vagrant/share"
		node.vm.synced_folder "~/Downloads", "/home/vagrant/Downloads"
		#node.vm.synced_folder "/Volumes/T5SSD", "/home/vagrant/T5SSD"
		# VirtaulBox相关配置
		node.vm.provider "virtualbox" do |v|
			# 设置虚拟机的名称
			v.name = "k3s-#{i}"
			# 设置虚拟机的内存大小
			v.memory = 4096
			# 设置虚拟机的CPU个数
			v.cpus = 2
		end
		# 使用shell脚本进行软件安装和配置
		node.vm.provision "shell", inline: <<-SHELL
			# 安装docker 1.11.0
			#wget -qO- https://get.docker.com/ | sed 's/docker-engine/docker-engine=1.11.0-0~trusty/' | sh
			#usermod -aG docker vagrant"
                        #route add default gw 192.168.33.1
                        #eval `route -n | awk '{ if ($8 ==\"eth0\" && $2 != \"0.0.0.0\") print \"route del default gw \" $2; }'`
		SHELL
		end
	end
end

And start virtual machine with vagrant up

Network route

root@k3s-1:/home/vagrant/Downloads/k3s/1.0_docker# ip route
default via 10.0.2.2 dev eth0 proto dhcp src 10.0.2.15 metric 100
10.0.2.0/24 dev eth0 proto kernel scope link src 10.0.2.15
10.0.2.2 dev eth0 proto dhcp scope link src 10.0.2.15 metric 100
10.42.0.0/24 dev cni0 proto kernel scope link src 10.42.0.1
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
192.168.33.0/24 dev eth1 proto kernel scope link src 192.168.33.21

Pods status

root@k3s-1:/home/vagrant/Downloads/k3s/1.0_docker# kubectl get pods -A
NAMESPACE     NAME                                      READY   STATUS      RESTARTS   AGE
kube-system   local-path-provisioner-58fb86bdfd-nxtp5   1/1     Running     0          36m
kube-system   metrics-server-6d684c7b5-mqrnm            1/1     Running     0          36m
kube-system   coredns-d798c9dd-8qzvr                    1/1     Running     0          36m
kube-system   helm-install-traefik-ftb58                0/1     Completed   0          36m
kube-system   svclb-traefik-qzs58                       3/3     Running     0          33m
kube-system   traefik-65bccdc4bd-8t87c                  1/1     Running     0          33m

The text was updated successfully, but these errors were encountered:

wglambert · 2019-12-05T22:13:08Z

Duplicate of #1067

handong0123 · 2019-12-07T06:21:22Z

I also encountered the same problem.
I can only view the data of node.
log： error no metrics known for pod

ofirmakmal · 2019-12-10T10:40:52Z

Anyone knows if there's any progress in finding the cause? or maybe a workaround?

ofirmakmal · 2019-12-10T12:13:49Z

I just found out that the issue happens (to me at least) only when running in docker mode.
export INSTALL_K3S_EXEC="--docker"

Otherwise it works.

The problem is, that I must use Docker mode... :(

rookieclover · 2020-01-06T13:26:13Z

I just found out that the issue happens (to me at least) only when running in docker mode.
export INSTALL_K3S_EXEC="--docker"

Otherwise it works.

The problem is, that I must use Docker mode... :(

dot you fix this problem?

estimate123 · 2020-03-24T10:37:45Z

I just found out that the issue happens (to me at least) only when running in docker mode.
export INSTALL_K3S_EXEC="--docker"

Otherwise it works.

The problem is, that I must use Docker mode... :(

did you fix the problem?

JoseThen · 2020-03-29T20:39:52Z

For those looking through this may help: kubernetes-sigs/metrics-server#349

ffxgamer · 2020-08-24T07:08:51Z

Finnally I solved this problem with new version 1.18_k3s.
In addition to using 1.18 vesion of k3s and docker, I fixed a mistake of coredns config file.
Looks like below:

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          upstream
          fallthrough in-addr.arpa ip6.arpa
        }
        hosts /etc/coredns/NodeHosts {

          reload 1s
          fallthrough
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
  NodeHosts: |
          10.0.2.15 host-name registry.local

NodeHosts must be configued as $ip_address $(hostname)

After modifying the above two points, the values of metrics can be displayed correctly

root@k3s-1:/home/vagrant/d/k3s/1.18_docker# kubectl top pods -A
NAMESPACE     NAME                                     CPU(cores)   MEMORY(bytes)
kube-system   coredns-595b6c5976-kk6vg                 3m           7Mi
kube-system   docker-registry-5fd6b96cf5-vf6zj         1m           7Mi
kube-system   docker-registry-ui-69f5878bb5-8vbp8      1m           4Mi
kube-system   local-path-provisioner-6d59f47c7-v52dw   3m           7Mi
kube-system   metrics-server-7566d596c8-7t552          1m           11Mi
kube-system   minio-6d9ccf658f-r5t9p                   1m           10Mi
kube-system   postgres-77c8b84656-pncmn                1m           34Mi

But this modification method does not work on version 1.17 and 0.09.

brandond · 2020-08-24T17:23:53Z

k3s has a built-in controller that manages the NodeHosts ConfigMap entries for you. It will add an entry for each cluster member. What did you change?

ffxgamer · 2020-08-26T07:27:24Z

k3s has a built-in controller that manages the NodeHosts ConfigMap entries for you. It will add an entry for each cluster member. What did you change?

( I'm sorry, my English is not good )

I have a private docker-registry service in k8s for images downloads service.
So I create a domain name 'registry.local' 'for docker-registry's ingress.
Because this is a offline environment, there's no dns server in this LAN.
so I changed the configmap of coredns yaml like below:

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          upstream
          fallthrough in-addr.arpa ip6.arpa
        }
        hosts /etc/coredns/NodeHosts {

          reload 1s
          fallthrough
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
  NodeHosts: |
          10.0.2.15 k3s-1 registry.local

I used this configmap, deployment and service merge into a coredns yaml file
and put it into /var/lib/rancher/server/manifests/ .

I wrote a bootstrap script like this for change ip and install customized allinone k3s

...
host_if=$(ip route|grep default|cut -d' ' -f5)
host_ip=$(ip a|grep $host_if|awk 'NR==2{print $2}'|cut -d'/' -f1)
sed -i "s/^.*registry.local$/          ${host_ip} $HOSTNAME registry.k8s.com/g" ./rancher/k3s/server/manifests/coredns.yaml

echo '# install k3s'
export INSTALL_K3S_SKIP_DOWNLOAD=true
export INSTALL_K3S_BIN_DIR=/usr/bin
export INSTALL_K3S_EXEC="--kube-apiserver-arg service-node-port-range=20000-40000 \
			 --kubelet-arg max-pods=500 \
			 --flannel-iface $host_if \
			 --no-deploy coredns \
			 --data-dir  $k3s_path \
			 --docker"
./install-k3s.sh
...

So I disabled coredns deploy and put my revised version coredns deploy.
That caused part of the problem.

ffxgamer mentioned this issue Dec 9, 2019

Unable to get pod metrics #1067

Closed

ffxgamer closed this as completed Oct 16, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

no metrics known for pod #1149

no metrics known for pod #1149

ffxgamer commented Nov 28, 2019

wglambert commented Dec 5, 2019

handong0123 commented Dec 7, 2019

ofirmakmal commented Dec 10, 2019

ofirmakmal commented Dec 10, 2019

rookieclover commented Jan 6, 2020

estimate123 commented Mar 24, 2020

JoseThen commented Mar 29, 2020

ffxgamer commented Aug 24, 2020 •

edited

Loading

brandond commented Aug 24, 2020

ffxgamer commented Aug 26, 2020

no metrics known for pod #1149

no metrics known for pod #1149

Comments

ffxgamer commented Nov 28, 2019

wglambert commented Dec 5, 2019

handong0123 commented Dec 7, 2019

ofirmakmal commented Dec 10, 2019

ofirmakmal commented Dec 10, 2019

rookieclover commented Jan 6, 2020

estimate123 commented Mar 24, 2020

JoseThen commented Mar 29, 2020

ffxgamer commented Aug 24, 2020 • edited Loading

brandond commented Aug 24, 2020

ffxgamer commented Aug 26, 2020

ffxgamer commented Aug 24, 2020 •

edited

Loading