Missing metrics after switching to Cgroups v2 (need bump Cadvisor version)

[judexzhu]

What happened?

After upgrade Flatcar-Linux and switch to the cgroups v2, kubectl top node shows <>

What did you expect to happen?

kubectl top node shows metrics correctly

How can we reproduce it (as minimally and precisely as possible)?

switch cgroups to v2 and check https://<>:10250/stats/summary?only_cpu_and_memory=true

    "cpu": {
      "time": "2021-11-12T00:33:29Z",
      "usageNanoCores": 0,
      "usageCoreNanoSeconds": 0
    },
    "memory": {
      "time": "2021-11-12T00:33:29Z",
      "availableBytes": 16785580032,
      "usageBytes": 0,
      "workingSetBytes": 0,
      "rssBytes": 0,
      "pageFaults": 0,
      "majorPageFaults": 0

Anything else we need to know?

I run multiple standalone cadvisor with various versions and can confirmed

That 0.39.0+ fixed the cpu metrics and the 0.41.0+ fixed the memory issue

Kubernetes version

Details

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.11", GitCommit:"27522a29febbcc4badac257763044d0d90c11abd", GitTreeState:"clean", BuildDate:"2021-09-15T19:21:44Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.11", GitCommit:"27522a29febbcc4badac257763044d0d90c11abd", GitTreeState:"clean", BuildDate:"2021-09-15T19:16:25Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider

Details

Baremetal

OS version

Details

# On Linux:
$ cat /etc/os-release
NAME="Flatcar Container Linux by Kinvolk"
ID=flatcar
ID_LIKE=coreos
VERSION=2983.2.0
VERSION_ID=2983.2.0
BUILD_ID=2021-11-05-1835
PRETTY_NAME="Flatcar Container Linux by Kinvolk 2983.2.0 (Oklo)"
ANSI_COLOR="38;5;75"
HOME_URL="https://flatcar-linux.org/"
BUG_REPORT_URL="https://issues.flatcar-linux.org"
FLATCAR_BOARD="amd64-usr"
$ uname -a
Linux e11qk8sma03.neqdc.com 5.10.77-flatcar #1 SMP Fri Nov 5 17:49:48 -00 2021 x86_64 Common KVM processor GenuineIntel GNU/Linux

Install tools

Details

Container runtime (CRI) and and version (if applicable)

Details

docker 20.10.10

Related plugins (CNI, CSI, ...) and versions (if applicable)

Details

#106282

[judexzhu]

/sig Node

[ehashman]

/sig node
/area cadvisor

[SergeyKanzhelev]

/triage accepted
/assign @bobbypage

[judexzhu]

May I know if the cAdvisor 0.43 will be backported to the 1.19+ kubernetes? For v1.19 is the first version to support cgroup v2

[SergeyKanzhelev]

May I know if the cAdvisor 0.43 will be backported to the 1.19+ kubernetes? For v1.19 is the first version to support cgroup v2

We typically only cherry critical issues, see: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-release/cherry-picks.md#what-kind-of-prs-are-good-for-cherry-picks Also 1.19 is out of support in upstream. Also updating dependencies may not be possible on earlier versions, so unlikely 0.43 can be backported to 1.20, for example, as is. So if cAdvisor fix is needed, it will likely need to be implemented on cAdvisor earlier version tag.

How critical those metrics are? Would adopting of 1.23 be possible?

[ymmt2005]

As a workaround for older releases, we can deploy cAdvisor as a DaemonSet.
https://github.com/google/cadvisor/tree/master/deploy/kubernetes