Cadvisor metrics not present when using newer AKS Node image versions

[davidg-jellyfish]

Since upgrading to newer AKS Node image versions the cadvisor metrics have stopped reporting on container_cpu_cfs:

metrics no longer reporting:

container_cpu_cfs_periods_total
container_cpu_cfs_throttled_periods_total
container_cpu_cfs_throttled_seconds_total

If I log onto a container running on a newer worker node image I can see the cpu.cfs_quota_us set to -1

cat  /sys/fs/cgroup/cpu,cpuacct/cpu.cfs_quota_us
-1

If I log onto a container using an older worker node image running the same deployment and container spec the cpu metrics are visible:

cat  /sys/fs/cgroup/cpu,cpuacct/cpu.cfs_quota_us
50000 

AKS version 1.22
Node images:
AKSUbuntu-1804gen2containerd-2022.08.10 <-- containers running on these nodes have an issue
AKSUbuntu-1804gen2containerd-2022.07.04 <-- containers running on these nodes appear to report the container cfs cpu cadvisor metrics correctly

cadvisor_version_info{dockerVersion="Unknown", endpoint="https-metrics", job="kubelet", kernelVersion="5.4.0-1090-azure", metrics_path="/metrics/cadvisor", namespace="kube-system", osVersion="Ubuntu 18.04.6 LTS"}

cadvisor_version_info{dockerVersion="Unknown", endpoint="https-metrics", job="kubelet", kernelVersion="5.4.0-1089-azure", metrics_path="/metrics/cadvisor", namespace="kube-system", osVersion="Ubuntu 18.04.6 LTS"}

[ghost]

Action required from @Azure/aks-pm

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[aerott]

AKS v1.25.4 got same issue - no CPU, memory metrics available in Prometheus :(

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[davidg-jellyfish]

This isssue can be closed

I've found where the issue lies (in our terraform that defines the node we set a kubelet_config .  For every node pool where this has been set we are missing the CFS quota metrics.  If I log into one of the nodes and check the kubelet config, I can see the value as false

cat ./etc/default/kubeletconfig.json | grep CFS
    "cpuCFSQuota": false,
    cat ./etc/default/kubeletconfig.json | grep -i log
    "containerLogMaxSize": "100M",   <-- we are only changing this value but cpuCFSQuota is false, hence no CFS stats

For where we are not setting a kubelet_config there is no  kubeletconfig.json file so accepts the defaults as per the Azure configuration - https://learn.microsoft.com/en-us/azure/aks/custom-node-configuration meaning CFS quota is true and you get the stats.