[BUG] Important labels are missing in container file system usage metrics(from CAdvisor)

[saurabhvagrawal]

Describe the bug

We are currently running with AKS engine(1.23.12) clusters with dockershim as CRI. We can see container file system metrics coming good with all required labels. Example is below:

container_fs_usage_bytes{container="", container_name="", device="/dev/sdb1", endpoint="https-metrics", id="", image="", instance="", job="kubelet", name="", namespace="", node="", pod="", pod_name="", service=""}

Now we are moving towards managed AKS clusters(1.23.x) which uses containerd as CRI. However, there we don't see these labels associated with the metrics. Example is below:

container_fs_usage_bytes{device="/dev/sda1", endpoint="https-metrics", id="/", instance="", job="kubelet", node="", service="infra-monitoring-prometheu-kubelet"}

This is breaking our observability since we have metrics and monitors are defined for it and we do not have any alternative metrics as well. Please suggest how to move ahead.

To Reproduce
Steps to reproduce the behavior:

1. Open prometheus dashboard locally
2. Check for metrics associated with container file systems

Expected behavior
File system metrics should show all important labels.

There are some known issues mentioned on different forums for the same issue. Please refer below:

kubernetes/kubernetes#111777
google/cadvisor#2785 (comment)

Screenshots
NA. Provided all details in description.

Environment (please complete the following information):

• "azure-cli": "2.41.0",
• Kubernetes version [1.23.8]
• CLI Extension version [e.g. 1.7.5] if applicable
• Browser [edge]

Additional context
Add any other context about the problem here.

[ghost]

Action required from @Azure/aks-pm

[ghost]

Issue needing attention of @Azure/aks-leads

[ThomasEhling]

Hello, any news on this? We are getting the very same issue.

[justindavies]

@alexeldeib - do we have guidance for customer who have relied on dockershim on other providers for the FS metrics?

[ghost]

Action required from @Azure/aks-pm

[miwithro]

Hello,
This is a no solution upstream for ca advisor support as was linked to the issue. Until this issue is resolved there is nothing we can do on the AKS side.

[ghost]

Action required from @Azure/aks-pm

[duongneo]

Hi, I have same issue, please give me any news on this. Thanks

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service]

Issue needing attention of @Azure/aks-leads