[BUG] Azure Defender for Containers memory metrics reading error

[mdanylyuk]

Describe the bug
From Jul 27 Defender pods stopped to read memory metrics because of changes cgroup to V2.

Now microsoft-defender-collector-ds pods spam messages

time="2023-08-11T06:55:06Z" level=error msg="Failed to get memory usage AnisTest open /sys/fs/cgroup/memory/memory.usage_in_bytes: no such file or directory"
time="2023-08-11T06:55:06Z" level=info msg="Heartbeat: {\"Performance\":{\"Memory\":\"Unknown\"}}"

Expected behavior
Reading the correct system files

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

• Kubernetes version - 1.26.3
• Pod image - mcr.microsoft.com/azuredefender/stable/low-level-collector:1.3.78
• Kubernetes node OS version - Ubuntu 22.04.2 LTS
• Kubernetes kernel version - 5.15.0-1041-azure

Additional context
Add any other context about the problem here.

[justindavies]

@miwithro - do we have and ETA on a fix for this

[mdanylyuk]

Hey guys,
Is there any news about upgrading azuredefender/stable/low-level-collector?

[EvolutionOli]

Also hitting this issue and it's generating 300m load on the cluster

[kaarthis]

@miwithro , @CocoWang-wql can we please help with next steps here.

[mdanylyuk]

Guys,
Are there any updates?
1 month without answers

[Srokap]

Seeing the same thing after enabling Defender for Containers

time="2023-09-22T19:59:23Z" level=error msg="Failed to get memory usage AnisTest open /sys/fs/cgroup/memory/memory.usage_in_bytes: no such file or directory"
time="2023-09-22T19:59:23Z" level=info msg="Heartbeat: {\"Performance\":{\"Memory\":\"Unknown\"}}"

• Kubernetes version - 1.26.3
• Pod image - mcr.microsoft.com/azuredefender/stable/low-level-collector:1.3.78
• Kubernetes node OS version - Ubuntu 22.04.2 LTS
• Kubernetes kernel version - 5.15.0-1039-azure

Inspecting one of the nodes, I can see 3 files

-r--r--r-- 1 root root 0 Sep 22 20:15 /sys/fs/cgroup/memory.numa_stat
-rw-r--r-- 1 root root 0 Sep 22 20:15 /sys/fs/cgroup/memory.pressure
-r--r--r-- 1 root root 0 Sep 22 20:01 /sys/fs/cgroup/memory.stat

One thing I noticed, is that kubernetes/kubernetes#118916 mentions memory.usage_in_bytes being part of cgroups v1. From what I can tell cgroups v2 is part of the kernel since 4.5. Perhaps Defender is still expecting cgroups v1 files that might no longer be there?

[mdanylyuk]

Guys,
Can you tell us when it is fixed?
The problem is still here

[Elenifr]

Also hitting the same issue on AKS 1.27.3.

[mdanylyuk]

Hey guys
Do you have a plan to fix it?

[mdanylyuk]

Hello,
As I see - problem is solved and ticket can be closed

[MisterKCloud]

+1
Have the same issue!

Kubernetes version - 1.27.3
Pod image - mcr.microsoft.com/azuredefender/stable/low-level-collector:1.3.81

[kbocock-krg]

k8s: 1.27.7
Same issue.

[microsoft-github-policy-service]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service]

Issue needing attention of @Azure/aks-leads

[mdanylyuk]

For me issue is fixed.