Kubelet becomes unresponsive - Parca Server v0.21.0, EKS v1.28, Bottlerocket vLinux v6.1.77

[gerhard]

Expectation

I think of this as an async debugging session that we can learn from, and maybe use for a future Let's Profile episode. There is no urgency to get this resolved, I am primarily trying to learn the process of debugging a Parca Server issue - at least that's what it seems to be, we didn't see this issue before installing Parca Server.

What is the issue?

After a few days of running Parca Server, the Kubelet becomes unresponsive. We've hit this issue on 2 separate EKS cluster. We suspect that this is related to the instance type, t3a.large.

Our setup

• EKS v1.28
• Bottlerocket Linux v6.1.77
• t3a.large
• Parca Server v0.21.0 (default manifest, no configs)

Next steps

• Switch from t3a.large to m7a.large
• Provide more details when this happens again

For the last point, what details would help? How can we collect them? FWIW, I couldn't find any Parca Server troubleshooting docs on https://www.parca.dev/docs/parca.

[gerhard]

Anything else to add @matipan?