Improve cadvisor (integration) tests

[discordianfish]

/kind feature

Every other kubernetes release the metrics provided by cadvisor seem to break. This not only just happened again in #55397 but also earlier in #48483, #33192 and #25131.

These issue usually not only end up in the stable releases but even on cloud providers like GKE where these are sometimes the only supported versions.

I think this should be prevented by improving the cadvisor integration tests (if there are any yet). They should at least check that cadvisor successfully exposes plausible metrics for mem, cpu, disk io, filesystem and networking.

[dims]

/sig node

[discordianfish]

In the meanwhile I ran into the next metrics related regression. This time it's not cadvisor so maybe the scope should be more integration/regression testing for metrics in general.
This time it's #52121 and #53485

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[discordianfish]

/remove-lifecycle stale
/lifecycle freeze

[discordianfish]

@brancz Is that something you can get someone to look into?

[brancz]

Unlikely, personally cAdvisor has been so unreliable for Prometheus metrics that we try not to rely on it and are even looking into getting the stats differently. Aside from that I would say this mostly belongs in the cAdvisor repo.

[discordianfish]

@brancz No need to stablized cadvisor or improve that project itself, I was more thinking about better testing when bumping cadvisor version. And it's not cadvisor specific but something that applies to other metric related changes too (e.g #52121 and #53485). It just would be great to get people to pay more attention on metric changes in PRs.

[brancz]

Understood. I don't have a great answer for you, yes I would love to, but don't have the resources to do this. Maybe it's a topic you can bring up to the CNCF CI working group? They already test Prometheus on Kubernetes, so maybe it's something they could look into (we're also working with them on some unrelated Prometheus CI things, maybe I can bring it up in there).

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale