Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve cadvisor (integration) tests #55398

Closed
discordianfish opened this issue Nov 9, 2017 · 13 comments
Closed

Improve cadvisor (integration) tests #55398

discordianfish opened this issue Nov 9, 2017 · 13 comments
Labels
lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. sig/node Categorizes an issue or PR as relevant to SIG Node.

Comments

@discordianfish
Copy link
Contributor

/kind feature

Every other kubernetes release the metrics provided by cadvisor seem to break. This not only just happened again in #55397 but also earlier in #48483, #33192 and #25131.

These issue usually not only end up in the stable releases but even on cloud providers like GKE where these are sometimes the only supported versions.

I think this should be prevented by improving the cadvisor integration tests (if there are any yet). They should at least check that cadvisor successfully exposes plausible metrics for mem, cpu, disk io, filesystem and networking.

@k8s-github-robot k8s-github-robot added the needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Nov 9, 2017
@dims
Copy link
Member

dims commented Nov 13, 2017

/sig node

@k8s-ci-robot k8s-ci-robot added the sig/node Categorizes an issue or PR as relevant to SIG Node. label Nov 13, 2017
@k8s-github-robot k8s-github-robot removed the needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Nov 13, 2017
@discordianfish
Copy link
Contributor Author

In the meanwhile I ran into the next metrics related regression. This time it's not cadvisor so maybe the scope should be more integration/regression testing for metrics in general.
This time it's #52121 and #53485

@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Feb 13, 2018
@discordianfish
Copy link
Contributor Author

/remove-lifecycle stale
/lifecycle freeze

@k8s-ci-robot k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Feb 13, 2018
@discordianfish
Copy link
Contributor Author

@brancz Is that something you can get someone to look into?

@brancz
Copy link
Member

brancz commented Feb 13, 2018

Unlikely, personally cAdvisor has been so unreliable for Prometheus metrics that we try not to rely on it and are even looking into getting the stats differently. Aside from that I would say this mostly belongs in the cAdvisor repo.

@discordianfish
Copy link
Contributor Author

@brancz No need to stablized cadvisor or improve that project itself, I was more thinking about better testing when bumping cadvisor version. And it's not cadvisor specific but something that applies to other metric related changes too (e.g #52121 and #53485). It just would be great to get people to pay more attention on metric changes in PRs.

@brancz
Copy link
Member

brancz commented Feb 14, 2018

Understood. I don't have a great answer for you, yes I would love to, but don't have the resources to do this. Maybe it's a topic you can bring up to the CNCF CI working group? They already test Prometheus on Kubernetes, so maybe it's something they could look into (we're also working with them on some unrelated Prometheus CI things, maybe I can bring it up in there).

@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label May 15, 2018
@discordianfish
Copy link
Contributor Author

/remove-lifecycle stale
/lifecycle freeze

@k8s-ci-robot k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label May 15, 2018
@techtonik
Copy link

So what are specific metrics that should be collected for now to close this issue? I guess the rest of metrics could be picked up when more issue arise.

@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 3, 2018
@discordianfish
Copy link
Contributor Author

Not going to fight that bot anymore.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. sig/node Categorizes an issue or PR as relevant to SIG Node.
Projects
None yet
Development

No branches or pull requests

7 participants