Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Archive hollow node logs in (pre-submit) kubemark tests #41613

Closed
yujuhong opened this issue Feb 17, 2017 · 8 comments
Closed

Archive hollow node logs in (pre-submit) kubemark tests #41613

yujuhong opened this issue Feb 17, 2017 · 8 comments
Assignees
@shyamjvs
Labels
area/test

Comments

@yujuhong
Copy link
Contributor

The logs are currently not uploaded to GCS. Debugging kubemark requires running the test yourself or ssh'ing into a jenkins cluster running kubemark. Can we collect the logs to facilitate debugging?

/cc @wojtek-t @gmarek @shyamjvs
@gmarek
Copy link
Contributor

gmarek commented Feb 17, 2017

Yes we can - we know that we don't do it. It was an accident at the beginning, but we decided that's a lucky one, as we're not sure how our test env will survive downloading logs from 5000 hollow nodes. I guess we can switch it on for some time.

@gmarek
Copy link
Contributor

gmarek commented Feb 17, 2017

Ideally it would be turned on by the flag and enabled only in pre-submit kubemark (+ maybe in some of the smaller ones)

@wojtek-t
Copy link
Member

Yup - this should be pretty simple to switch on. And I agree we should first start with enabling it only for small kubemarks (e.g. <=100 nodes)

@yujuhong
Copy link
Contributor Author

I think it should definitely be enabled for presubmit. The next to consider is the submit queue blocking "ci-kubernetes-kubemark-500-gce" suite, but that depends on how often it fails.

@wojtek-t
Copy link
Member

+1 for presubmit

The 500-node one is super stable - e.g. all the failures from last week were regressions.

@yujuhong
Copy link
Contributor Author

The 500-node one is super stable - e.g. all the failures from last week were regressions.

Agreed. Hopefully the presubmit tests will catch most of the regressions from now on.

@shyamjvs
Copy link
Member

I'll look into enabling the hollow-node logs (both kubelet and proxy) for presubmit and 100-node tests then. Seems like that should suffice for now from above discussion.

This was referenced Feb 20, 2017
Merged
Merged
k8s-github-robot pushed a commit that referenced this issue Feb 21, 2017
Merge pull request #41739 from shyamjvs/hollow-node-logs
70c9eeb 
Automatic merge from submit-queue (batch tested with PRs 41706, 39063, 41330, 41739, 41576)

[Kubemark] Add option to log hollow-node logs

Ref #41613

Added an option to log kubemark hollow-node logs which includes kubelet, kubeproxy and npd logs for each hollow-node.
Setting the env var `ENABLE_HOLLOW_NODE_LOGS=true` should now enable logging for tests.

cc @kubernetes/sig-scalability-misc @wojtek-t @gmarek @yujuhong @Random-Liu
@shyamjvs
Copy link
Member

Closing this issue as hollow-node logs are getting artifacted after PR #41800 has been merged.
We can reopen this if we have problems around hollow-node logging in future.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@shyamjvs shyamjvs

Labels
area/test
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@gmarek @shyamjvs @yujuhong @wojtek-t