Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[k8s.io] Kubelet [Serial] [Slow] [k8s.io] regular resource usage tracking resource tracking for 35 pods per node {Kubernetes e2e suite} #32942

Closed
k8s-github-robot opened this issue Sep 17, 2016 · 21 comments · Fixed by #36623
Closed

[k8s.io] Kubelet [Serial] [Slow] [k8s.io] regular resource usage tracking resource tracking for 35 pods per node {Kubernetes e2e suite} #32942

k8s-github-robot opened this issue Sep 17, 2016 · 21 comments · Fixed by #36623
Assignees
@yujuhong
Labels
kind/flake Categorizes issue or PR as related to a flaky test. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.
Milestone
v1.5

Comments

@k8s-github-robot
Copy link

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-serial/2273/

Failed: [k8s.io] Kubelet [Serial] [Slow] [k8s.io] regular resource usage tracking resource tracking for 35 pods per node {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubelet_perf.go:278
Sep 16 22:17:09.002: CPU usage exceeding limits:
 node gke-jenkins-e2e-default-pool-6b926500-7gt3:
 container "kubelet": expected 50th% usage < 0.120; got 0.127, container "kubelet": expected 95th% usage < 0.140; got 0.219
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubelet_perf.go:187

Previous issues for this test: #28220
@k8s-github-robot k8s-github-robot added kind/flake Categorizes issue or PR as related to a flaky test. priority/backlog Higher priority than priority/awaiting-more-evidence. labels Sep 17, 2016
@madhusudancs
Copy link
Contributor

Looks like a duplicate of #28220. Assigning it to @dchen1107 for further triaging.

@davidopp davidopp mentioned this issue Sep 17, 2016
Closed
@davidopp
Copy link
Member

davidopp commented Sep 18, 2016 •
edited
Loading

failing in
https://k8s-testgrid.appspot.com/google-1.2-1.4-upgrade#gke-cvm-1.2-gci-1.4-upgrade-cluster
https://k8s-testgrid.appspot.com/google-1.2-1.4-upgrade#gke-gci-1.2-gci-1.4-upgrade-cluster
https://k8s-testgrid.appspot.com/google-1.3-1.4-upgrade#gke-cvm-1.3-gci-1.4-upgrade-cluster
https://k8s-testgrid.appspot.com/google-1.3-1.4-upgrade#gke-cvm-1.3-gci-1.4-upgrade-master
https://k8s-testgrid.appspot.com/google-1.3-1.4-upgrade#gke-gci-1.3-cvm-1.4-upgrade-master
https://k8s-testgrid.appspot.com/google-1.3-1.4-upgrade#gke-gci-1.3-gci-1.4-upgrade-master

@davidopp davidopp added this to the v1.4 milestone Sep 18, 2016
@k8s-github-robot k8s-github-robot mentioned this issue Sep 18, 2016
Closed
@davidopp
Copy link
Member

dup #32703

@davidopp
Copy link
Member

sorry, not a dup

@davidopp davidopp reopened this Sep 18, 2016
This was referenced Sep 19, 2016
Closed
Closed
Closed
Closed
@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-test/3/

Failed: [k8s.io] Kubelet [Serial] [Slow] [k8s.io] regular resource usage tracking resource tracking for 35 pods per node {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubelet_perf.go:277
Sep 22 23:17:16.369: Memory usage exceeding limits:
 node gke-jenkins-e2e-default-pool-3987555c-a8z8:
 container "runtime": expected RSS memory (MB) < 157286400; got 240611328
node gke-jenkins-e2e-default-pool-3987555c-cutg:
 container "runtime": expected RSS memory (MB) < 157286400; got 252616704
node gke-jenkins-e2e-default-pool-3987555c-mzyr:
 container "runtime": expected RSS memory (MB) < 157286400; got 242028544

This was referenced Sep 23, 2016
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-soak-continuous-e2e-gke-gci/229/

Failed: [k8s.io] Kubelet [Serial] [Slow] [k8s.io] regular resource usage tracking resource tracking for 35 pods per node {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubelet_perf.go:279
Oct 19 03:48:43.555: Memory usage exceeding limits:
 node gke-jenkins-e2e-default-pool-f0421644-cv4y:
 container "kubelet": expected RSS memory (MB) < 73400320; got 75268096
node gke-jenkins-e2e-default-pool-f0421644-fv0f:
 container "kubelet": expected RSS memory (MB) < 73400320; got 74674176
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubelet_perf.go:154

@k8s-github-robot k8s-github-robot mentioned this issue Oct 21, 2016
Closed
@k8s-github-robot
Copy link
Author

[FLAKE-PING] @dchen1107

This flaky-test issue would love to have more attention.

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gce-serial/212/

Failed: [k8s.io] Kubelet [Serial] [Slow] [k8s.io] regular resource usage tracking resource tracking for 35 pods per node {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubelet_perf.go:279
Oct 23 09:30:35.642: CPU usage exceeding limits:
 node jenkins-e2e-minion-group-yrg9:
 container "kubelet": expected 95th% usage < 0.140; got 0.142
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubelet_perf.go:188

@k8s-github-robot
Copy link
Author

[FLAKE-PING] @dchen1107

This flaky-test issue would love to have more attention.

This was referenced Oct 29, 2016
Closed
Closed
Closed
Closed
Closed
Closed
@k8s-github-robot k8s-github-robot mentioned this issue Nov 6, 2016
Closed
@k8s-github-robot
Copy link
Author

[FLAKE-PING] @dchen1107

This flaky-test issue would love to have more attention.

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-serial/283/

Failed: [k8s.io] Kubelet [Serial] [Slow] [k8s.io] regular resource usage tracking resource tracking for 35 pods per node {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubelet_perf.go:279
Nov  7 18:47:44.431: CPU usage exceeding limits:
 node gke-jenkins-e2e-default-pool-77c5b216-jala:
 container "kubelet": expected 95th% usage < 0.160; got 0.180
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubelet_perf.go:188

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gce-serial/300/

Failed: [k8s.io] Kubelet [Serial] [Slow] [k8s.io] regular resource usage tracking resource tracking for 35 pods per node {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubelet_perf.go:279
Nov  7 20:00:35.832: CPU usage exceeding limits:
 node jenkins-e2e-minion-group-9k7c:
 container "kubelet": expected 95th% usage < 0.160; got 0.175
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubelet_perf.go:188

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gce-serial/306/

Failed: [k8s.io] Kubelet [Serial] [Slow] [k8s.io] regular resource usage tracking resource tracking for 35 pods per node {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubelet_perf.go:279
Nov  8 23:00:00.751: CPU usage exceeding limits:
 node jenkins-e2e-minion-group-hff5:
 container "kubelet": expected 95th% usage < 0.160; got 0.162
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubelet_perf.go:188

@k8s-github-robot k8s-github-robot mentioned this issue Nov 9, 2016
Closed
@calebamiles calebamiles modified the milestones: v1.5, v1.4 Nov 9, 2016
@yujuhong yujuhong assigned yujuhong and unassigned dchen1107 Nov 11, 2016
@yujuhong yujuhong mentioned this issue Nov 11, 2016
Merged
@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gce-serial/2510/

Failed: [k8s.io] Kubelet [Serial] [Slow] [k8s.io] regular resource usage tracking resource tracking for 35 pods per node {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubelet_perf.go:279
Nov 12 21:52:49.624: CPU usage exceeding limits:
 node jenkins-e2e-minion-group-3i3p:
 container "kubelet": expected 95th% usage < 0.160; got 0.162
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubelet_perf.go:188

@k8s-github-robot
Copy link
Author

[FLAKE-PING] @yujuhong

This flaky-test issue would love to have more attention.

@k8s-github-robot k8s-github-robot mentioned this issue Nov 14, 2016
Closed
k8s-github-robot pushed a commit that referenced this issue Nov 14, 2016
Merge pull request #36623 from yujuhong/adjust_limits
03455d0 
Automatic merge from submit-queue

Use generous limits in the resource usage tracking tests

These tests are mainly used to catch resource leaks in the soak cluster. Using
higher limits to reduce noise.

This should fix #32942 and #32214.
@k8s-github-robot k8s-github-robot mentioned this issue Nov 20, 2016
Closed
@k8s-github-robot k8s-github-robot mentioned this issue Dec 3, 2016
Closed
@k8s-github-robot k8s-github-robot mentioned this issue Jan 10, 2017
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yujuhong yujuhong

Labels
kind/flake Categorizes issue or PR as related to a flaky test. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.
Projects
None yet
Milestone
v1.5
Development

Successfully merging a pull request may close this issue.

Use generous limits in the resource usage tracking tests yujuhong/kubernetes
6 participants
@madhusudancs @calebamiles @dchen1107 @davidopp @yujuhong @k8s-github-robot