Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[k8s.io] InitContainer should not start app containers and fail the pod if init containers fail on a RestartNever pod {E2eNode Suite} #36159

Closed
k8s-github-robot opened this issue Nov 3, 2016 · 45 comments
Closed

[k8s.io] InitContainer should not start app containers and fail the pod if init containers fail on a RestartNever pod {E2eNode Suite} #36159

k8s-github-robot opened this issue Nov 3, 2016 · 45 comments
Assignees
@vishh
Labels
kind/flake Categorizes issue or PR as related to a flaky test. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now.
Milestone
v1.5

Comments

@k8s-github-robot
Copy link

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/10837/

Failed: [k8s.io] InitContainer should not start app containers and fail the pod if init containers fail on a RestartNever pod {E2eNode Suite}

/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:375
Expected
    <*errors.errorString | 0xc820ff4500>: {
        s: "first init container should have exitCode != 0: api.ContainerStatus{Name:\"init1\", State:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(0xc820f01f80)}, LastTerminationState:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(nil)}, Ready:false, RestartCount:0, Image:\"gcr.io/google_containers/busybox:1.24\", ImageID:\"docker-pullable://gcr.io/google_containers/busybox@sha256:4bdd623e848417d96127e16037743f0cd8b528c026e9175e22a84f639eca58ff\", ContainerID:\"docker://b34cf1c2d87a157a5733d485ed4f965da9d6210518fdb8228b8da3421c261030\"}",
    }
to be nil
/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:363
@k8s-github-robot k8s-github-robot added kind/flake Categorizes issue or PR as related to a flaky test. priority/backlog Higher priority than priority/awaiting-more-evidence. labels Nov 3, 2016
@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/10840/

Failed: [k8s.io] InitContainer should not start app containers and fail the pod if init containers fail on a RestartNever pod {E2eNode Suite}

/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:375
Expected
    <*errors.errorString | 0xc820eb12e0>: {
        s: "first init container should have exitCode != 0: api.ContainerStatus{Name:\"init1\", State:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(0xc820eb67e0)}, LastTerminationState:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(nil)}, Ready:false, RestartCount:0, Image:\"gcr.io/google_containers/busybox:1.24\", ImageID:\"docker-pullable://gcr.io/google_containers/busybox@sha256:4bdd623e848417d96127e16037743f0cd8b528c026e9175e22a84f639eca58ff\", ContainerID:\"docker://f16701665944180dba85864c4d914816a5645426ece4cc40969fcba7822ddd1f\"}",
    }
to be nil
/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:363

@Random-Liu
Copy link
Member

Random-Liu commented Nov 3, 2016 •
edited

It seems that something (docker daemon? infra container? the oom score is -999) got oom killed during init container test on ubuntu docker 9.
From test result:

{kubelet tmp-node-e2e-eb833f82-e2e-node-ubuntu-trusty-docker9-v1-image} Failed: Failed to start container with docker id f16701665944 with error: Error response from daemon: Cannot start container f16701665944180dba85864c4d914816a5645426ece4cc40969fcba7822ddd1f: [8] System error: read parent: connection reset by peer

From docker log:

�[34mINFO�[0m[0488] GET /containers/5d0cfa948cb763144b66cb364505d0258b5cb24b3153576663f460e1213b9dde/json 
�[33mWARN�[0m[0489] signal: killed                               
�[31mERRO�[0m[0489] Handler for POST /v1.21/containers/5d0cfa948cb763144b66cb364505d0258b5cb24b3153576663f460e1213b9dde/start returned error: Cannot start container 5d0cfa948cb763144b66cb364505d0258b5cb24b3153576663f460e1213b9dde: [8] System error: read parent: connection reset by peer 
�[31mERRO�[0m[0489] HTTP Error                                    �[31merr�[0m=Cannot start container 5d0cfa948cb763144b66cb364505d0258b5cb24b3153576663f460e1213b9dde: [8] System error: read parent: connection reset by peer �[31mstatusCode�[0m=500

From kernel log:

Nov  3 12:51:17 tmp-node-e2e-ff1b60c0-e2e-node-ubuntu-trusty-docker9-v1-image kernel: [  510.412235] exe invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=-999
Nov  3 12:51:17 tmp-node-e2e-ff1b60c0-e2e-node-ubuntu-trusty-docker9-v1-image kernel: [  510.412241] exe cpuset=5d0cfa948cb763144b66cb364505d0258b5cb24b3153576663f460e1213b9dde mems_allowed=0
Nov  3 12:51:17 tmp-node-e2e-ff1b60c0-e2e-node-ubuntu-trusty-docker9-v1-image kernel: [  510.412248] CPU: 0 PID: 7238 Comm: exe Tainted: G         C     3.19.0-51-generic #58~14.04.1-Ubuntu
Nov  3 12:51:17 tmp-node-e2e-ff1b60c0-e2e-node-ubuntu-trusty-docker9-v1-image kernel: [  510.412250] Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Nov  3 12:51:17 tmp-node-e2e-ff1b60c0-e2e-node-ubuntu-trusty-docker9-v1-image kernel: [  510.412252]  ffff8800b7568400 ffff880129d3fbd8 ffffffff817afe47 0000000000000000
Nov  3 12:51:17 tmp-node-e2e-ff1b60c0-e2e-node-ubuntu-trusty-docker9-v1-image kernel: [  510.412255]  ffff8800b84f3110 ffff880129d3fc78 ffffffff817aad44 ffff8800b84f36e8
Nov  3 12:51:17 tmp-node-e2e-ff1b60c0-e2e-node-ubuntu-trusty-docker9-v1-image kernel: [  510.412257]  ffffffff810a0b81 ffff88012fc13e80 ffffffff811792c7 ffff8800bb38fc00
Nov  3 12:51:17 tmp-node-e2e-ff1b60c0-e2e-node-ubuntu-trusty-docker9-v1-image kernel: [  510.412260] Call Trace:
Nov  3 12:51:17 tmp-node-e2e-ff1b60c0-e2e-node-ubuntu-trusty-docker9-v1-image kernel: [  510.412269]  [<ffffffff817afe47>] dump_stack+0x45/0x57
Nov  3 12:51:17 tmp-node-e2e-ff1b60c0-e2e-node-ubuntu-trusty-docker9-v1-image kernel: [  510.412272]  [<ffffffff817aad44>] dump_header+0x7f/0x1f1
Nov  3 12:51:17 tmp-node-e2e-ff1b60c0-e2e-node-ubuntu-trusty-docker9-v1-image kernel: [  510.412276]  [<ffffffff810a0b81>] ? try_to_wake_up+0x1f1/0x340
Nov  3 12:51:17 tmp-node-e2e-ff1b60c0-e2e-node-ubuntu-trusty-docker9-v1-image kernel: [  510.412286]  [<ffffffff811792c7>] ? find_lock_task_mm+0x47/0xa0
Nov  3 12:51:17 tmp-node-e2e-ff1b60c0-e2e-node-ubuntu-trusty-docker9-v1-image kernel: [  510.412297]  [<ffffffff8117976b>] oom_kill_process+0x22b/0x390
Nov  3 12:51:17 tmp-node-e2e-ff1b60c0-e2e-node-ubuntu-trusty-docker9-v1-image kernel: [  510.412301]  [<ffffffff811e184d>] mem_cgroup_oom_synchronize+0x56d/0x5d0
Nov  3 12:51:17 tmp-node-e2e-ff1b60c0-e2e-node-ubuntu-trusty-docker9-v1-image kernel: [  510.412305]  [<ffffffff811dcdd0>] ? mem_cgroup_css_online+0x260/0x260
Nov  3 12:51:17 tmp-node-e2e-ff1b60c0-e2e-node-ubuntu-trusty-docker9-v1-image kernel: [  510.412308]  [<ffffffff81179fb8>] pagefault_out_of_memory+0x18/0x90
Nov  3 12:51:17 tmp-node-e2e-ff1b60c0-e2e-node-ubuntu-trusty-docker9-v1-image kernel: [  510.412310]  [<ffffffff817a9aaa>] mm_fault_error+0x67/0x140
Nov  3 12:51:17 tmp-node-e2e-ff1b60c0-e2e-node-ubuntu-trusty-docker9-v1-image kernel: [  510.412314]  [<ffffffff81062012>] __do_page_fault+0x452/0x5a0
Nov  3 12:51:17 tmp-node-e2e-ff1b60c0-e2e-node-ubuntu-trusty-docker9-v1-image kernel: [  510.412318]  [<ffffffff810a5138>] ? __enqueue_entity+0x78/0x80
Nov  3 12:51:17 tmp-node-e2e-ff1b60c0-e2e-node-ubuntu-trusty-docker9-v1-image kernel: [  510.412320]  [<ffffffff810a705d>] ? set_next_entity+0x9d/0xb0
Nov  3 12:51:17 tmp-node-e2e-ff1b60c0-e2e-node-ubuntu-trusty-docker9-v1-image kernel: [  510.412323]  [<ffffffff810af4bb>] ? pick_next_task_fair+0x61b/0x880
Nov  3 12:51:17 tmp-node-e2e-ff1b60c0-e2e-node-ubuntu-trusty-docker9-v1-image kernel: [  510.412327]  [<ffffffff8101356c>] ? __switch_to+0xdc/0x570
Nov  3 12:51:17 tmp-node-e2e-ff1b60c0-e2e-node-ubuntu-trusty-docker9-v1-image kernel: [  510.412330]  [<ffffffff81062191>] do_page_fault+0x31/0x70
Nov  3 12:51:17 tmp-node-e2e-ff1b60c0-e2e-node-ubuntu-trusty-docker9-v1-image kernel: [  510.412333]  [<ffffffff817b99e8>] page_fault+0x28/0x30
Nov  3 12:51:17 tmp-node-e2e-ff1b60c0-e2e-node-ubuntu-trusty-docker9-v1-image kernel: [  510.412375] Task in /pod35267c8a-a1c4-11e6-9188-42010a800005/5d0cfa948cb763144b66cb364505d0258b5cb24b3153576663f460e1213b9dde killed as a result of limit of /pod35267c8a-a1c4-11e6-9188-42010a800005

May be related to the pod cgroup pr, since this happens from that PR got merged.

@derekwaynecarr @dchen1107
/cc @kubernetes/sig-node

@vishh
Copy link
Contributor

vishh commented Nov 3, 2016

I can look into it.

@vishh vishh assigned vishh and unassigned maisem Nov 3, 2016
@mwielgus mwielgus mentioned this issue Nov 3, 2016
Merged
@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/10866/

Failed: [k8s.io] InitContainer should not start app containers and fail the pod if init containers fail on a RestartNever pod {E2eNode Suite}

/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:375
Expected
    <*errors.errorString | 0xc8206bdf10>: {
        s: "second init container should have reason PodInitializing: api.ContainerStatus{Name:\"init1\", State:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(0xc8206bfb60), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(nil)}, LastTerminationState:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(nil)}, Ready:false, RestartCount:0, Image:\"gcr.io/google_containers/busybox:1.24\", ImageID:\"docker-pullable://gcr.io/google_containers/busybox@sha256:4bdd623e848417d96127e16037743f0cd8b528c026e9175e22a84f639eca58ff\", ContainerID:\"docker://3eadd5a35dfeb43a8c86eb74dcaffb536ed043cd877eb83a0fcd7d3201a97b04\"}",
    }
to be nil
/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:363

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/10868/

Failed: [k8s.io] InitContainer should not start app containers and fail the pod if init containers fail on a RestartNever pod {E2eNode Suite}

/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:375
Expected
    <*errors.errorString | 0xc820252dd0>: {
        s: "first init container should have exitCode != 0: api.ContainerStatus{Name:\"init1\", State:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(0xc8203d4380)}, LastTerminationState:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(nil)}, Ready:false, RestartCount:0, Image:\"gcr.io/google_containers/busybox:1.24\", ImageID:\"docker-pullable://gcr.io/google_containers/busybox@sha256:4bdd623e848417d96127e16037743f0cd8b528c026e9175e22a84f639eca58ff\", ContainerID:\"docker://f6662bef0f750656f2ffeffa584c1f4091a2fe9db7c2732762b9348e6227e401\"}",
    }
to be nil
/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:363

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/10894/

Failed: [k8s.io] InitContainer should not start app containers and fail the pod if init containers fail on a RestartNever pod {E2eNode Suite}

/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:375
Expected
    <*errors.errorString | 0xc820f08440>: {
        s: "first init container should have exitCode != 0: api.ContainerStatus{Name:\"init1\", State:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(0xc8210ef340)}, LastTerminationState:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(nil)}, Ready:false, RestartCount:0, Image:\"gcr.io/google_containers/busybox:1.24\", ImageID:\"docker-pullable://gcr.io/google_containers/busybox@sha256:4bdd623e848417d96127e16037743f0cd8b528c026e9175e22a84f639eca58ff\", ContainerID:\"docker://376b97b8100b0b6ad8a919bd8c5e141ea8b617cf30eba965828361ccf13c5e4e\"}",
    }
to be nil
/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:363

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/10895/

Failed: [k8s.io] InitContainer should not start app containers and fail the pod if init containers fail on a RestartNever pod {E2eNode Suite}

/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:375
Expected
    <*errors.errorString | 0xc820fb5c90>: {
        s: "first init container should have exitCode != 0: api.ContainerStatus{Name:\"init1\", State:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(0xc820f4dab0)}, LastTerminationState:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(nil)}, Ready:false, RestartCount:0, Image:\"gcr.io/google_containers/busybox:1.24\", ImageID:\"docker-pullable://gcr.io/google_containers/busybox@sha256:4bdd623e848417d96127e16037743f0cd8b528c026e9175e22a84f639eca58ff\", ContainerID:\"docker://57b7268f6e30b319cad94c65a1d63d33f4818c74389bda126ec2b9a56abef169\"}",
    }
to be nil
/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:363

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/10896/

Failed: [k8s.io] InitContainer should not start app containers and fail the pod if init containers fail on a RestartNever pod {E2eNode Suite}

/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:375
Expected
    <*errors.errorString | 0xc820e46b60>: {
        s: "first init container should have exitCode != 0: api.ContainerStatus{Name:\"init1\", State:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(0xc820ebe540)}, LastTerminationState:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(nil)}, Ready:false, RestartCount:0, Image:\"gcr.io/google_containers/busybox:1.24\", ImageID:\"docker-pullable://gcr.io/google_containers/busybox@sha256:4bdd623e848417d96127e16037743f0cd8b528c026e9175e22a84f639eca58ff\", ContainerID:\"docker://cd2b95904224c0395205fe9e6189b178d3b0ed20ddffbf13687767782591d170\"}",
    }
to be nil
/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:363

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/10897/

Failed: [k8s.io] InitContainer should not start app containers and fail the pod if init containers fail on a RestartNever pod {E2eNode Suite}

/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:375
Expected
    <*errors.errorString | 0xc8204613c0>: {
        s: "first init container should have exitCode != 0: api.ContainerStatus{Name:\"init1\", State:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(0xc820ce2e70)}, LastTerminationState:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(nil)}, Ready:false, RestartCount:0, Image:\"gcr.io/google_containers/busybox:1.24\", ImageID:\"docker-pullable://gcr.io/google_containers/busybox@sha256:4bdd623e848417d96127e16037743f0cd8b528c026e9175e22a84f639eca58ff\", ContainerID:\"docker://0d849e857b43f256e7097f2bcc87fd61bce8b7da83ae204e60c3e5814f3fad2a\"}",
    }
to be nil
/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:363

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/10907/

Failed: [k8s.io] InitContainer should not start app containers and fail the pod if init containers fail on a RestartNever pod {E2eNode Suite}

/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:375
Expected
    <*errors.errorString | 0xc820f435c0>: {
        s: "first init container should have exitCode != 0: api.ContainerStatus{Name:\"init1\", State:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(0xc820c69dc0)}, LastTerminationState:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(nil)}, Ready:false, RestartCount:0, Image:\"gcr.io/google_containers/busybox:1.24\", ImageID:\"docker-pullable://gcr.io/google_containers/busybox@sha256:4bdd623e848417d96127e16037743f0cd8b528c026e9175e22a84f639eca58ff\", ContainerID:\"docker://6ae867dfbc668a6063d46dacccd55e19ce94a4817cee750e8679b75333345a31\"}",
    }
to be nil
/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:363

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/10914/

Failed: [k8s.io] InitContainer should not start app containers and fail the pod if init containers fail on a RestartNever pod {E2eNode Suite}

/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:375
Expected
    <*errors.errorString | 0xc8202e5270>: {
        s: "first init container should have exitCode != 0: api.ContainerStatus{Name:\"init1\", State:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(0xc820aab880)}, LastTerminationState:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(nil)}, Ready:false, RestartCount:0, Image:\"gcr.io/google_containers/busybox:1.24\", ImageID:\"docker-pullable://gcr.io/google_containers/busybox@sha256:4bdd623e848417d96127e16037743f0cd8b528c026e9175e22a84f639eca58ff\", ContainerID:\"docker://3af9445507e19f34d046f522bb5ad34413a37ba46dd9307b2cf4acee9a1a8a48\"}",
    }
to be nil
/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:363

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/10915/

Failed: [k8s.io] InitContainer should not start app containers and fail the pod if init containers fail on a RestartNever pod {E2eNode Suite}

/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:375
Expected
    <*errors.errorString | 0xc820dfa3d0>: {
        s: "first init container should have exitCode != 0: api.ContainerStatus{Name:\"init1\", State:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(0xc820bee5b0)}, LastTerminationState:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(nil)}, Ready:false, RestartCount:0, Image:\"gcr.io/google_containers/busybox:1.24\", ImageID:\"docker-pullable://gcr.io/google_containers/busybox@sha256:4bdd623e848417d96127e16037743f0cd8b528c026e9175e22a84f639eca58ff\", ContainerID:\"docker://dd65cf8e7576985ffdb146c06870d7094c741dd27b794c6a5c37ab7ab9737817\"}",
    }
to be nil
/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:363

@k8s-github-robot k8s-github-robot added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. and removed priority/backlog Higher priority than priority/awaiting-more-evidence. labels Nov 6, 2016
@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/10925/

Failed: [k8s.io] InitContainer should not start app containers and fail the pod if init containers fail on a RestartNever pod {E2eNode Suite}

/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:375
Expected
    <*errors.errorString | 0xc820f4a780>: {
        s: "first init container should have exitCode != 0: api.ContainerStatus{Name:\"init1\", State:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(0xc820dd1dc0)}, LastTerminationState:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(nil)}, Ready:false, RestartCount:0, Image:\"gcr.io/google_containers/busybox:1.24\", ImageID:\"docker-pullable://gcr.io/google_containers/busybox@sha256:4bdd623e848417d96127e16037743f0cd8b528c026e9175e22a84f639eca58ff\", ContainerID:\"docker://49d99fa755441a2dbc12ca95158d947e9f9c266fbce2233ffd3538720d542378\"}",
    }
to be nil
/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:363

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/10934/

Failed: [k8s.io] InitContainer should not start app containers and fail the pod if init containers fail on a RestartNever pod {E2eNode Suite}

/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:375
Expected
    <*errors.errorString | 0xc821085140>: {
        s: "first init container should have exitCode != 0: api.ContainerStatus{Name:\"init1\", State:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(0xc820db6230)}, LastTerminationState:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(nil)}, Ready:false, RestartCount:0, Image:\"gcr.io/google_containers/busybox:1.24\", ImageID:\"docker-pullable://gcr.io/google_containers/busybox@sha256:4bdd623e848417d96127e16037743f0cd8b528c026e9175e22a84f639eca58ff\", ContainerID:\"docker://3304f0961979f24ca694e9a94df4922dd79504d6e4c7bd1b4a6a0f6ff3cde74a\"}",
    }
to be nil
/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:363

@MrHohn MrHohn mentioned this issue Nov 6, 2016
Merged
@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/10947/

Failed: [k8s.io] InitContainer should not start app containers and fail the pod if init containers fail on a RestartNever pod {E2eNode Suite}

/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:375
Expected
    <*errors.errorString | 0xc820d301d0>: {
        s: "first init container should have exitCode != 0: api.ContainerStatus{Name:\"init1\", State:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(0xc820d487e0)}, LastTerminationState:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(nil)}, Ready:false, RestartCount:0, Image:\"gcr.io/google_containers/busybox:1.24\", ImageID:\"docker-pullable://gcr.io/google_containers/busybox@sha256:4bdd623e848417d96127e16037743f0cd8b528c026e9175e22a84f639eca58ff\", ContainerID:\"docker://0207aa043e148b68a39f3be380462735cee3b59df24d05215ec7b85c485bca5e\"}",
    }
to be nil
/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:363

@yujuhong yujuhong mentioned this issue Nov 7, 2016
Merged
@yujuhong
Copy link
Contributor

yujuhong commented Nov 7, 2016

It seems that something (docker daemon? infra container? the oom score is -999) got oom killed during init container test on ubuntu docker 9.
From test result:

The pod level cgroup exceeded the memory limit and got oom killed. I think the memory limit of the pod in the test needs to be adjusted.

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/10970/

Failed: [k8s.io] InitContainer should not start app containers and fail the pod if init containers fail on a RestartNever pod {E2eNode Suite}

/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:375
Expected
    <*errors.errorString | 0xc820bc0d70>: {
        s: "first init container should have exitCode != 0: api.ContainerStatus{Name:\"init1\", State:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(0xc820c21030)}, LastTerminationState:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(nil)}, Ready:false, RestartCount:0, Image:\"gcr.io/google_containers/busybox:1.24\", ImageID:\"docker-pullable://gcr.io/google_containers/busybox@sha256:4bdd623e848417d96127e16037743f0cd8b528c026e9175e22a84f639eca58ff\", ContainerID:\"docker://bb68ae14b6e9e438d54c9465f443b9379a3fd29ed36b5c6b636f7c3c543fe855\"}",
    }
to be nil
/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:363

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/10972/

Failed: [k8s.io] InitContainer should not start app containers and fail the pod if init containers fail on a RestartNever pod {E2eNode Suite}

/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:375
Expected
    <*errors.errorString | 0xc820ef3c40>: {
        s: "first init container should have exitCode != 0: api.ContainerStatus{Name:\"init1\", State:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(0xc820efcf50)}, LastTerminationState:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(nil)}, Ready:false, RestartCount:0, Image:\"gcr.io/google_containers/busybox:1.24\", ImageID:\"docker-pullable://gcr.io/google_containers/busybox@sha256:4bdd623e848417d96127e16037743f0cd8b528c026e9175e22a84f639eca58ff\", ContainerID:\"docker://a6f503ca4d384a3164e606309114834a5cd91a4f53ad917e7695ae5624760349\"}",
    }
to be nil
/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:363

@0xmichalis 0xmichalis mentioned this issue Nov 8, 2016
Merged
@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/10985/

Failed: [k8s.io] InitContainer should not start app containers and fail the pod if init containers fail on a RestartNever pod {E2eNode Suite}

/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:375
Expected
    <*errors.errorString | 0xc82015fa20>: {
        s: "first init container should have exitCode != 0: api.ContainerStatus{Name:\"init1\", State:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(0xc820dc5500)}, LastTerminationState:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(nil)}, Ready:false, RestartCount:0, Image:\"gcr.io/google_containers/busybox:1.24\", ImageID:\"docker-pullable://gcr.io/google_containers/busybox@sha256:4bdd623e848417d96127e16037743f0cd8b528c026e9175e22a84f639eca58ff\", ContainerID:\"docker://b7bac5ba0160b56a2745af9a22dee050ecb0aadf14c6e56ad3e46661273a2277\"}",
    }
to be nil
/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:363

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/11043/

Failed: [k8s.io] InitContainer should not start app containers and fail the pod if init containers fail on a RestartNever pod {E2eNode Suite}

/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:375
Expected
    <*errors.errorString | 0xc820fd4ec0>: {
        s: "first init container should have exitCode != 0: api.ContainerStatus{Name:\"init1\", State:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(0xc821043960)}, LastTerminationState:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(nil)}, Ready:false, RestartCount:0, Image:\"gcr.io/google_containers/busybox:1.24\", ImageID:\"docker-pullable://gcr.io/google_containers/busybox@sha256:4bdd623e848417d96127e16037743f0cd8b528c026e9175e22a84f639eca58ff\", ContainerID:\"docker://377e4e76ff6b17351ad08be6f8bcbf7f08f16e6567ac3c726ab50017e8177c22\"}",
    }
to be nil
/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:363

@euank
Copy link
Contributor

euank commented Nov 10, 2016

The last few happened on ubuntu. It's potentially related to the docker configuration or kernel. Haven't dug deeper than that yet..

@yujuhong
Copy link
Contributor

last few happened on ubuntu. It's potentially related to the docker configuration or kernel. Haven't dug deeper than that yet..

@euank, I think my #36159 (comment) is still valid...

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/11071/

Failed: [k8s.io] InitContainer should not start app containers and fail the pod if init containers fail on a RestartNever pod {E2eNode Suite}

/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:375
Expected
    <*errors.errorString | 0xc82032bd20>: {
        s: "first init container should have exitCode != 0: api.ContainerStatus{Name:\"init1\", State:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(0xc820b87730)}, LastTerminationState:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(nil)}, Ready:false, RestartCount:0, Image:\"gcr.io/google_containers/busybox:1.24\", ImageID:\"docker-pullable://gcr.io/google_containers/busybox@sha256:4bdd623e848417d96127e16037743f0cd8b528c026e9175e22a84f639eca58ff\", ContainerID:\"docker://e61d8de3a3d05662285bd821fb5f2065e4d7aa9bcd55660a4dbeb728d0b23522\"}",
    }
to be nil
/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:363

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/11089/

Failed: [k8s.io] InitContainer should not start app containers and fail the pod if init containers fail on a RestartNever pod {E2eNode Suite}

/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:375
Expected
    <*errors.errorString | 0xc820f8be60>: {
        s: "first init container should have exitCode != 0: api.ContainerStatus{Name:\"init1\", State:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(0xc820ff2bd0)}, LastTerminationState:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(nil)}, Ready:false, RestartCount:0, Image:\"gcr.io/google_containers/busybox:1.24\", ImageID:\"docker-pullable://gcr.io/google_containers/busybox@sha256:4bdd623e848417d96127e16037743f0cd8b528c026e9175e22a84f639eca58ff\", ContainerID:\"docker://b6dff84ed745ab52687cebd067fb7b6c94c09424449b3c46d102f8053d835c2a\"}",
    }
to be nil
/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:363

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/11101/

Failed: [k8s.io] InitContainer should not start app containers and fail the pod if init containers fail on a RestartNever pod {E2eNode Suite}

/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:375
Expected
    <*errors.errorString | 0xc820c694e0>: {
        s: "first init container should have exitCode != 0: api.ContainerStatus{Name:\"init1\", State:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(0xc821117b90)}, LastTerminationState:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(nil)}, Ready:false, RestartCount:0, Image:\"gcr.io/google_containers/busybox:1.24\", ImageID:\"docker-pullable://gcr.io/google_containers/busybox@sha256:4bdd623e848417d96127e16037743f0cd8b528c026e9175e22a84f639eca58ff\", ContainerID:\"docker://84fd6d8fe6f568659ab26603f921551834aa2f35c00cfd39c7844bd92301b134\"}",
    }
to be nil
/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:363

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/11102/

Failed: [k8s.io] InitContainer should not start app containers and fail the pod if init containers fail on a RestartNever pod {E2eNode Suite}

/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:375
Expected
    <*errors.errorString | 0xc82018eb20>: {
        s: "first init container should have exitCode != 0: api.ContainerStatus{Name:\"init1\", State:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(0xc820049d50)}, LastTerminationState:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(nil)}, Ready:false, RestartCount:0, Image:\"gcr.io/google_containers/busybox:1.24\", ImageID:\"docker-pullable://gcr.io/google_containers/busybox@sha256:4bdd623e848417d96127e16037743f0cd8b528c026e9175e22a84f639eca58ff\", ContainerID:\"docker://3ab3598ed8150e48284143ef3e59360f4a9c37152cf1815870a7a0909440b731\"}",
    }
to be nil
/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:363

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/11119/

Failed: [k8s.io] InitContainer should not start app containers and fail the pod if init containers fail on a RestartNever pod {E2eNode Suite}

/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:375
Expected
    <*errors.errorString | 0xc820253090>: {
        s: "first init container should have exitCode != 0: api.ContainerStatus{Name:\"init1\", State:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(0xc82024a1c0)}, LastTerminationState:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(nil)}, Ready:false, RestartCount:0, Image:\"gcr.io/google_containers/busybox:1.24\", ImageID:\"docker-pullable://gcr.io/google_containers/busybox@sha256:4bdd623e848417d96127e16037743f0cd8b528c026e9175e22a84f639eca58ff\", ContainerID:\"docker://b328a4b534b957128e118f3e164d40147d5f559576e669267ea6f8671a7cfb2c\"}",
    }
to be nil
/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:363

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/11122/

Failed: [k8s.io] InitContainer should not start app containers and fail the pod if init containers fail on a RestartNever pod {E2eNode Suite}

/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:375
Expected
    <*errors.errorString | 0xc821149bd0>: {
        s: "first init container should have exitCode != 0: api.ContainerStatus{Name:\"init1\", State:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(0xc82115a620)}, LastTerminationState:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(nil)}, Ready:false, RestartCount:0, Image:\"gcr.io/google_containers/busybox:1.24\", ImageID:\"docker-pullable://gcr.io/google_containers/busybox@sha256:4bdd623e848417d96127e16037743f0cd8b528c026e9175e22a84f639eca58ff\", ContainerID:\"docker://36014caf1f896c38306dfe8ced222c2124aad68df72ba8604f1c356ba220ab23\"}",
    }
to be nil
/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:363

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/11124/

Failed: [k8s.io] InitContainer should not start app containers and fail the pod if init containers fail on a RestartNever pod {E2eNode Suite}

/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:375
Expected
    <*errors.errorString | 0xc8201dce40>: {
        s: "first init container should have exitCode != 0: api.ContainerStatus{Name:\"init1\", State:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(0xc820b32150)}, LastTerminationState:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(nil)}, Ready:false, RestartCount:0, Image:\"gcr.io/google_containers/busybox:1.24\", ImageID:\"docker-pullable://gcr.io/google_containers/busybox@sha256:4bdd623e848417d96127e16037743f0cd8b528c026e9175e22a84f639eca58ff\", ContainerID:\"docker://eebcd56ee7341a10334cfcd840a6194e9a11b018485ba71a949c06a8f3131130\"}",
    }
to be nil
/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:363

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/11136/

Failed: [k8s.io] InitContainer should not start app containers and fail the pod if init containers fail on a RestartNever pod {E2eNode Suite}

/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:375
Expected
    <*errors.errorString | 0xc820eece70>: {
        s: "first init container should have exitCode != 0: api.ContainerStatus{Name:\"init1\", State:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(0xc820c8a0e0)}, LastTerminationState:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(nil)}, Ready:false, RestartCount:0, Image:\"gcr.io/google_containers/busybox:1.24\", ImageID:\"docker-pullable://gcr.io/google_containers/busybox@sha256:4bdd623e848417d96127e16037743f0cd8b528c026e9175e22a84f639eca58ff\", ContainerID:\"docker://fec4cf73a42e6e2f2cabdb949032bf1ce3d97172bd2769546d984af14e2e1654\"}",
    }
to be nil
/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:363

@saad-ali
Copy link
Member

Ping, can we get an update on this.

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/11150/

Failed: [k8s.io] InitContainer should not start app containers and fail the pod if init containers fail on a RestartNever pod {E2eNode Suite}

/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:375
Expected
    <*errors.errorString | 0xc820f26920>: {
        s: "first init container should have exitCode != 0: api.ContainerStatus{Name:\"init1\", State:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(0xc820f1cc40)}, LastTerminationState:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(nil)}, Ready:false, RestartCount:0, Image:\"gcr.io/google_containers/busybox:1.24\", ImageID:\"docker-pullable://gcr.io/google_containers/busybox@sha256:4bdd623e848417d96127e16037743f0cd8b528c026e9175e22a84f639eca58ff\", ContainerID:\"docker://6b18a147a01d5247ada25b392f8a9f38497344f832b0fe12937ec38cf8b22565\"}",
    }
to be nil
/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:363

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/11152/

Failed: [k8s.io] InitContainer should not start app containers and fail the pod if init containers fail on a RestartNever pod {E2eNode Suite}

/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:375
Expected
    <*errors.errorString | 0xc820e4c6f0>: {
        s: "second init container should have reason PodInitializing: api.ContainerStatus{Name:\"init1\", State:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(0xc820e4fec0), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(nil)}, LastTerminationState:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(nil)}, Ready:false, RestartCount:0, Image:\"gcr.io/google_containers/busybox:1.24\", ImageID:\"docker-pullable://gcr.io/google_containers/busybox@sha256:4bdd623e848417d96127e16037743f0cd8b528c026e9175e22a84f639eca58ff\", ContainerID:\"docker://82f71cfcc9c0ee7d592f260e4a0f7b7e1131253f4ba86f4a5ffd0efa56d2876a\"}",
    }
to be nil
/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:363

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/11155/

Failed: [k8s.io] InitContainer should not start app containers and fail the pod if init containers fail on a RestartNever pod {E2eNode Suite}

/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:375
Expected
    <*errors.errorString | 0xc820b58c00>: {
        s: "first init container should have exitCode != 0: api.ContainerStatus{Name:\"init1\", State:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(0xc820ebbf10)}, LastTerminationState:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(nil)}, Ready:false, RestartCount:0, Image:\"gcr.io/google_containers/busybox:1.24\", ImageID:\"docker-pullable://gcr.io/google_containers/busybox@sha256:4bdd623e848417d96127e16037743f0cd8b528c026e9175e22a84f639eca58ff\", ContainerID:\"docker://91cccc26d5c0771c011a4b6814bfe411807172cefb1601343f0323ba1452040d\"}",
    }
to be nil
/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:363

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/11160/

Failed: [k8s.io] InitContainer should not start app containers and fail the pod if init containers fail on a RestartNever pod {E2eNode Suite}

/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:375
Expected
    <*errors.errorString | 0xc820f10b40>: {
        s: "first init container should have exitCode != 0: api.ContainerStatus{Name:\"init1\", State:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(0xc821017ea0)}, LastTerminationState:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(nil)}, Ready:false, RestartCount:0, Image:\"gcr.io/google_containers/busybox:1.24\", ImageID:\"docker-pullable://gcr.io/google_containers/busybox@sha256:4bdd623e848417d96127e16037743f0cd8b528c026e9175e22a84f639eca58ff\", ContainerID:\"docker://1df71201a81f593d1faff3354fd0cf6207341aa0f5b52a6da737af9a7b3b8d74\"}",
    }
to be nil
/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:363

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/11161/

Failed: [k8s.io] InitContainer should not start app containers and fail the pod if init containers fail on a RestartNever pod {E2eNode Suite}

/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:375
Expected
    <*errors.errorString | 0xc8201a52e0>: {
        s: "first init container should have exitCode != 0: api.ContainerStatus{Name:\"init1\", State:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(0xc82029b340)}, LastTerminationState:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(nil)}, Ready:false, RestartCount:0, Image:\"gcr.io/google_containers/busybox:1.24\", ImageID:\"docker-pullable://gcr.io/google_containers/busybox@sha256:4bdd623e848417d96127e16037743f0cd8b528c026e9175e22a84f639eca58ff\", ContainerID:\"docker://c348e0d53a121521fd7c7005a5ebd139e572308990b417455365d3711317666d\"}",
    }
to be nil
/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:363

@euank euank mentioned this issue Nov 14, 2016
Merged
@euank
Copy link
Contributor

euank commented Nov 14, 2016 •
edited

PR that might fix this over #36771. I skimmed the kern.log for a couple of them and, as @yujuhong pointed out, there are cgroup OOMs.

Hopefully that PR does fix this, but I haven't done any testing or deeper examination.

euank added a commit to euank/kubernetes that referenced this issue Nov 14, 2016
@euank
test/e2e: update init container memory limit
08382c1 
On ubuntu, the `RestartNever` test OOMs its cgroup limit fairly
frequently.

This bumps the number up to something suitably large since the test
isn't testing anything related to this anyways.

Fixes kubernetes#36159
@saad-ali
Copy link
Member

Thanks @euank. @yujuhong could you review Euan's PR please? This issue is becoming more frequent.

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/11163/

Failed: [k8s.io] InitContainer should not start app containers and fail the pod if init containers fail on a RestartNever pod {E2eNode Suite}

/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:375
Expected
    <*errors.errorString | 0xc820fd0660>: {
        s: "first init container should have exitCode != 0: api.ContainerStatus{Name:\"init1\", State:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(0xc820945260)}, LastTerminationState:api.ContainerState{Waiting:(*api.ContainerStateWaiting)(nil), Running:(*api.ContainerStateRunning)(nil), Terminated:(*api.ContainerStateTerminated)(nil)}, Ready:false, RestartCount:0, Image:\"gcr.io/google_containers/busybox:1.24\", ImageID:\"docker-pullable://gcr.io/google_containers/busybox@sha256:4bdd623e848417d96127e16037743f0cd8b528c026e9175e22a84f639eca58ff\", ContainerID:\"docker://a443c42214a62a53df34c896c8442101d615e033cf12056e14d90604d1915732\"}",
    }
to be nil
/var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:363

@saad-ali
Copy link
Member

kubelet-gce-e2e-ci is consistently failing with this test. Raising priority to P0

@saad-ali saad-ali added priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. and removed priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. labels Nov 14, 2016
jessfraz pushed a commit to jessfraz/kubernetes that referenced this issue Nov 15, 2016
Merge pull request kubernetes#36771 from euank/bump-some-memory
122a228 
Automatic merge from submit-queue

tests: update memory resource limits

```release-note
NONE
```

On ubuntu, the `RestartNever` test OOMs its cgroup limit fairly
frequently.

This bumps the number up to something suitably large since the test
isn't testing anything related to this anyways.

Fixes kubernetes#36159

Fix based on kubernetes#36159 (comment)

cc @yujuhong @saad-ali
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vishh vishh

Labels
kind/flake Categorizes issue or PR as related to a flaky test. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now.
Projects
None yet
Milestone
v1.5
Development

No branches or pull requests
8 participants
@calebamiles @euank @maisem @Random-Liu @vishh @saad-ali @yujuhong @k8s-github-robot