Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[k8s.io] Nodes [Disruptive] [k8s.io] Network when a node becomes unreachable [replication controller] recreates pods scheduled on the unreachable node AND allows scheduling of pods on a node after it rejoins the cluster {Kubernetes e2e suite} #35880

Closed
k8s-github-robot opened this issue Oct 30, 2016 · 12 comments
Closed

[k8s.io] Nodes [Disruptive] [k8s.io] Network when a node becomes unreachable [replication controller] recreates pods scheduled on the unreachable node AND allows scheduling of pods on a node after it rejoins the cluster {Kubernetes e2e suite} #35880

k8s-github-robot opened this issue Oct 30, 2016 · 12 comments
Assignees
@timothysc
Labels
kind/flake Categorizes issue or PR as related to a flaky test. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling.
Milestone
v1.6

Comments

@k8s-github-robot
Copy link

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-test/13853/

Failed: [k8s.io] Nodes [Disruptive] [k8s.io] Network when a node becomes unreachable [replication controller] recreates pods scheduled on the unreachable node AND allows scheduling of pods on a node after it rejoins the cluster {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resize_nodes.go:543
Expected error:
    <*errors.errorString | 0xc8210de5a0>: {
        s: "failed to wait for pods responding: timed out waiting for the condition",
    }
    failed to wait for pods responding: timed out waiting for the condition
not to have occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resize_nodes.go:338

Previous issues for this test: #27324 #35852
@k8s-github-robot k8s-github-robot added kind/flake Categorizes issue or PR as related to a flaky test. priority/backlog Higher priority than priority/awaiting-more-evidence. labels Oct 30, 2016
@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-pre-release/1092/

Failed: [k8s.io] Nodes [Disruptive] [k8s.io] Network when a node becomes unreachable [replication controller] recreates pods scheduled on the unreachable node AND allows scheduling of pods on a node after it rejoins the cluster {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resize_nodes.go:544
Oct 30 10:06:36.768: Node gke-jenkins-e2e-default-pool-1bdfae9b-pd44 did not become ready within 2m0s
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resize_nodes.go:521

This was referenced Oct 31, 2016
Closed
Closed
Closed
@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-pre-release/1104/

Failed: [k8s.io] Nodes [Disruptive] [k8s.io] Network when a node becomes unreachable [replication controller] recreates pods scheduled on the unreachable node AND allows scheduling of pods on a node after it rejoins the cluster {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resize_nodes.go:544
Nov  2 21:31:51.184: Node gke-jenkins-e2e-default-pool-a43aafa3-g8l1 did not become ready within 2m0s
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resize_nodes.go:521

This was referenced Nov 4, 2016
Closed
Closed
Closed
@k8s-github-robot
Copy link
Author

k8s-github-robot commented Nov 12, 2016 •
edited by rmmh
Loading

Builds:
ci-kubernetes-e2e-gci-gke-prod 19 29
ci-kubernetes-e2e-gci-gke-serial-release-1.4 18 42
ci-kubernetes-e2e-gci-gke-staging 22 34
ci-kubernetes-e2e-gci-gke-subnet 3 17
ci-kubernetes-e2e-gci-gke-test 19
ci-kubernetes-e2e-gke-staging 28 35
ci-kubernetes-e2e-gke-test 20
kubernetes-e2e-gke-staging 6664 6672 6681
kubernetes-e2e-gke-test 13901

Failed: [k8s.io] Nodes [Disruptive] [k8s.io] Network when a node becomes unreachable [replication controller] recreates pods scheduled on the unreachable node AND allows scheduling of pods on a node after it rejoins the cluster {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resize_nodes.go:543
Expected error:
    <*errors.errorString | 0xc821aedd20>: {
        s: "failed to wait for pods responding: timed out waiting for the condition",
    }
    failed to wait for pods responding: timed out waiting for the condition
not to have occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resize_nodes.go:338

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-staging/7/

Failed: [k8s.io] Nodes [Disruptive] [k8s.io] Network when a node becomes unreachable [replication controller] recreates pods scheduled on the unreachable node AND allows scheduling of pods on a node after it rejoins the cluster {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resize_nodes.go:543
Nov 20 10:46:14.538: Unexpected error: <nil>
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:4835

@k8s-github-robot k8s-github-robot added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. and removed priority/backlog Higher priority than priority/awaiting-more-evidence. labels Nov 26, 2016
@k8s-github-robot
Copy link
Author

[FLAKE-PING] @alex-mohr

This flaky-test issue would love to have more attention.

@calebamiles
Copy link
Contributor

@alex-mohr should this issue be moved into the 1.5 milestone and is it a release blocker? Please update this issue by 1PM PST 30 November 2016. Thanks!

cc: @saad-ali, @dims

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-staging/39/

Failed: [k8s.io] Nodes [Disruptive] [k8s.io] Network when a node becomes unreachable [replication controller] recreates pods scheduled on the unreachable node AND allows scheduling of pods on a node after it rejoins the cluster {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resize_nodes.go:543
Expected error:
    <*errors.errorString | 0xc8212fa4d0>: {
        s: "failed to wait for pods responding: timed out waiting for the condition",
    }
    failed to wait for pods responding: timed out waiting for the condition
not to have occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resize_nodes.go:338

@saad-ali
Copy link
Member

@alex-mohr should this issue be moved into the 1.5 milestone and is it a release blocker? Please update this issue by 1PM PST 30 November 2016. Thanks!

cc: @saad-ali, @dims

These flakes appear to be only on GKE not GCE. CC @kubernetes/goog-gke

@k8s-github-robot
Copy link
Author

k8s-github-robot commented Nov 29, 2016 •
edited by rmmh
Loading

Builds:
ci-kubernetes-e2e-gci-gke-prod 88
ci-kubernetes-e2e-gci-gke-serial-release-1.4 89 99 134 178 181 192
ci-kubernetes-e2e-gci-gke-staging 38
ci-kubernetes-e2e-gci-gke-subnet 61 84 94
ci-kubernetes-e2e-gci-gke-test 62 64
ci-kubernetes-e2e-gke-staging 47 63
ci-kubernetes-e2e-gke-test 102 109 114 118

Failed: [k8s.io] Nodes [Disruptive] [k8s.io] Network when a node becomes unreachable [replication controller] recreates pods scheduled on the unreachable node AND allows scheduling of pods on a node after it rejoins the cluster {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resize_nodes.go:543
Expected error:
    <*errors.errorString | 0xc821f3a990>: {
        s: "failed to wait for pods responding: timed out waiting for the condition",
    }
    failed to wait for pods responding: timed out waiting for the condition
not to have occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resize_nodes.go:338

This was referenced Dec 1, 2016
Closed
Closed
Closed
Closed
Closed
@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-subnet/48/

Failed: [k8s.io] Nodes [Disruptive] [k8s.io] Network when a node becomes unreachable [replication controller] recreates pods scheduled on the unreachable node AND allows scheduling of pods on a node after it rejoins the cluster {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:134
Dec  1 00:04:13.894: Couldn't delete ns: "e2e-tests-resize-nodes-lf621": an error on the server ("Internal Server Error: \"/api/v1/namespaces/e2e-tests-resize-nodes-lf621/persistentvolumeclaims\"") has prevented the request from succeeding (get persistentvolumeclaims) (&errors.StatusError{ErrStatus:unversioned.Status{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, ListMeta:unversioned.ListMeta{SelfLink:"", ResourceVersion:""}, Status:"Failure", Message:"an error on the server (\"Internal Server Error: \\\"/api/v1/namespaces/e2e-tests-resize-nodes-lf621/persistentvolumeclaims\\\"\") has prevented the request from succeeding (get persistentvolumeclaims)", Reason:"InternalError", Details:(*unversioned.StatusDetails)(0xc82090ce10), Code:500}})
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:338

This was referenced Dec 2, 2016
Closed
Closed
Closed
@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-subnet/56/

Failed: [k8s.io] Nodes [Disruptive] [k8s.io] Network when a node becomes unreachable [replication controller] recreates pods scheduled on the unreachable node AND allows scheduling of pods on a node after it rejoins the cluster {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resize_nodes.go:543
Dec  3 03:00:14.725: Node gke-bootstrap-e2e-default-pool-94c40648-hknz did not become ready within 2m0s
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resize_nodes.go:520

@timothysc timothysc added the sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. label Dec 9, 2016
@k8s-github-robot k8s-github-robot mentioned this issue Dec 10, 2016
Closed
This was referenced Dec 10, 2016
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
This was referenced Dec 18, 2016
Closed
Closed
Closed
@k8s-github-robot k8s-github-robot mentioned this issue Jan 5, 2017
Closed
This was referenced Jan 30, 2017
Closed
Closed
Closed
@calebamiles calebamiles modified the milestone: v1.6 Mar 3, 2017
@timothysc timothysc assigned timothysc and unassigned alex-mohr Mar 6, 2017
@timothysc
Copy link
Member

Closing, not seen in months / 1.6 release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@timothysc timothysc

Labels
kind/flake Categorizes issue or PR as related to a flaky test. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling.
Projects
None yet
Milestone
v1.6
Development

No branches or pull requests
5 participants
@timothysc @calebamiles @alex-mohr @saad-ali @k8s-github-robot