Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[k8s.io] Port forwarding [k8s.io] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite} #27680

Closed
k8s-github-robot opened this issue Jun 19, 2016 · 88 comments
Closed

[k8s.io] Port forwarding [k8s.io] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite} #27680

k8s-github-robot opened this issue Jun 19, 2016 · 88 comments
Assignees
@sttts @bprashanth
Labels
kind/flake Categorizes issue or PR as related to a flaky test. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now.
Milestone
v1.6

Comments

@k8s-github-robot
Copy link

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-staging-parallel/4404/

Failed: [k8s.io] Port forwarding [k8s.io] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/portforward.go:263
Jun 18 20:37:24.831: Expected "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" from server, got "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
@k8s-github-robot k8s-github-robot added priority/backlog Higher priority than priority/awaiting-more-evidence. kind/flake Categorizes issue or PR as related to a flaky test. labels Jun 19, 2016
@sttts sttts mentioned this issue Jun 21, 2016
Merged
@sttts
Copy link
Contributor

sttts commented Jun 21, 2016

Two hunks (2x 10 "x"s) are not sent back. #27766 extends the logging. At least it will tell us whether the sender gets an error or not.

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/9695/

Failed: [k8s.io] Port forwarding [k8s.io] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/portforward.go:281
Jun 22 05:44:32.507: Expected "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" from server, got "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

@k8s-github-robot k8s-github-robot mentioned this issue Jun 23, 2016
Closed
k8s-github-robot pushed a commit that referenced this issue Jun 30, 2016
Merge pull request #27766 from sttts/sttts-portforward-e2e-more-logs
ede2fa2 
Automatic merge from submit-queue

portforward e2e: extend log for flake hunting

For some more insight into flake #27680
@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/10921/

Failed: [k8s.io] Port forwarding [k8s.io] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/portforward.go:287
Jul 10 06:57:38.736: Expected "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" from server, got "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

@k8s-github-robot k8s-github-robot added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. and removed priority/backlog Higher priority than priority/awaiting-more-evidence. labels Jul 10, 2016
@k8s-github-robot k8s-github-robot mentioned this issue Jul 11, 2016
Closed
@sttts
Copy link
Contributor

sttts commented Jul 12, 2016

No more insights here. Pod log doesn't show anything special.

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-staging-parallel/5642/

Failed: [k8s.io] Port forwarding [k8s.io] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/portforward.go:281
Jul 14 12:37:41.492: Expected "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" from server, got "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

@k8s-github-robot k8s-github-robot mentioned this issue Jul 19, 2016
Closed
@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/25458/kubernetes-pull-build-test-e2e-gce/51254/

Failed: [k8s.io] Port forwarding [k8s.io] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/portforward.go:287
Aug  1 13:21:15.438: Pod did not stop running: timed out waiting for the condition
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/portforward.go:276

Happened on a presubmit run in #25458.

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gce/21576/

Failed: [k8s.io] Port forwarding [k8s.io] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/portforward.go:287
Aug 13 14:30:36.495: Pod did not stop running: timed out waiting for the condition
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/portforward.go:276

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/30348/kubernetes-pull-build-test-e2e-gce/53441/

Failed: [k8s.io] Port forwarding [k8s.io] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/portforward.go:287
Aug 14 20:01:00.516: Expected "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" from server, got ""
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/portforward.go:271

Happened on a presubmit run in #30348.

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/30739/kubernetes-pull-build-test-e2e-gce/53986/

Failed: [k8s.io] Port forwarding [k8s.io] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/portforward.go:287
Aug 16 22:13:24.052: Expected "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" from server, got "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/portforward.go:271

Happened on a presubmit run in #30739.

@k8s-github-robot k8s-github-robot mentioned this issue Aug 23, 2016
Closed
@k8s-github-robot k8s-github-robot mentioned this issue Aug 25, 2016
Closed
This was referenced Aug 26, 2016
Closed
Closed
@sttts
Copy link
Contributor

sttts commented Aug 26, 2016

I analysed this flake instance: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gce/21576/

It looks like the pfpod container terminates, the kubelet realizes that and wants to remove the network interface from the net ns using CNI via kubenet. This fails two times (somehow TeardownNetwork is called multiple times, succeeds twice, but fails after that; compare #20379 about multiple TeardownNetwork calls). These failures interrupt the kubelet's tries to kill the pod. It succeeds >>30 seconds later during reconciliation. But at that point the e2e test timed out already waiting for the pod to terminate.

@k8s-github-robot
Copy link
Author

[FLAKE-PING] @sttts

This flaky-test issue would love to have more attention.

@k8s-github-robot k8s-github-robot mentioned this issue Sep 1, 2016
Closed
@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-staging-parallel/8098/

Failed: [k8s.io] Port forwarding [k8s.io] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/portforward.go:281
Sep  3 20:05:23.509: Expected "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" from server, got "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

@k8s-github-robot
Copy link
Author

[FLAKE-PING] @sttts

This flaky-test issue would love to have more attention.

@dims
Copy link
Member

dims commented Dec 9, 2016

@sttts Is it appropriate to move this to the next milestone? (and remove the non-release-blocker tag as well)

This was referenced Dec 9, 2016
Closed
Closed
@k8s-github-robot
Copy link
Author

k8s-github-robot commented Dec 10, 2016 •
edited by rmmh

Builds:
ci-kubernetes-e2e-gci-gce-release-1.4 1422
ci-kubernetes-e2e-gci-gce-release-1.5 1260

Failed: [k8s.io] Port forwarding [k8s.io] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/portforward.go:287
Dec 10 13:54:40.636: Expected "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" from server, got ""
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/portforward.go:271

@k8s-github-robot k8s-github-robot mentioned this issue Dec 11, 2016
Closed
@k8s-github-robot
Copy link
Author

k8s-github-robot commented Dec 11, 2016 •
edited by rmmh

Builds:
ci-kubernetes-e2e-gci-gke-release-1.4 1648
ci-kubernetes-e2e-gci-gke-staging-parallel 1617

Failed: [k8s.io] Port forwarding [k8s.io] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/portforward.go:287
Dec 11 15:37:44.856: Expected "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" from server, got "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/portforward.go:271

@sttts sttts modified the milestones: v1.6, v1.5 Dec 12, 2016
This was referenced Dec 12, 2016
Closed
Closed
Closed
This was referenced Dec 21, 2016
Closed
Closed
Closed
@k8s-github-robot k8s-github-robot mentioned this issue Jan 1, 2017
Closed
@k8s-github-robot k8s-github-robot mentioned this issue Jan 15, 2017
Closed
This was referenced Jan 28, 2017
Closed
Closed
Closed
@k8s-github-robot k8s-github-robot mentioned this issue Feb 4, 2017
Closed
@calebamiles
Copy link
Contributor

Closing this issue as the corresponding test has been removed from test/test_owners.csv

cc: @ethernetdan, @pwittrock

@k8s-github-robot k8s-github-robot mentioned this issue Mar 6, 2017
Closed
This was referenced Mar 28, 2017
Closed
Closed
This was referenced Apr 24, 2017
Closed
Closed
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sttts sttts

@bprashanth bprashanth

Labels
kind/flake Categorizes issue or PR as related to a flaky test. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now.
Projects
None yet
Milestone
v1.6
Development

Successfully merging a pull request may close this issue.

WIP: portforward: symmetric half-open connection logic for client and kubelet sttts/kubernetes