Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[k8s.io] Proxy version v1 should proxy through a service and a pod [Conformance] {Kubernetes e2e suite} #26210

Closed
k8s-github-robot opened this issue May 24, 2016 · 360 comments
Closed

[k8s.io] Proxy version v1 should proxy through a service and a pod [Conformance] {Kubernetes e2e suite} #26210

k8s-github-robot opened this issue May 24, 2016 · 360 comments
Assignees
@sttts
Labels
area/provider/gcp Issues or PRs related to gcp provider kind/flake Categorizes issue or PR as related to a flaky test. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery.
Milestone
v1.5

Comments

@k8s-github-robot
Copy link

k8s-github-robot commented May 24, 2016 •
edited by rmmh

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/7676/

Failed: [k8s.io] Proxy version v1 should proxy through a service and a pod [Conformance] {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/proxy.go:248
0: path /api/v1/namespaces/e2e-tests-proxy-5q81p/pods/proxy-service-xugyo-n1z2c:80/proxy/ gave error: an error on the server has prevented the request from succeeding
@k8s-github-robot k8s-github-robot added the kind/flake Categorizes issue or PR as related to a flaky test. label May 24, 2016
@dims
Copy link
Member

dims commented May 24, 2016 •
edited

We are treating http error code 0 with no body as an error:
"May 24 14:24:12.349: INFO: /api/v1/namespaces/e2e-tests-proxy-5q81p/pods/proxy-service-xugyo-n1z2c:80/proxy/: no body (0; 13.886231ms)"

here:
https://github.com/dims/kubernetes/blob/326b2132311214b274b833584578fd7f81d6a659/test/e2e/proxy.go#L226-L226

May be we should relax this a bit? Since this was the first one and since we got 200 OK in the next 19 calls.

@k8s-github-robot
Copy link
Author

https://console.cloud.google.com/storage/kubernetes-jenkins/logs/kubernetes-e2e-gke/7696/

Failed: [k8s.io] Proxy version v1 should proxy through a service and a pod [Conformance] {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/proxy.go:248
0: path /api/v1/namespaces/e2e-tests-proxy-eaxzs/pods/proxy-service-mxjpn-cgqdh:80/proxy/ gave error: an error on the server has prevented the request from succeeding

@k8s-github-robot
Copy link
Author

https://console.cloud.google.com/storage/kubernetes-jenkins/logs/kubernetes-e2e-gke/7706/

Failed: [k8s.io] Proxy version v1 should proxy through a service and a pod [Conformance] {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/proxy.go:248
0: path /api/v1/proxy/namespaces/e2e-tests-proxy-f2s1s/pods/proxy-service-znd39-iv25q:80/ gave error: an error on the server has prevented the request from succeeding

@0xmichalis 0xmichalis mentioned this issue May 25, 2016
Merged
@dims
Copy link
Member

dims commented May 25, 2016

Isn't this a dup of #21484 ?

@dims dims mentioned this issue May 25, 2016
Closed
@sttts
Copy link
Contributor

sttts commented May 25, 2016

@dims yes, it's the same issue.

@sttts
Copy link
Contributor

sttts commented May 25, 2016

@dims in #21484 you can find backtraces from the apiserver during that request. It looks like a 501 or 503 error in fact.

Relaxing the error condition will fix the flake, but probably hide some other problem.

@krousey krousey added the priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. label May 25, 2016
@krousey
Copy link
Contributor

krousey commented May 25, 2016

Transferred assignee and priority from de-duped bug.

@dims
Copy link
Member

dims commented May 25, 2016

@sttts interesting, i haven't been able to find 501 or 503 in kubernetes-e2e-gke logs that i pulled this morning

@krousey
Copy link
Contributor

krousey commented May 25, 2016

@dims These would be in the API server log on the master. Due to GKE masters being hosted in a separate project, our Jenkins builder doesn't collect them. The logs @sttts found in the duped issue were from vanilla GCE runs where we do collect the master logs.

@dims
Copy link
Member

dims commented May 25, 2016

Thanks @krousey i am just learning the ropes.

@metral metral mentioned this issue May 25, 2016
Merged
@k8s-github-robot
Copy link
Author

https://console.cloud.google.com/storage/kubernetes-jenkins/logs/kubernetes-e2e-gke/7766/

Failed: [k8s.io] Proxy version v1 should proxy through a service and a pod [Conformance] {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/proxy.go:255
0: path /api/v1/proxy/namespaces/e2e-tests-proxy-s3msq/pods/http:proxy-service-w5jj4-frgk9:80/ gave error: an error on the server has prevented the request from succeeding

@k8s-github-robot
Copy link
Author

https://console.cloud.google.com/storage/kubernetes-jenkins/logs/kubernetes-e2e-gke/7779/

Failed: [k8s.io] Proxy version v1 should proxy through a service and a pod [Conformance] {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/proxy.go:255
0: path /api/v1/namespaces/e2e-tests-proxy-5anbx/pods/proxy-service-9at1i-mhzfb/proxy/ gave error: an error on the server has prevented the request from succeeding

@k8s-github-robot
Copy link
Author

https://console.cloud.google.com/storage/kubernetes-jenkins/logs/kubernetes-e2e-gke/7782/

Failed: [k8s.io] Proxy version v1 should proxy through a service and a pod [Conformance] {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/proxy.go:255
0: path /api/v1/proxy/namespaces/e2e-tests-proxy-527hn/pods/http:proxy-service-qelss-nu9on:80/ gave error: an error on the server has prevented the request from succeeding

@k8s-github-robot
Copy link
Author

https://console.cloud.google.com/storage/kubernetes-jenkins/logs/kubernetes-e2e-gke/7785/

Failed: [k8s.io] Proxy version v1 should proxy through a service and a pod [Conformance] {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/proxy.go:255
0: path /api/v1/namespaces/e2e-tests-proxy-xk9bm/pods/proxy-service-wg1kw-xcelv/proxy/ gave error: an error on the server has prevented the request from succeeding

@k8s-github-robot
Copy link
Author

https://console.cloud.google.com/storage/kubernetes-jenkins/logs/kubernetes-e2e-gke/7813/

Failed: [k8s.io] Proxy version v1 should proxy through a service and a pod [Conformance] {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/proxy.go:255
0: path /api/v1/namespaces/e2e-tests-proxy-whfo2/pods/http:proxy-service-f5tot-5qlho:80/proxy/ gave error: an error on the server has prevented the request from succeeding

@janetkuo janetkuo mentioned this issue May 27, 2016
Merged
@sttts
Copy link
Contributor

sttts commented May 27, 2016

With the error logging merged, we now know that there is nothing logged in the proxy-service pod:

May 26 12:56:33.201: INFO: Pod proxy-service-f5tot-5qlho has the following error logs:

@wojtek-t wojtek-t mentioned this issue May 27, 2016
Merged
@sttts
Copy link
Contributor

sttts commented May 27, 2016

@krousey @gmarek in #21484 (comment) it was mentioned that Google-internally there is access to the gke apiserver logs. Could one of you try to get access to the relevant part in one of the upper gke flakes?

Recently it looks like this flake happens on gke only. Here is one for example: https://console.cloud.google.com/storage/browser/kubernetes-jenkins/logs/kubernetes-e2e-gke/7813

@krousey
Copy link
Contributor

krousey commented May 27, 2016

@sttts #26379 😢

Is there one before 5/23 that would help?

@sttts
Copy link
Contributor

sttts commented May 27, 2016

@krousey https://console.cloud.google.com/storage/kubernetes-jenkins/logs/kubernetes-e2e-gke/7336/ is the last in the old issue, 8 days old.

@sttts sttts mentioned this issue May 27, 2016
Merged
@k8s-github-robot
Copy link
Author

https://console.cloud.google.com/storage/kubernetes-jenkins/logs/kubernetes-e2e-gke/7877/

Failed: [k8s.io] Proxy version v1 should proxy through a service and a pod [Conformance] {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/proxy.go:255
0: path /api/v1/namespaces/e2e-tests-proxy-qlk8d/pods/http:proxy-service-ncbr2-2nqf8:80/proxy/ gave error: an error on the server has prevented the request from succeeding

This was referenced Jan 3, 2017
Closed
Closed
This was referenced Jan 13, 2017
Closed
Closed
Closed
This was referenced Jan 23, 2017
Closed
Closed
Closed
Closed
@k8s-github-robot k8s-github-robot mentioned this issue Feb 5, 2017
Closed
This was referenced Feb 27, 2017
Closed
Closed
Closed
@k8s-github-robot k8s-github-robot mentioned this issue Mar 13, 2017
Closed
This was referenced Mar 20, 2017
Closed
Closed
Closed
Closed
Closed
Closed
This was referenced Mar 28, 2017
Closed
Closed
This was referenced Apr 7, 2017
Closed
Closed
This was referenced Apr 21, 2017
Closed
Closed
Closed
This was referenced Apr 28, 2017
Closed
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sttts sttts

Labels
area/provider/gcp Issues or PRs related to gcp provider kind/flake Categorizes issue or PR as related to a flaky test. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery.
Projects
None yet
Milestone
v1.5
Development

No branches or pull requests