e2e flake: Proxy version v1 [It] should proxy through a service and a pod

[gmarek]

https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/21251/kubernetes-pull-build-test-e2e-gce/29155/build-log.txt

Failure [132.205 seconds]
Proxy
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/proxy.go:40
  version v1
  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/proxy.go:39
    should proxy through a service and a pod [Conformance] [It]
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/proxy.go:224

    0: path /api/v1/namespaces/e2e-tests-proxy-ehd91/pods/proxy-service-yckdo-cd23f/proxy/ gave error: an error on the server has prevented the request from succeeding

    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/proxy.go:222

cc @kubernetes/goog-cluster

[lavalamp]

#17493 merged recently, but it looks like after this flake happened, so it shouldn't be the cause.

[lavalamp]

Huh, somehow we returned a 0 status code.

Feb 18 07:48:29.875: INFO: /api/v1/namespaces/e2e-tests-proxy-ehd91/pods/proxy-service-yckdo-cd23f/proxy/: no body (0; 3.900034ms)

[wojtek-t]

Another failure:
http://kubekins.dls.corp.google.com:8080/view/Critical%20Builds/job/kubernetes-e2e-gce/12049/

[bprashanth]

Can we dupe this with #21484, or change the title to reflect to 0 status code issue?

[lavalamp]

@bprashanth this is #21484, did you mean something else?

[bprashanth]

Bah i meant #18963

[lavalamp]

OK, so I misread. The '0' is the index of the attempt, not the status code.

[lavalamp]

I think the pod probably just needs a readiness check.

[lavalamp]

I added one & tried it--indeed, the pod spent a fair amount of time in running but not ready state. PR sent.

[bprashanth]

https://pantheon.corp.google.com/storage/browser/kubernetes-jenkins/logs/kubernetes-e2e-gce/12131/?debugUI=CLOUD

[caesarxuchao]

Occurred again:

• Failure [154.406 seconds]
Proxy
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/proxy.go:40
  version v1
  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/proxy.go:39
    should proxy through a service and a pod [Conformance] [It]
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/proxy.go:244

    0: path /api/v1/proxy/namespaces/e2e-tests-proxy-2gz9u/pods/proxy-service-j63nq-ur6c1:80/ gave error: an error on the server has prevented the request from succeeding

    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/proxy.go:242
------------------------------

https://console.cloud.google.com/storage/browser/kubernetes-jenkins/pr-logs/pull/21175/kubernetes-pull-build-test-e2e-gce/30727/?debugUI=CLOUD

[lavalamp]

Huh, it worked through the service but not through the pod directly.

[janetkuo]

https://pantheon.corp.google.com/storage/browser/kubernetes-jenkins/pr-logs/pull/22202/kubernetes-pull-build-test-e2e-gce/31071/artifacts/?debugUI=CLOUD

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/proxy.go:244
0: path /api/v1/namespaces/e2e-tests-proxy-31hny/pods/http:proxy-service-b9v0f-v5kdh:80/proxy/ gave error: an error on the server has prevented the request from succeeding

[a-robinson]

http://pr-test.k8s.io/22367/kubernetes-pull-build-test-e2e-gce/31438/

[spxtr]

This test is exceptionally flaky on GKE (~6% flake rate). Here is a recent flake.

[roberthbailey]

/cc @cjcullen

[gmarek]

This is a P0 since Mar 4th, so over 2 months now. Can we really, really, really prioritize this? @goltermann

[bprashanth]

I think the first problem is wrong triage. Who owns apiserver proxy? I debugged it simply because it was labelled team/cluster and figured it isn't something that worried me. In fact it seemed like a test fix (#21484 (comment))

[gmarek]

@thockin - can you please reassign if cluster team is a wrong one? This is noticeably flaky.

[gmarek]

https://console.cloud.google.com/storage/browser/kubernetes-jenkins/logs/kubernetes-e2e-gke/7303

[lavalamp]

https://console.cloud.google.com/storage/kubernetes-jenkins/logs/kubernetes-e2e-gke/7336/

[gmarek]

@lavalamp - should this be assigned to API machinery team?

[lavalamp]

@gmarek yes, probably :(

[sttts]

Some analysis:

• because the proxy error log output of https://github.com/golang/go/blob/91911e39f0fdfecc5453f9eca7ff74215ffb28a2/src/net/http/httputil/reverseproxy.go#L212 is not visible, the WriteHeader call we see in e2e flake: Proxy version v1 [It] should proxy through a service and a pod #21484 (comment) is actually this: https://github.com/golang/go/blob/91911e39f0fdfecc5453f9eca7ff74215ffb28a2/src/net/http/httputil/reverseproxy.go#L233
  => the 50x error actually originates from the proxy-service pod, not the apiserver
• a 50x from the proxy-service pod also means that it's not a readiness issue as @ixdy mentions in e2e flake: Proxy version v1 [It] should proxy through a service and a pod #21484 (comment) and also tries to fix with a merged PR
• it is always port 80 accessed via the pod url, not via the service name
• it is always the first access to the url
• it does not matter, whether other requests have been made to the pod-service pod before.
• between pod running and the failed request >1.5s can pass. It still fails.

[caesarxuchao]

@sttts do you want to take this one? I haven't started working on it (I was OOO until this week) and have no prior knowledge.

[sttts]

@caesarxuchao @krousey I can continue to investigate that. Feel free to assign it to me.

[dims]

Is this a dup of the bot proposed one? #26210

[krousey]

Closing in favor of the automated report #26210