Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Explicitly flush headers when proxying #75887

Merged
merged 1 commit into from
Apr 2, 2019

Conversation

liggitt
Copy link
Member

@liggitt liggitt commented Mar 29, 2019

What type of PR is this?
/kind bug

What this PR does / why we need it:
Works around a regression in the go reverse proxy, in which headers from a backend aren't flushed correctly until the backend also sends body content.

Which issue(s) this PR fixes:
Fixes #75837

Special notes for your reviewer:
This is a temporary workaround we can drop once we pick up a version of go that resolves the regression, but this should be picked to make 1.14.1 in the meantime

Does this PR introduce a user-facing change?:

Fixes a regression proxying responses from aggregated API servers which could cause watch requests to hang until the first event was received

/sig api-machinery

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/bug Categorizes issue or PR as related to a bug. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Mar 29, 2019
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: liggitt

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 29, 2019
@liggitt liggitt added this to the v1.14 milestone Mar 29, 2019
@liggitt liggitt added the priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. label Mar 29, 2019
@k8s-ci-robot k8s-ci-robot removed the needs-priority Indicates a PR lacks a `priority/foo` label and requires one. label Mar 29, 2019
Copy link
Contributor

@simonferquel simonferquel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

newFlushHeaderWriter should either return an error for writers not implementing http.Flusher, or be renamed something like tryWrapInFlushHeaderWriter (and potentially returning a (w http.ResonseWriter, ok bool)).

Also, even if the component only requires the flusher interface, it also checks for CloseNotifier and Hijacker interfaces. Is it relevant ? (If it is, a comment about that would be welcome).

@simonferquel
Copy link
Contributor

Anyway, thanks a lot for the responsiveness on this issue, it is much appreciated !

@liggitt
Copy link
Member Author

liggitt commented Mar 29, 2019

updated with clearer comments

@liggitt liggitt force-pushed the flush-headers branch 2 times, most recently from 650d523 to 2d6c548 Compare March 29, 2019 15:34
Copy link
Contributor

@simonferquel simonferquel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm 👍

@liggitt
Copy link
Member Author

liggitt commented Mar 29, 2019

/retest

@fedebongio
Copy link
Contributor

/assign @cheftako

// flusher, hijacker, and closeNotifier are all used by the ReverseProxy implementation.
// if the given writer can't support all three, return the original writer.
if !isFlusher || !isHijacker || !isCloseNotifier {
return w
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like it would be good to track that this happened. Maybe log that we may not flush the headers correctly along with the type of the writer not supporting this?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NM, https://golang.org/src/net/http/httputil/reverseproxy.go?h=handleUpgradeResponse#L518 looks like it will give us the error message I was looking for.

type flushHeadersWriter struct {
http.ResponseWriter
http.Flusher
http.Hijacker
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we know why we need Hijacker (Other than it was in the sample code from golang/go#31125)? I don't see us actually calling Hijack().

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Httputil.ReverseProxy requires it. (Note that all these interfaces are embedded in the structure, to make it implement all of them)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I think that cleared up both my concerns.

@dims
Copy link
Member

dims commented Apr 2, 2019

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 2, 2019
Copy link
Member

@cheftako cheftako left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@liggitt
Copy link
Member Author

liggitt commented Apr 2, 2019

picked to 1.14.1 in #76046

@liggitt liggitt deleted the flush-headers branch April 3, 2019 20:50
@cheftako
Copy link
Member

cheftako commented Apr 4, 2019

This is supposed to be temporary. Do we have a bug or anything similar to tell us to remove it once it is no longer necessary?

@liggitt
Copy link
Member Author

liggitt commented Apr 4, 2019

opened #76154 to track

k8s-ci-robot added a commit that referenced this pull request Apr 4, 2019
…7-upstream-release-1.14

Automated cherry pick of #75887: Explicitly flush headers when proxying
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Kubernetes 1.14.0] Watch on aggregated API resource hangs until first event
6 participants