Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/net/http2: connection-level flow control not returned if stream errors, causes server hang #40423

Closed
jared2501 opened this issue Jul 27, 2020 · 34 comments
Labels
Milestone

Comments

@jared2501
Copy link

@jared2501 jared2501 commented Jul 27, 2020

What version of Go are you using (go version)?

$ go version
go version go1.14.2 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What's the issue

A hang was reported between a gRPC client (grpc-go v1.27.0) hitting a gRPC server in one of our production environments. The client and server are both running on the same host. I captured a core dump of the client and server code to analyze with delve. I noticed that the google.golang.org/grpc/internal/transport.loopyWriter.cbuf.sendQuota was 0 in the client code, which indicates that the client's connection-level send window had run out and was at 0. In the server's core dump, I tracked down the corresponding http2.serverConn and noticed that it's serverConn.inflow.n was set to 0 too. I then tracked down the two places in http2/server.go that call inflow.take and noticed what I believe is the issue in processData:

func (sc *serverConn) processData(f *DataFrame) error {
	...

	if f.Length > 0 {
		// Check whether the client has flow control quota.
		if st.inflow.available() < int32(f.Length) {
			return streamError(id, ErrCodeFlowControl)
		}
		st.inflow.take(int32(f.Length))

		if len(data) > 0 {
			wrote, err := st.body.Write(data)
			if err != nil {
				return streamError(id, ErrCodeStreamClosed)
			}
			if wrote != len(data) {
				panic("internal error: bad Writer")
			}
			st.bodyBytes += int64(len(data))
		}

		// Return any padded flow control now, since we won't
		// refund it later on body reads.
		if pad := int32(f.Length) - int32(len(data)); pad > 0 {
			sc.sendWindowUpdate32(nil, pad)
			sc.sendWindowUpdate32(st, pad)
		}
	}
...

In this code, st.inflow.take is called, but if st.body.Write returns an error then the flow control is not refunded to the client since the code bails and returns a streamError (nor is it added to the st.body's pipeBuffer since pipe.Write returns immediately if it has an error to return).

Side note: st.body.Write may return an error if st.body.Close is called. The server which had this issue is using grpc-go's serverHandlerTransport which does, in fact, call req.Body.Close (see here). A gRPC bi-directional streaming endpoint is running between the client and server, and what i suspect is happening is the client is sending the server data over the bi-di stream while an error happens in the gRPC server that causes the request to end, and therefore req.Body.Close to be called while data is in flight.

Here's what I think a possible fix to net/http2 could look like:

diff --git a/http2/server.go b/http2/server.go
index 01f4ecc..ba3ebd1 100644
--- a/http2/server.go
+++ b/http2/server.go
@@ -1650,6 +1650,7 @@ func (sc *serverConn) processData(f *DataFrame) error {
                if len(data) > 0 {
                        wrote, err := st.body.Write(data)
                        if err != nil {
+                               sc.sendWindowUpdate32(nil, int32(f.Length))
                                return streamError(id, ErrCodeStreamClosed)
                        }
                        if wrote != len(data) {
@jared2501 jared2501 changed the title x/net/http2: connection-level flow control not rendered if stream errors, cases server hang x/net/http2: connection-level flow control not returned if stream errors, cases server hang Jul 27, 2020
@jared2501 jared2501 changed the title x/net/http2: connection-level flow control not returned if stream errors, cases server hang x/net/http2: connection-level flow control not returned if stream errors, causes server hang Jul 27, 2020
@networkimprov
Copy link

@networkimprov networkimprov commented Jul 27, 2020

@gopherbot
Copy link

@gopherbot gopherbot commented Jul 28, 2020

Change https://golang.org/cl/245158 mentions this issue: net/http2: send WINDOW_UPDATE on a body's write failure

@cagedmantis cagedmantis added this to the Backlog milestone Jul 28, 2020
@cagedmantis
Copy link
Contributor

@cagedmantis cagedmantis commented Jul 28, 2020

@jared2501
Copy link
Author

@jared2501 jared2501 commented Jul 28, 2020

Thanks for looking into this so quickly! Once the fix is merged, is there any way we could get this fix backported to 1.14 (or make it's way into a 1.15 release)?

@wader
Copy link

@wader wader commented Aug 5, 2020

Hello, i'm suspecting that I might have run into this issue when talking to GCS using the google-cloud-go package. It's a service that more or less proxies HTTP to GCS and under quite high load it usually stops talking to GCS after ~15min.

When running with GODEBUG=http2debug=2 this is what is see just before it stops doing requests:

...
2020/08/05 15:13:06 http2: Framer 0xc0000c8460: wrote HEADERS flags=END_STREAM|END_HEADERS stream=3067 len=121
2020/08/05 15:13:11 http2: Framer 0xc0000c8460: wrote WINDOW_UPDATE stream=3061 len=4 incr=5120
2020/08/05 15:19:06 http2: Framer 0xc0000c8460: read RST_STREAM stream=21 len=4 ErrCode=INTERNAL_ERROR
2020/08/05 15:19:06 http2: Transport received RST_STREAM stream=21 len=4 ErrCode=INTERNAL_ERROR
2020/08/05 15:19:06 http2: Framer 0xc0000c8460: read RST_STREAM stream=23 len=4 ErrCode=INTERNAL_ERROR
2020/08/05 15:19:06 http2: Transport received RST_STREAM stream=23 len=4 ErrCode=INTERNAL_ERROR
2020/08/05 15:19:06 http2: Framer 0xc0000c8460: read RST_STREAM stream=2725 len=4 ErrCode=INTERNAL_ERROR
2020/08/05 15:19:06 http2: Transport received RST_STREAM stream=2725 len=4 ErrCode=INTERNAL_ERROR
2020/08/05 15:19:06 http2: Framer 0xc0000c8460: read RST_STREAM stream=2971 len=4 ErrCode=INTERNAL_ERROR
2020/08/05 15:19:06 http2: Transport received RST_STREAM stream=2971 len=4 ErrCode=INTERNAL_ERROR
2020/08/05 15:19:06 http2: Framer 0xc0000c8460: read RST_STREAM stream=3037 len=4 ErrCode=INTERNAL_ERROR
2020/08/05 15:19:06 http2: Transport received RST_STREAM stream=3037 len=4 ErrCode=INTERNAL_ERROR
2020/08/05 15:19:06 http2: Framer 0xc0000c8460: read RST_STREAM stream=3053 len=4 ErrCode=INTERNAL_ERROR
2020/08/05 15:19:06 http2: Transport received RST_STREAM stream=3053 len=4 ErrCode=INTERNAL_ERROR
2020/08/05 15:19:06 http2: Framer 0xc0000c8460: read RST_STREAM stream=3067 len=4 ErrCode=INTERNAL_ERROR
2020/08/05 15:19:06 http2: Transport received RST_STREAM stream=3067 len=4 ErrCode=INTERNAL_ERROR
2020/08/05 15:19:06 http2: Framer 0xc0000c8460: read RST_STREAM stream=3061 len=4 ErrCode=INTERNAL_ERROR
2020/08/05 15:19:06 http2: Transport received RST_STREAM stream=3061 len=4 ErrCode=INTERNAL_ERROR
2020/08/05 15:19:06 http2: Framer 0xc0000c8460: read RST_STREAM stream=2717 len=4 ErrCode=INTERNAL_ERROR
2020/08/05 15:19:06 http2: Transport received RST_STREAM stream=2717 len=4 ErrCode=INTERNAL_ERROR
2020/08/05 15:19:06 http2: Framer 0xc0000c8460: read RST_STREAM stream=547 len=4 ErrCode=INTERNAL_ERROR
2020/08/05 15:19:06 http2: Transport received RST_STREAM stream=547 len=4 ErrCode=INTERNAL_ERROR
2020/08/05 15:19:06 http2: Transport encoding header ":authority" = "storage.googleapis.com"
2020/08/05 15:19:06 http2: Transport encoding header ":method" = "GET"
2020/08/05 15:19:06 RoundTrip failure: stream error: stream ID 21; INTERNAL_ERROR
...

Or is this something else or i'm doing something wrong?

@networkimprov
Copy link

@networkimprov networkimprov commented Aug 5, 2020

@jared2501
Copy link
Author

@jared2501 jared2501 commented Aug 13, 2020

Heya, just checking in post-1.15 release to see if we can get the CR above merged? I wouldn't be surprised if @wadells' issue above is the result of this bug, and I suspect it wouldn't be surprised if this bug is affecting several other users (basically anyone that uses the gRPC http.Handler server).

cc @menghanl

@wader
Copy link

@wader wader commented Aug 18, 2020

Let me know if there something i can help with, try to create a small reproduction case etc?

Does anyone know if this is regression? is it an issue with specific combination of client and server implementation and version?

@jared2501
Copy link
Author

@jared2501 jared2501 commented Aug 25, 2020

@wader it's super easy to test if the patch fixes your issue - you can just apply it to a fork of https://github.com/golang/net, and then use go module's replace statement to depend upon your fork. If you can no longer repro the issue then this is the root cause!

@wader
Copy link

@wader wader commented Aug 25, 2020

@jared2501 Thanks, good idea. Tried with patched net package (hope i patched it correctly, seems c is now sc) but get similar errors.

...
2020/08/25 17:03:25 http2: Framer 0xc0002fe1c0: read RST_STREAM stream=63 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:03:25 http2: Transport received RST_STREAM stream=63 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:03:25 http2: Framer 0xc0002fe1c0: read RST_STREAM stream=93 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:03:25 http2: Transport received RST_STREAM stream=93 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:03:25 http2: Framer 0xc0002fe1c0: read RST_STREAM stream=73 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:03:25 http2: Transport received RST_STREAM stream=73 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:03:25 http2: Framer 0xc0002fe1c0: read RST_STREAM stream=77 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:03:25 http2: Transport received RST_STREAM stream=77 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:03:25 http2: Framer 0xc0002fe1c0: read RST_STREAM stream=87 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:03:25 http2: Transport received RST_STREAM stream=87 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:03:25 http2: Framer 0xc0002fe1c0: read RST_STREAM stream=79 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:03:25 http2: Transport received RST_STREAM stream=79 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:03:25 http2: Framer 0xc0002fe1c0: read RST_STREAM stream=95 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:03:25 http2: Transport received RST_STREAM stream=95 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:03:25 http2: Framer 0xc0002fe1c0: read RST_STREAM stream=91 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:03:25 http2: Transport received RST_STREAM stream=91 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:03:25 http2: Framer 0xc0002fe1c0: read RST_STREAM stream=3449 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:03:25 http2: Transport received RST_STREAM stream=3449 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:03:25 http2: Framer 0xc0002fe1c0: read RST_STREAM stream=3525 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:03:25 http2: Transport received RST_STREAM stream=3525 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:03:25 http2: Framer 0xc0002fe1c0: read RST_STREAM stream=97 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:03:25 http2: Transport received RST_STREAM stream=97 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:03:25 http2: Framer 0xc0002fe1c0: read RST_STREAM stream=99 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:03:25 http2: Transport received RST_STREAM stream=99 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:03:25 RoundTrip failure: stream error: stream ID 77; INTERNAL_ERROR
2020/08/25 17:03:25 http2: Framer 0xc0002fe1c0: read RST_STREAM stream=3437 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:03:25 http2: Transport received RST_STREAM stream=3437 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:03:25 http2: Framer 0xc0002fe1c0: read RST_STREAM stream=3439 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:03:25 http2: Transport received RST_STREAM stream=3439 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:03:25 http2: Framer 0xc0002fe1c0: read RST_STREAM stream=3519 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:03:25 http2: Transport received RST_STREAM stream=3519 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:03:25 http2: Framer 0xc0002fe1c0: read RST_STREAM stream=3527 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:03:25 http2: Transport received RST_STREAM stream=3527 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:03:25 RoundTrip failure: stream error: stream ID 73; INTERNAL_ERROR
2020/08/25 17:03:25 RoundTrip failure: stream error: stream ID 93; INTERNAL_ERROR
...

Patch with HEAD at c89045814202410a2d67ec20ecf177ec77ceae7f

diff --git a/http2/server.go b/http2/server.go
index 345b7cd..8f4d17a 100644
--- a/http2/server.go
+++ b/http2/server.go
@@ -1694,6 +1694,7 @@ func (sc *serverConn) processData(f *DataFrame) error {
                if len(data) > 0 {
                        wrote, err := st.body.Write(data)
                        if err != nil {
+                               sc.sendWindowUpdate32(nil, int32(f.Length))
                                return streamError(id, ErrCodeStreamClosed)
                        }
                        if wrote != len(data) {

go.mod

module ...

go 1.14

require (
	cloud.google.com/go v0.62.0 // indirect
	gocloud.dev v0.20.0
)

replace golang.org/x/net => /Users/wader/src/net
@wader
Copy link

@wader wader commented Aug 25, 2020

Left it running for while after the error and it seems to be able to service some request after a while. Not sure why it's exactly 10 minutes between cancel and new errors. Could be some timeout duration in the client using the service also i guess?

2020/08/25 17:29:29 http2: Framer 0xc0002fe1c0: wrote RST_STREAM stream=3843 len=4 ErrCode=CANCEL
2020/08/25 17:35:29 http2: Framer 0xc0002fe1c0: read RST_STREAM stream=3901 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:35:29 http2: Transport received RST_STREAM stream=3901 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:35:29 http2: Framer 0xc0002fe1c0: read RST_STREAM stream=3905 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:35:29 http2: Transport received RST_STREAM stream=3905 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:35:29 http2: Framer 0xc0002fe1c0: read RST_STREAM stream=3903 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:35:29 http2: Transport received RST_STREAM stream=3903 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:35:29 http2: Framer 0xc0002fe1c0: read RST_STREAM stream=3915 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:35:29 http2: Transport received RST_STREAM stream=3915 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:35:29 http2: Framer 0xc0002fe1c0: read RST_STREAM stream=3907 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:35:29 http2: Transport received RST_STREAM stream=3907 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:35:29 http2: Framer 0xc0002fe1c0: read RST_STREAM stream=3913 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:35:29 http2: Transport received RST_STREAM stream=3913 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:35:29 http2: Framer 0xc0002fe1c0: read RST_STREAM stream=3911 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:35:29 http2: Transport received RST_STREAM stream=3911 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:35:29 http2: Framer 0xc0002fe1c0: read RST_STREAM stream=3909 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:35:29 http2: Transport received RST_STREAM stream=3909 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:35:29 http2: Framer 0xc0002fe1c0: read RST_STREAM stream=3917 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:35:29 http2: Transport received RST_STREAM stream=3917 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:35:29 http2: Framer 0xc0002fe1c0: read RST_STREAM stream=3867 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:35:29 http2: Transport received RST_STREAM stream=3867 len=4 ErrCode=INTERNAL_ERROR
2020/08/25 17:35:29 RoundTrip failure: stream error: stream ID 3903; INTERNAL_ERROR
2020/08/25 17:35:29 RoundTrip failure: stream error: stream ID 3901; INTERNAL_ERROR
2020/08/25 17:35:29 RoundTrip failure: stream error: stream ID 3913; INTERNAL_ERROR
@jared2501
Copy link
Author

@jared2501 jared2501 commented Aug 27, 2020

Got ya, yeah sounds like it's not the issue then. FWIW, I've applied the linked patch to our production environments and have seen no further hangs. Before the patch, we were getting hangs once a day.

@jared2501
Copy link
Author

@jared2501 jared2501 commented Aug 27, 2020

@networkimprov @cagedmantis @fraenkel - sorry for the ping here, but any way we could find a person to CR https://go-review.googlesource.com/c/net/+/245158/? The fix and test look simple enough to me, and I can confirm this fix works in our prod env.

@networkimprov
Copy link

@networkimprov networkimprov commented Aug 27, 2020

@wader
Copy link

@wader wader commented Aug 27, 2020

@jared2501 Ok, i get those error after 30min-1h. Are these kind of error "normal" to get if you run into throttling/quotas etc?

@jared2501
Copy link
Author

@jared2501 jared2501 commented Sep 9, 2020

@andybons @ianlancetaylor @rsc - just a quick ping on this. We've applied the above patch to our production environment and seen it fix the above issues. Would it be possible to get https://go-review.googlesource.com/c/net/+/245158/ merged and back ported onto 1.15 and/or 1.14?

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Sep 14, 2020

@gopherbot Please open backport issues.

This problem can reportedly cause an HTTP/2 connection to hang. There doesn't seem to be any reasonable workaround.

@gopherbot
Copy link

@gopherbot gopherbot commented Sep 14, 2020

Backport issue(s) opened: #41386 (for 1.14), #41387 (for 1.15).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

@jared2501
Copy link
Author

@jared2501 jared2501 commented Sep 26, 2020

@ianlancetaylor - any chance we can find someone else to +1 the CR?

@networkimprov
Copy link

@networkimprov networkimprov commented Sep 26, 2020

cc @odeke-em as possible reviewer

@odeke-em
Copy link
Member

@odeke-em odeke-em commented Sep 26, 2020

Thank you for the ping @networkimprov :) I've provided a review after Ian's and should be ready to roll out soon, and then we'll need to pull it into net/http/h2_bundle.go

@jared2501
Copy link
Author

@jared2501 jared2501 commented Sep 29, 2020

Woo, thank you! Can't wait to get this in the next 1.14/1.15 hot fixes!

@jared2501
Copy link
Author

@jared2501 jared2501 commented Sep 30, 2020

Btw @networkimprov - does anything have to be done to get the cherry picks made, or do those get picked up automatically?

@networkimprov
Copy link

@networkimprov networkimprov commented Sep 30, 2020

If no changes are required for the 1.14 & 1.15 branches, I think the release folks can cherrypick the CL. You might want to ping the CL author on the two backport issues to ask about it.

@gopherbot
Copy link

@gopherbot gopherbot commented Sep 30, 2020

Change https://golang.org/cl/258359 mentions this issue: src/go.mod, net/http: update bundled and latest golang.org/x/net

@odeke-em
Copy link
Member

@odeke-em odeke-em commented Sep 30, 2020

Thank you @jared2501 and @networkimprov for raising it!
So this is a tricky one to backport given that it is from x/net/http2 that then needs to be bundled into net/http/h2_bundle.go, so to begin with:
1.) I have mailed out CL 258359 firstly to net/http/h2_bundle.go
2.) After that merge, I'll make backports to Go1.14 and Go1.15
and after that merge, we'll be good to go.
However, please note that those issues are still "CherryPickCandidate" labelled, thus will only be released if approved and when the releases are being cut.

@dmitshur dmitshur modified the milestones: Backlog, Go1.16 Sep 30, 2020
@gopherbot
Copy link

@gopherbot gopherbot commented Sep 30, 2020

Change https://golang.org/cl/258478 mentions this issue: [release-branch.go1.15] net/http2: send WINDOW_UPDATE on a body's write failure

@gopherbot
Copy link

@gopherbot gopherbot commented Sep 30, 2020

Change https://golang.org/cl/258497 mentions this issue: [release-branch.go1.14] net/http2: send WINDOW_UPDATE on a body's write failure

gopherbot pushed a commit that referenced this issue Sep 30, 2020
Updates x/net/http2 to git rev 5d4f7005572804eaf7f5ecdd2473a62557f733ba

    http2: send WINDOW_UPDATE on a body's write failure
    https://golang.org/cl/245158 (fixes #40423)

also updates the vendored version of golang.org/x/net as per

$ go get golang.org/x/net@5d4f700557
$ go mod tidy
$ go mod vendor
$ go generate -run bundle std

For #40423.

Change-Id: I3270d0fb6f28889266596f7365d36d30ef2bb368
Reviewed-on: https://go-review.googlesource.com/c/go/+/258359
Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com>
TryBot-Result: Go Bot <gobot@golang.org>
Trust: Emmanuel Odeke <emm.odeke@gmail.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Sep 30, 2020

Change https://golang.org/cl/258537 mentions this issue: [release-branch.go1.14] net/http: backport HTTP2 send WINDOW_UPDATE on a body's write failure

@gopherbot
Copy link

@gopherbot gopherbot commented Sep 30, 2020

Change https://golang.org/cl/258538 mentions this issue: [release-branch.go1.14] net/http: backport HTTP2 send WINDOW_UPDATE on a body's write failure

@gopherbot
Copy link

@gopherbot gopherbot commented Sep 30, 2020

Change https://golang.org/cl/258540 mentions this issue: [release-branch.go1.15] net/http: backport HTTP2 send WINDOW_UPDATE on a body's write failure

@jared2501
Copy link
Author

@jared2501 jared2501 commented Oct 1, 2020

Awesome, thank you @odeke-em & @networkimprov for your quick responses and effort pushing this through!

gopherbot pushed a commit to golang/net that referenced this issue Oct 8, 2020
…te failure

When the body.Write fails during processData, the connection flow
control must be updated to account for the data received. The connection's
WINDOW_UPDATE should reflect the amount of data that was not successfully
written. The stream is about to be closed, so no update is required.

Updates golang/go#40423.
Fixes golang/go#41387.

Change-Id: I546597cedf3715e6617babcb3b62140bf1857a27
Reviewed-on: https://go-review.googlesource.com/c/net/+/245158
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com>
TryBot-Result: Go Bot <gobot@golang.org>
Trust: Emmanuel Odeke <emm.odeke@gmail.com>
(cherry picked from commit 5d4f700)
Reviewed-on: https://go-review.googlesource.com/c/net/+/258478
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
Trust: Dmitri Shuralyov <dmitshur@golang.org>
gopherbot pushed a commit to golang/net that referenced this issue Oct 12, 2020
…te failure

When the body.Write fails during processData, the connection flow
control must be updated to account for the data received. The connection's
WINDOW_UPDATE should reflect the amount of data that was not successfully
written. The stream is about to be closed, so no update is required.

Updates golang/go#40423.
Fixes golang/go#41386.

Change-Id: I546597cedf3715e6617babcb3b62140bf1857a27
Reviewed-on: https://go-review.googlesource.com/c/net/+/245158
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com>
TryBot-Result: Go Bot <gobot@golang.org>
Trust: Emmanuel Odeke <emm.odeke@gmail.com>
(cherry picked from commit 5d4f700)
Reviewed-on: https://go-review.googlesource.com/c/net/+/258497
Trust: Dmitri Shuralyov <dmitshur@golang.org>
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Oct 13, 2020

Change https://golang.org/cl/261919 mentions this issue: [release-branch.go1.15] src, net/http: update vendor, h2_bundle.go regeneration

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 21, 2020

Change https://golang.org/cl/264058 mentions this issue: [release-branch.go1.15-bundle] net/http2: send WINDOW_UPDATE on a body's write failure

gopherbot pushed a commit to golang/net that referenced this issue Oct 22, 2020
…y's write failure

When the body.Write fails during processData, the connection flow
control must be updated to account for the data received. The connection's
WINDOW_UPDATE should reflect the amount of data that was not successfully
written. The stream is about to be closed, so no update is required.

Updates golang/go#40423.
For golang/go#41387.

Change-Id: I546597cedf3715e6617babcb3b62140bf1857a27
Reviewed-on: https://go-review.googlesource.com/c/net/+/245158
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com>
TryBot-Result: Go Bot <gobot@golang.org>
Trust: Emmanuel Odeke <emm.odeke@gmail.com>
(cherry picked from commit 5d4f700)
Reviewed-on: https://go-review.googlesource.com/c/net/+/264058
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
Trust: Dmitri Shuralyov <dmitshur@golang.org>
gopherbot pushed a commit that referenced this issue Oct 22, 2020
…undle.go

Features CL:

    net/http2: send WINDOW_UPDATE on a body's write failure (fixes #41386)
    https://golang.org/cl/258497

Created by:

go get -d golang.org/x/net@release-branch.go1.14
go mod tidy
go mod vendor
go generate -run=bundle std

Updates #40423
Fixes #41386

Change-Id: I3e75527d381dd4c4262db5f2ff755029d448c48b
Reviewed-on: https://go-review.googlesource.com/c/go/+/258538
Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Trust: Emmanuel Odeke <emm.odeke@gmail.com>
TryBot-Result: Go Bot <gobot@golang.org>
gopherbot pushed a commit that referenced this issue Oct 22, 2020
…undle.go

Features CL:

    net/http2: send WINDOW_UPDATE on a body's write failure
    https://golang.org/cl/258478 (updates #41387)

Created by:

go mod edit -replace=golang.org/x/net=golang.org/x/net@release-branch.go1.15-bundle
GOFLAGS='-mod=mod' go generate -run=bundle std
go mod edit -dropreplace=golang.org/x/net
go get -d golang.org/x/net@release-branch.go1.15
go mod tidy
go mod vendor

Updates #40423
Fixes #41387

Change-Id: I052037d6b6ed38b9d9782e19b8ce283875354c92
Reviewed-on: https://go-review.googlesource.com/c/go/+/258540
Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Trust: Emmanuel Odeke <emm.odeke@gmail.com>
airshipbot pushed a commit to airshipit/promenade that referenced this issue Oct 24, 2020
There are several kubernetes bugs [0,1,2] involving connection problems
that seem related to the Go net/http2 library, where the stream state
and connection state can get out of sync. This can manifest as a kubelet
issue, where the node status gets stuck in a NotReady state, but can
also happen elsewhere.

In newer versions of the Go libraries some issues are fixed [3,4], but
the fixes are not present in k8s 1.18.

This change disables http2 in kube-apiserver and webhook-apiserver. This
should be sufficient to avoid the majority of the issues, as disabling
on one side of the connection is enough, and apiserver is generally
either the client or the server.

0: kubernetes/kubernetes#87615
1: kubernetes/kubernetes#80313
2: kubernetes/client-go#374
3: golang/go#40423
4: golang/go#40201

Change-Id: Id693a7201acffccbc4b3db8f4e4b96290fd50288
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
8 participants