x/net/http2: failures with "client: missing connection" in TestTransportBodyReadError_*

[bcmills]

--- FAIL: TestTransportBodyReadError_Immediately (0.00s)
    transport_test.go:4590: server: ReadFrame = [FrameHeader WINDOW_UPDATE len=4], <nil>
    transport_test.go:774: client: missing connection
    transport_test.go:4590: server: ReadFrame = [FrameHeader HEADERS flags=END_HEADERS stream=1 len=36], <nil>
    transport_test.go:4590: server: ReadFrame = <nil>, read tcp4 127.0.0.1:51605->127.0.0.1:39284: use of closed network connection
FAIL
exitcode=1FAIL	golang.org/x/net/http2	22.996s

2021-01-19T19:43:25-5f4716e/android-amd64-emu
2020-10-02T20:24:02-0a1ea39/android-amd64-emu
2020-05-20T00:47:42-59133d7/android-amd64-emu
2019-12-04T02:50:24-5ee1b9f/aix-ppc64
2019-10-14T21:28:45-da9a3fd/android-amd64-emu
2019-09-26T02:58:31-c00fd9a/android-386-emu
2019-09-09T00:30:24-a7b1673/linux-ppc64-buildlet
2019-08-13T14:13:03-74dc4d7/dragonfly-amd64
2019-07-24T01:30:45-ca1201d/android-386-emu
2019-04-24T11:20:56-4829fb1/linux-arm64-packet
2019-04-03T14:48:56-b630fd6/android-386-emu
2019-03-26T08:36:53-a33f666/linux-amd64-race
2018-11-14T22:03:01-adae6a3/freebsd-arm-paulzhol

CC @bradfitz @tombergan @empijei @neild

[networkimprov]

cc @fraenkel

[fraenkel]

The failure makes sense when the connection is closed. It is unclear who/what is closing the connection though.

[bcmills]

greplogs --dashboard -md -l -e '(?m)FAIL: TestTransportBodyReadError_.*\n( .+)*client: missing connection' --since=2021-02-17

2021-11-12T20:21:33-69e39ba-b954f58/android-386-emu
2021-11-05T04:20:33-ce13745-0a5ca24/linux-ppc64le-buildlet
2021-11-04T21:53:05-ce13745-76c48e9/linux-ppc64-buildlet
2021-10-13T04:00:15-caeb26a-e8f99da/freebsd-arm-paulzhol
2021-09-15T03:29:46-943fd67-4847c47/android-386-emu
2021-04-21T04:27:01-798c215-f53c2fa/android-386-emu
2021-03-29T15:50:54-61e0566-be87281/android-386-emu
2021-03-15T10:23:23-e18ecbb-6ccb5c4/dragonfly-amd64-5_8

[bcmills]

The failure makes sense when the connection is closed. It is unclear who/what is closing the connection though.

Does that theory also match this write: broken pipe failure mode?

--- FAIL: TestTransportBodyReadError_Immediately (0.00s)
    transport_test.go:818: write tcp4 127.0.0.1:50091->127.0.0.1:50092: write: broken pipe
FAIL
FAIL	golang.org/x/net/http2	10.802s

greplogs --dashboard -md -l -e '(?m)FAIL: TestTransportBodyReadError_.*\n( .+)*write: broken pipe' --since=2021-01-01

2022-01-05T18:14:20-5b0dc2d-ab70d7c/darwin-amd64-nocgo

[bcmills]

greplogs --dashboard -md -l -e '(?m)FAIL: TestTransportBodyReadError_.*\n( .+\n)* .+(?:client: missing connection|write: broken pipe)' --since=2021-11-13

2022-01-05T18:14:31-5b0dc2d-a845a56/android-amd64-emu
2022-01-05T18:14:20-5b0dc2d-ab70d7c/darwin-amd64-nocgo
2021-12-09T13:36:38-491a49a-474ebb9/android-amd64-emu
2021-12-03T21:25:05-4852103-d20a0bf/linux-amd64-race
2021-11-19T18:35:56-6a13c67-ac0da79/dragonfly-amd64
2021-11-19T00:05:59-6a13c67-e8cda0a/dragonfly-amd64

[bcmills]

The past month or so includes failures on linux/amd64 and darwin/amd64, which are both first-class ports, and x/net/http2 is vendored into net/http.

This is not a new failure mode, but it is occurring frequently enough to be a problem. Marking as release-blocker, but for Go 1.19 (since it isn't new and could take significant time to debug).

[heschi]

ping @neild

[bcmills]

greplogs --dashboard -md -l -e '(?m)FAIL: TestTransportBodyReadError_.*\n( .+\n)* .+(?:client: missing connection|write: broken pipe)' --since=2022-01-06

2022-03-29T10:35:56-de3da57-9b90838/android-amd64-emu
2022-03-29T02:25:12-de3da57-1724077/freebsd-arm-paulzhol
2022-03-10T01:34:17-27dd868-604140d/dragonfly-amd64
2022-03-03T17:28:58-27dd868-78070ec/dragonfly-amd64
2022-03-01T19:46:09-27dd868-d6d2ebb/linux-386-sid
2022-02-25T14:32:41-3bcbab3-55e5b03/linux-ppc64-buildlet
2022-02-24T22:11:25-cd36cc0-c0840a7/dragonfly-amd64
2022-02-18T00:51:15-cd36cc0-a222963/dragonfly-amd64
2022-02-14T19:02:39-cd36cc0-ecf3b39/dragonfly-amd64
2022-02-14T02:34:05-cd36cc0-93b5309/linux-ppc64le-buildlet
2022-02-13T02:11:11-cd36cc0-5bd7348/darwin-amd64-12_0
2022-02-09T21:52:21-cd36cc0-2e2ef31/dragonfly-amd64
2022-02-07T20:21:30-cd36cc0-49030c8/android-amd64-emu
2022-01-31T20:35:07-cd36cc0-360e1b8/dragonfly-amd64

[bcmills]

greplogs --dashboard -md -l -e '(?m)FAIL: TestTransportBodyReadError_.*\n( .+\n)* .+(?:client: missing connection|write: broken pipe)' --since=2022-03-30

2022-04-21T19:06:52-1850ba1-aac1d3a/linux-amd64-race
2022-04-18T20:11:49-a630d4f-3df9df8/linux-amd64-race
2022-04-07T20:39:32-749bd19-6f6942e/android-amd64-emu
2022-03-31T22:21:57-de3da57-fb2a9d2/dragonfly-amd64
2022-03-30T19:14:04-de3da57-1ac1658/android-amd64-emu

[bcmills]

greplogs -l -e '(?m)FAIL: TestTransportBodyReadError_.*\n( .+\n)* .+(?:client: missing connection|write: broken pipe)' --since=2022-04-22
2022-05-11T13:30:43-2871e0c-636c5f0/android-amd64-emu

[bcmills]

greplogs -l -e '(?m)FAIL: TestTransportBodyReadError_.*\n( .+\n)* .+(?:client: missing connection|write: broken pipe)' --since=2022-05-12
2022-05-16T09:35:17-9564170-2a6e138/linux-ppc64-buildlet
2022-05-16T09:31:42-9564170-3caf67d/dragonfly-amd64

[heschi]

@neild is this OK after beta 1?

[gopherbot]

Change https://go.dev/cl/410934 mentions this issue: http2: add newly dialed conns to the pool before signaling completion