net/http: new TestDontCacheBrokenHTTP2Conn fails #35113

cuonglm · 2019-10-23T15:40:19Z

Build log: https://storage.googleapis.com/go-build-log/76c387a0/linux-amd64-race_356de3ff.log

--- FAIL: TestDontCacheBrokenHTTP2Conn (0.03s)
    transport_test.go:5885: for iteration 5, doBreak=false; unexpected error Get "https://127.0.0.1:35315": some write error
    transport_test.go:5889: GotConn calls = 0; want 1
FAIL
FAIL	net/http	8.568s
FAIL
2019/10/23 15:34:13 Failed: exit status 1

cuonglm · 2019-10-23T15:42:27Z

This happens when I test https://go-review.googlesource.com/c/go/+/202797.

Is that a known flaky one cc @bradfitz @bcmills

bradfitz · 2019-10-23T15:46:14Z

Sigh

bradfitz · 2019-10-23T16:05:42Z

This was just introduced in https://go-review.googlesource.com/c/go/+/202078 for #34978 (and backported to Go 1.13, sadly too early apparently).

Looking.

bradfitz · 2019-10-23T16:06:55Z

I can repro:

bradfitzlaptop http$ go test -v -run=TestDontCacheBrokenHTTP2Conn -race -count=50
=== RUN   TestDontCacheBrokenHTTP2Conn
--- PASS: TestDontCacheBrokenHTTP2Conn (0.04s)
=== RUN   TestDontCacheBrokenHTTP2Conn
--- PASS: TestDontCacheBrokenHTTP2Conn (0.04s)
=== RUN   TestDontCacheBrokenHTTP2Conn
--- PASS: TestDontCacheBrokenHTTP2Conn (0.05s)
=== RUN   TestDontCacheBrokenHTTP2Conn
--- PASS: TestDontCacheBrokenHTTP2Conn (0.04s)
=== RUN   TestDontCacheBrokenHTTP2Conn
    TestDontCacheBrokenHTTP2Conn: transport_test.go:5885: for iteration 5, doBreak=false; unexpected error Get "https://127.0.0.1:58689": some write error
    TestDontCacheBrokenHTTP2Conn: transport_test.go:5889: GotConn calls = 0; want 1
--- FAIL: TestDontCacheBrokenHTTP2Conn (0.03s)
panic: Log in goroutine after TestDontCacheBrokenHTTP2Conn has completed

goroutine 205 [running]:
testing.(*common).logDepth(0xc000192100, 0xc00002a460, 0x4c, 0x3)
	/Users/bradfitz/go/src/testing/testing.go:679 +0x67c
testing.(*common).log(...)
	/Users/bradfitz/go/src/testing/testing.go:661
testing.(*common).Errorf(0xc000192100, 0x1822436, 0x19, 0xc0001a5678, 0x1, 0x1)
	/Users/bradfitz/go/src/testing/testing.go:710 +0x90
net/http_test.TestDontCacheBrokenHTTP2Conn.func2(0x1813c27, 0x3, 0xc00047b1d0, 0xf, 0xc000238360, 0x1041a70, 0x1d25470, 0x1d25470)
	/Users/bradfitz/go/src/net/http/transport_test.go:5849 +0x123
net/http.(*Transport).dial(0xc000384000, 0x1917980, 0xc00046b860, 0x1813c27, 0x3, 0xc00047b1d0, 0xf, 0x0, 0x205, 0x0, ...)
	/Users/bradfitz/go/src/net/http/transport.go:1035 +0xf8
net/http.(*Transport).dialConn(0xc000384000, 0x1917980, 0xc00046b860, 0x0, 0xc0003b8500, 0x5, 0xc00047b1d0, 0xf, 0x0, 0x0, ...)
	/Users/bradfitz/go/src/net/http/transport.go:1457 +0x2b1c
net/http.(*Transport).dialConnFor(0xc000384000, 0xc0003746e0)
	/Users/bradfitz/go/src/net/http/transport.go:1300 +0x150
created by net/http.(*Transport).queueForDial
	/Users/bradfitz/go/src/net/http/transport.go:1269 +0x68b
exit status 2
FAIL	net/http	0.386s

bcmills · 2019-10-23T16:11:36Z

and backported to Go 1.13, sadly too early apparently

Note that #35087 never actually got the CherryPickApproved label. I was kind of intentionally holding off on pinging it to give the CL a bit of soak time, but I guess I should have been more explicit about that.

bradfitz · 2019-10-23T16:20:41Z

Soaking before cherry-picking is new to me. That's a thing we do now? I mean, not a terrible idea, but first I'd heard.

bcmills · 2019-10-23T16:22:10Z

It's a thing I've been doing informally, at least. (#31887) Not a formal policy, as far as I'm aware.

bcmills · 2019-10-23T16:23:23Z

At any rate: for the race condition!

Would it make sense to add synchronization between the Dial hook and the loop, so that we know that all of the Conns returned by Dial have been Closed before we move to the next iteration? I suspect that the problem here is the aliasing on brokenState.

cuonglm · 2019-10-24T17:37:58Z

@bradfitz not sure it's related, but if I apply this patch, the test won't fail even with -count=500:

diff --git a/src/net/http/transport_test.go b/src/net/http/transport_test.go
index f76530b8fa..b60c164630 100644
--- a/src/net/http/transport_test.go
+++ b/src/net/http/transport_test.go
@@ -5871,9 +5871,7 @@ func TestDontCacheBrokenHTTP2Conn(t *testing.T) {
                        TLSHandshakeDone: func(cfg tls.ConnectionState, err error) {
                                brokenState.Lock()
                                defer brokenState.Unlock()
-                               if doBreak {
-                                       brokenState.broken = true
-                               }
+                               brokenState.broken = doBreak
                        },
                })
                req, err := NewRequestWithContext(ctx, "GET", cst.ts.URL, nil)

cuonglm · 2019-10-25T03:04:21Z

@bradfitz not sure it's related, but if I apply this patch, the test won't fail even with -count=500:

diff --git a/src/net/http/transport_test.go b/src/net/http/transport_test.go
index f76530b8fa..b60c164630 100644
--- a/src/net/http/transport_test.go
+++ b/src/net/http/transport_test.go
@@ -5871,9 +5871,7 @@ func TestDontCacheBrokenHTTP2Conn(t *testing.T) {
                        TLSHandshakeDone: func(cfg tls.ConnectionState, err error) {
                                brokenState.Lock()
                                defer brokenState.Unlock()
-                               if doBreak {
-                                       brokenState.broken = true
-                               }
+                               brokenState.broken = doBreak
                        },
                })
                req, err := NewRequestWithContext(ctx, "GET", cst.ts.URL, nil)

The test still fails at -count=1000

cuonglm · 2019-10-26T11:58:12Z

Another one https://storage.googleapis.com/go-build-log/810d776d/android-amd64-emu_97fb69fa.log

bcmills · 2019-10-31T15:54:33Z

linux-386-clang: https://build.golang.org/log/9c05e68c70e7bf8388daf62daf8e777c4ccec143

--- FAIL: TestDontCacheBrokenHTTP2Conn (0.02s)
    transport_test.go:5889: for iteration 5, doBreak=false; unexpected error Get "https://127.0.0.1:45761": some write error
    transport_test.go:5893: GotConn calls = 0; want 1
    transport_test.go:5853: unexpected Dial error: dial tcp 127.0.0.1:45761: connect: connection reset by peer

bcmills · 2019-11-09T01:56:50Z

Looks like it doesn't really vary by platform, but does tend to affect the slower configures more frequently.

2019-11-12T05:06:42-b8cb75f/linux-arm
2019-11-12T01:09:06-5b17b65/darwin-amd64-10_12
2019-11-11T20:07:54-e9f8d67/windows-amd64-race
2019-11-08T22:44:29-42db1da/illumos-amd64
2019-11-07T19:11:26-39e8cb0/freebsd-amd64-race
2019-11-07T14:53:38-e8f01d5/linux-arm
2019-11-06T17:02:19-4cb9260/darwin-amd64-race
2019-11-04T18:51:13-bf7e55b/windows-amd64-race
2019-11-02T21:51:12-2d031dc/darwin-amd64-10_11
2019-10-30T14:15:21-301bc66/freebsd-amd64-race
2019-10-30T08:17:29-f4e32ae/darwin-amd64-race
2019-10-29T22:56:47-15ea61c/linux-386-clang
2019-10-29T19:58:24-cc47b0d/windows-amd64-race

gopherbot · 2019-11-12T18:39:24Z

Change https://golang.org/cl/206819 mentions this issue: net/http: add some debugging to TestDontCacheBrokenHTTP2Conn


          net/http: add some debugging to TestDontCacheBrokenHTTP2Conn

Not a fix, but will give us more info when it flakes again. Updates #35113 Change-Id: I2f90c24530c1bea81dd9d8c7a59f4b0640dfa4c2 Reviewed-on: https://go-review.googlesource.com/c/go/+/206819 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>

bcmills · 2020-01-22T14:34:42Z

@bradfitz, any thoughts on how we should address this?

bradfitz · 2020-01-22T21:20:06Z

No immediate thoughts. I need to get it all in my head again and debug but I've been busy with other stuff. I'll try work on it this week.

gopherbot · 2020-02-06T08:35:39Z

Change https://golang.org/cl/218097 mentions this issue: net/http: fix TestDontCacheBrokenHTTP2Conn flaky

bcmills · 2020-02-20T13:51:17Z

cuonglm · 2020-02-21T06:49:42Z

@bcmills is there anyone I can add to reviewers list?

@bradfitz seems to be busy at this time.

bcmills · 2020-02-21T14:12:38Z

I'm looking for another reviewer. If I can't find one I'll dig into the code a bit more and see if I can get comfortable enough to +2 it myself.

rsc · 2020-02-25T17:00:41Z

I'm looking at this and the earlier CL.

gopherbot · 2020-02-25T18:10:44Z

Change https://golang.org/cl/220905 mentions this issue: net/http: fix handling of HTTP/2 upgrade failures

bakul · 2020-02-26T22:59:54Z

FWIW, I see the same test failing on Mojave (as part of running all.bash on 1.4).

    --- FAIL: TestDontCacheBrokenHTTP2Conn (0.01s)
        transport_test.go:6079: for iteration 5, doBreak=false; unexpected
    error Get "https://127.0.0.1:63138": some write error
        transport_test.go:6083: GotConn calls = 0; want 1
    panic: Fail in goroutine after TestDontCacheBrokenHTTP2Conn has completed

    goroutine 3645 [running]:
    testing.(*common).Fail(0xc0003d9200)
            /usr/local/go/src/testing/testing.go:613 +0x11a
    testing.(*common).Errorf(0xc0003d9200, 0x1585951, 0x19, 0xc0002b5920, 0x1,
    0x1)
            /usr/local/go/src/testing/testing.go:712 +0x90
    net/http_test.TestDontCacheBrokenHTTP2Conn.func2(0x157700a, 0x3,
    0xc0005bacb0, 0xf, 0xc0004519a8, 0xc00003b000, 0xc0004519a8, 0x10f65fe)
            /usr/local/go/src/net/http/transport_test.go:6044 +0x10a
    net/http.(*Transport).dial(0xc0004a32c0, 0x165aa60, 0xc000692990,
    0x157700a, 0x3, 0xc0005bacb0, 0xf, 0x0, 0x120, 0x0, ...)
            /usr/local/go/src/net/http/transport.go:1088 +0x7b
    net/http.(*Transport).dialConn(0xc0004a32c0, 0x165aa60, 0xc000692990, 0x0,
    0xc000283020, 0x5, 0xc0005bacb0, 0xf, 0x0, 0xc0002eb8c0, ...)
            /usr/local/go/src/net/http/transport.go:1519 +0x19ee
    net/http.(*Transport).dialConnFor(0xc0004a32c0, 0xc0007c0630)
            /usr/local/go/src/net/http/transport.go:1365 +0xc6
    created by net/http.(*Transport).queueForDial
            /usr/local/go/src/net/http/transport.go:1334 +0x3fe
    FAIL    net/http        0.732s

bradfitz self-assigned this Oct 23, 2019

bradfitz changed the title ~~TestDontCacheBrokenHTTP2Conn fails~~ net/http: new TestDontCacheBrokenHTTP2Conn fails Oct 23, 2019

bradfitz mentioned this issue Oct 23, 2019

net/http: transport caches permanently broken persistent connections if write error happens during h2 handshake #34978

Closed

bcmills closed this Nov 12, 2019

bcmills reopened this Nov 12, 2019

cuonglm mentioned this issue Feb 20, 2020

net/http: TestDontCacheBrokenHTTP2Conn flakes #37307

Closed

gopherbot closed this in ea1437a Mar 3, 2020

bradfitz mentioned this issue May 17, 2020

net/http, x/net/http2: client loses the request body when retrying after HTTP2 server close connection gracefully(GOAWAY) #39086

Closed

mdlayher mentioned this issue Jun 7, 2020

go: 1.14.3 -> 1.14.4 NixOS/nixpkgs#89334

Merged

5 of 10 tasks complete

golang / go

net/http: new TestDontCacheBrokenHTTP2Conn fails #35113

net/http: new TestDontCacheBrokenHTTP2Conn fails #35113

cuonglm commented Oct 23, 2019

cuonglm commented Oct 23, 2019

bradfitz commented Oct 23, 2019

bradfitz commented Oct 23, 2019

bradfitz commented Oct 23, 2019

bcmills commented Oct 23, 2019

bradfitz commented Oct 23, 2019

bcmills commented Oct 23, 2019

bcmills commented Oct 23, 2019

cuonglm commented Oct 24, 2019

cuonglm commented Oct 25, 2019

cuonglm commented Oct 26, 2019

bcmills commented Oct 31, 2019

bcmills commented Nov 9, 2019 •

edited

gopherbot commented Nov 12, 2019

bcmills commented Jan 22, 2020

bradfitz commented Jan 22, 2020

gopherbot commented Feb 6, 2020

bcmills commented Feb 20, 2020

cuonglm commented Feb 21, 2020

bcmills commented Feb 21, 2020

rsc commented Feb 25, 2020

gopherbot commented Feb 25, 2020

bakul commented Feb 26, 2020

golang / go

Join GitHub today

net/http: new TestDontCacheBrokenHTTP2Conn fails #35113

net/http: new TestDontCacheBrokenHTTP2Conn fails #35113

Comments

cuonglm commented Oct 23, 2019

cuonglm commented Oct 23, 2019

bradfitz commented Oct 23, 2019

bradfitz commented Oct 23, 2019

bradfitz commented Oct 23, 2019

bcmills commented Oct 23, 2019

bradfitz commented Oct 23, 2019

bcmills commented Oct 23, 2019

bcmills commented Oct 23, 2019

cuonglm commented Oct 24, 2019

cuonglm commented Oct 25, 2019

cuonglm commented Oct 26, 2019

bcmills commented Oct 31, 2019

bcmills commented Nov 9, 2019 • edited

gopherbot commented Nov 12, 2019

bcmills commented Jan 22, 2020

bradfitz commented Jan 22, 2020

gopherbot commented Feb 6, 2020

bcmills commented Feb 20, 2020

cuonglm commented Feb 21, 2020

bcmills commented Feb 21, 2020

rsc commented Feb 25, 2020

gopherbot commented Feb 25, 2020

bakul commented Feb 26, 2020

bcmills commented Nov 9, 2019 •

edited