net/http: TestRetryRequestsOnError is flaky #30938

bradfitz · 2019-03-19T21:39:09Z

From https://storage.googleapis.com/go-build-log/c7862f5a/linux-amd64-race_4ed6f3c3.log

--- FAIL: TestRetryRequestsOnError (0.02s)
    --- FAIL: TestRetryRequestsOnError/NothingWrittenGetBody (0.00s)
        transport_test.go:3116: i=1: Do = Post http://fake.golang: read tcp 127.0.0.1:52650->127.0.0.1:42017: use of closed network connection; log:
            Dial
            Write("POST / HTTP/1.1\r\nHost: fake.golang\r\nUser-Agent: Go-http-client/1.1\r\nContent-Length: 4\r\nAccept-Encoding: gzip\r\n\r\nfoo\n")
            Handler
            intentional write failure
2019/03/19 21:17:09 http: TLS handshake error from 127.0.0.1:38418: write tcp 127.0.0.1:43483->127.0.0.1:38418: use of closed network connection

The text was updated successfully, but these errors were encountered:

cuonglm · 2019-03-20T04:18:14Z

@bradfitz is there a way to reliably reproduce this?

I tried:

for _ in {1..100}; do go-tip test -count=1 -race .;done

with no error.

bradfitz · 2019-03-25T15:17:02Z

I haven't tried.

bradfitz · 2019-04-29T17:01:07Z

Another:
https://storage.googleapis.com/go-build-log/8b56461e/linux-amd64-race_ed25b175.log

bcmills · 2019-05-16T17:46:46Z

https://storage.googleapis.com/go-build-log/de0346c9/linux-amd64-race_86344abc.log

--- FAIL: TestRetryRequestsOnError (0.02s)
    --- FAIL: TestRetryRequestsOnError/NothingWrittenNoBody (0.00s)
        transport_test.go:3217: i=1: Do = Delete http://fake.golang: read tcp 127.0.0.1:56096->127.0.0.1:40507: use of closed network connection; log:
            Dial
            Write("DELETE / HTTP/1.1\r\nHost: fake.golang\r\nUser-Agent: Go-http-client/1.1\r\nAccept-Encoding: gzip\r\n\r\n")
            Handler
            intentional write failure
2019/05/16 17:31:29 http: TLS handshake error from 127.0.0.1:49704: read tcp 127.0.0.1:33311->127.0.0.1:49704: use of closed network connection
2019/05/16 17:31:29 http: TLS handshake error from 127.0.0.1:49712: write tcp 127.0.0.1:33311->127.0.0.1:49712: use of closed network connection
2019/05/16 17:31:29 http: TLS handshake error from 127.0.0.1:49710: read tcp 127.0.0.1:33311->127.0.0.1:49710: use of closed network connection
2019/05/16 17:31:29 http: TLS handshake error from 127.0.0.1:49716: read tcp 127.0.0.1:33311->127.0.0.1:49716: use of closed network connection
2019/05/16 17:31:29 http: TLS handshake error from 127.0.0.1:49706: read tcp 127.0.0.1:33311->127.0.0.1:49706: use of closed network connection
2019/05/16 17:31:29 http: TLS handshake error from 127.0.0.1:49708: write tcp 127.0.0.1:33311->127.0.0.1:49708: use of closed network connection
2019/05/16 17:31:29 http: TLS handshake error from 127.0.0.1:49714: write tcp 127.0.0.1:33311->127.0.0.1:49714: use of closed network connection
2019/05/16 17:31:29 http: TLS handshake error from 127.0.0.1:49720: read tcp 127.0.0.1:33311->127.0.0.1:49720: use of closed network connection
2019/05/16 17:31:29 http: TLS handshake error from 127.0.0.1:49734: write tcp 127.0.0.1:33311->127.0.0.1:49734: use of closed network connection
2019/05/16 17:31:29 http: TLS handshake error from 127.0.0.1:49722: read tcp 127.0.0.1:33311->127.0.0.1:49722: use of closed network connection
2019/05/16 17:31:29 http: TLS handshake error from 127.0.0.1:49718: read tcp 127.0.0.1:33311->127.0.0.1:49718: use of closed network connection
2019/05/16 17:31:29 http: TLS handshake error from 127.0.0.1:49726: read tcp 127.0.0.1:33311->127.0.0.1:49726: use of closed network connection
2019/05/16 17:31:29 http: TLS handshake error from 127.0.0.1:49728: read tcp 127.0.0.1:33311->127.0.0.1:49728: use of closed network connection
2019/05/16 17:31:29 http: TLS handshake error from 127.0.0.1:49724: read tcp 127.0.0.1:33311->127.0.0.1:49724: use of closed network connection
2019/05/16 17:31:29 http: TLS handshake error from 127.0.0.1:49732: read tcp 127.0.0.1:33311->127.0.0.1:49732: use of closed network connection
2019/05/16 17:31:29 http: TLS handshake error from 127.0.0.1:49730: read tcp 127.0.0.1:33311->127.0.0.1:49730: use of closed network connection
FAIL

cuonglm · 2019-05-28T17:07:49Z

Another https://storage.googleapis.com/go-build-log/ff7e60ef/android-amd64-emu_19080ded.log

bcmills · 2019-06-07T23:00:57Z

https://storage.googleapis.com/go-build-log/868e4d85/android-amd64-emu_10321f7f.log

bcmills · 2019-07-31T18:10:13Z

darwin-386-10_14: https://build.golang.org/log/4742b5e51eebf76feb5249e2650939eaebb16793

See previously #25366.

bcmills · 2019-08-20T22:11:47Z

android-amd64-emu (https://build.golang.org/log/92ec6ce5e426c6e197dc01557d726a680dc5134a):

--- FAIL: TestRetryRequestsOnError (0.03s)
    --- FAIL: TestRetryRequestsOnError/NothingWrittenNoBody (0.00s)
        transport_test.go:3224: i=1: Do = Delete http://fake.golang: read tcp 127.0.0.1:36280->127.0.0.1:41698: use of closed network connection; log:
            Dial
            Write("DELETE / HTTP/1.1\r\nHost: fake.golang\r\nUser-Agent: Go-http-client/1.1\r\nAccept-Encoding: gzip\r\n\r\n")
            Handler
            intentional write failure
FAIL
exitcode=1FAIL	net/http	9.085s

jtarchie · 2019-09-11T16:53:13Z

We may be experiencing this error, too.
We have a test that is testing retry logic when a connection (in <=go1.12) would receive a io.EOF when the server closed the client connection.
When we upgraded to go1.13, we no longer receive io.EOF on the closed connection. We receive tls.errClosed.
With retry logic, we were able to compare the type of io.EOF, but with tls.errClosed, we now have to inspect the error string (Error()). This doesn't feel right to do.
We don't have a reproducible simple test case at the moment, but hopefully I've provided enough information from our current test case.

Read for more info: golang/go#30938 (comment) Signed-off-by: Jeremy Alvis <jalvis@pivotal.io>

jtarchie · 2019-09-13T17:15:18Z

It appears that tls.errClosed may not be the only culprit.
We are experiencing also a bunch of write: broken pipe in CI.
🤔

jtarchie · 2019-09-13T23:09:25Z

We've reverted back to go1.12.

bcmills · 2019-09-16T12:47:00Z

linux-s390x-ibm: https://build.golang.org/log/7b2dcb45b72837f2cb08401d2083f7a857443d7b

(same failure mode as in #30938 (comment))

bcmills · 2019-10-04T15:37:57Z

windows-amd64-2016: https://build.golang.org/log/bef07597bea372590c2de133c9ad1afeddd7a42f

Same failure, mode, but no TLS handshake error (consistent with @jtarchie's observation).

jtarchie · 2019-10-04T16:40:40Z

I might be able to figure out a reproducible (flakey) test, but I don't know if that actually hunts down the issue.
I'm not sure what changed in the go 1.12 to 1.13 source for TLS. I've looked at the diffs via git.
The fact that the error types changed (io.EOF to tls.errClosed | write: broken pipe) is probably where to start.

This could also be due to changes in httptest TLS Server.

bcmills · 2019-10-21T17:53:12Z

dragonfly-amd64: https://build.golang.org/log/782c3e02489de8e3203dcd50f037538956bab465
...at least we know it's not platform-dependent?

Actually, there is a pattern here: the only two cases that seem to flake are NothingWrittenNoBody and NothingWrittenGetBody. The Idempotent cases have not yet been implicated.

bcmills · 2019-10-21T17:57:08Z

This note in the test may provide a clue:

go/src/net/http/transport_test.go

Lines 3263 to 3266 in 06ac262

    
           // NOTE: we resend a request only if: 
        
           //   - we reused a keep-alive connection 
        
           //   - we haven't yet received any header data 
        
           //   - either we wrote no bytes to the server, or the request is idempotent

I think the TLS handshake error is a red herring: it isn't necessarily even coming from the same test function.

bcmills · 2019-11-21T15:02:17Z

2019-11-19T22:42:59-ce7829f/linux-amd64-longtest
2019-11-14T01:57:00-3f21c23/darwin-arm-mg912baios
2019-10-28T22:10:06-59a6847/darwin-arm64-mn4m2zdaios
2019-10-04T13:47:53-8c74bfb/windows-amd64-2016
2019-09-17T16:25:15-38543c2/android-amd64-emu
2019-09-16T11:53:57-d9b8ffa/linux-s390x-ibm
2019-09-02T21:44:50-fb7b50c/freebsd-386-11_2
2019-08-20T19:26:11-53c088f/android-amd64-emu
2019-07-30T20:49:54-1ad64fa/darwin-386-10_14
2019-07-15T16:28:58-4b36588/android-amd64-emu
2019-04-18T21:49:02-c8aaec2/linux-amd64-race
2019-03-17T19:13:01-bedb6a1/netbsd-arm-bsiegert
2019-03-05T14:19:59-94cbfc2/android-386-emu
2019-03-01T18:26:37-820ad17/android-amd64-emu
2019-03-01T16:27:27-4e10ce4/android-386-emu

bradfitz · 2019-11-22T22:01:14Z

I can reproduce this if I burn a bunch of CPU (https://gist.github.com/bradfitz/233894f4aec11b221cb06abd8489d2e4) and run

$ go test  -run=TestRetryRequestsOnError/NothingWrittenGetBody -race -count=500
--- FAIL: TestRetryRequestsOnError (0.02s) 
    --- FAIL: TestRetryRequestsOnError/NothingWrittenGetBody (0.02s)
        transport_test.go:3395: i=1: Do = Post "http://fake.golang": read tcp 127.0.0.1:50552->127.0.0.1:46615: use of closed network connection; log:
            Dial 
            Write("POST / HTTP/1.1\r\nHost: fake.golang\r\nUser-Agent: Go-http-client/1.1\r\nContent-Length: 4\r\nAccept-Encoding: gzip\r\n\r\nfoo\n")
            Handler
            intentional write failure      
--- FAIL: TestRetryRequestsOnError (0.01s) 
    --- FAIL: TestRetryRequestsOnError/NothingWrittenGetBody (0.00s)
        transport_test.go:3395: i=1: Do = Post "http://fake.golang": read tcp 127.0.0.1:42122->127.0.0.1:38355: use of closed network connection; log:
            Dial
            Write("POST / HTTP/1.1\r\nHost: fake.golang\r\nUser-Agent: Go-http-client/1.1\r\nContent-Length: 4\r\nAccept-Encoding: gzip\r\n\r\nfoo\n")
            Handler
            intentional write failure
FAIL
exit status 1   
FAIL    net/http        55.811s

This check involving checking for a failure under 25 milliseconds scares me:

if time.Since(t0) < MaxWriteWaitBeforeConnReuse/2 {

... because all test code involving time scares me.

bcmills · 2020-12-02T15:21:27Z

bcmills · 2021-02-22T15:21:35Z

2021-02-19T16:09:17-02e5a8f/linux-amd64-race
2021-02-10T22:44:54-ed80790/linux-amd64-race

bcmills · 2021-03-05T16:06:45Z

https://storage.googleapis.com/go-build-log/4637c62e/android-amd64-emu_6918d69c.log (a TryBot)

bcmills · 2021-04-07T17:15:55Z

https://storage.googleapis.com/go-build-log/f67e340a/android-amd64-emu_cde79d93.log (a TryBot)

bcmills · 2021-05-07T13:38:15Z

2021-05-06T20:30:00-4dbad79/android-386-emu
2021-04-30T05:18:37-9c72078/freebsd-arm64-dmgk
2021-04-29T17:47:48-6d95e5a/android-386-emu
2021-04-20T20:58:52-dbade77/android-arm64-corellium
2021-04-09T17:49:01-2698be4/windows-amd64-2016
2021-04-06T20:28:21-b56177a/linux-amd64-race
2021-03-30T03:28:19-3394586/dragonfly-amd64-5_8
2021-03-10T04:06:13-4d608eb/android-amd64-emu
2021-02-23T15:03:39-811167e/android-amd64-emu

bcmills · 2021-05-07T13:38:39Z

CC @neild @empijei

bcmills · 2021-08-30T18:51:52Z

$ greplogs --dashboard -md -l -e FAIL: TestRetryRequestsOnError --since=2021-05-07

2021-08-26T20:39:56-af80af2/android-386-emu
2021-08-25T18:27:41-d2f002c/linux-amd64-jessie
2021-08-24T15:55:38-d70c69d/android-386-emu
2021-08-20T18:45:05-e49775e/android-amd64-emu
2021-08-16T08:37:46-fcdc3c0/android-386-emu
2021-08-12T21:33:09-0d01934/android-386-emu
2021-06-21T17:37:39-20bdfba/android-amd64-emu
2021-06-08T22:35:56-bcecae2/android-386-emu
2021-06-04T01:31:23-962d5c9/dragonfly-amd64

bradfitz added Testing help wanted NeedsFix labels Mar 19, 2019

bradfitz added this to the Go1.13 milestone Mar 19, 2019

bradfitz self-assigned this Mar 19, 2019

dmitshur added NeedsInvestigation and removed NeedsFix labels Mar 21, 2019

andybons removed this from the Go1.13 milestone Jul 8, 2019

andybons added this to the Go1.14 milestone Jul 8, 2019

jtarchie pushed a commit to pivotal-cf/om that referenced this issue Sep 11, 2019

fix flakey test by handling go1.13 err

ebba04d

Read for more info: golang/go#30938 (comment) Signed-off-by: Jeremy Alvis <jalvis@pivotal.io>

rsc removed this from the Go1.14 milestone Oct 9, 2019

rsc added this to the Backlog milestone Oct 9, 2019

bcmills changed the title ~~net/http: TestRetryRequestsOnError flake~~ net/http: TestRetryRequestsOnError is flaky Nov 21, 2019

golang / go Public

net/http: TestRetryRequestsOnError is flaky #30938

net/http: TestRetryRequestsOnError is flaky #30938

bradfitz commented Mar 19, 2019 •

edited

cuonglm commented Mar 20, 2019

bradfitz commented Mar 25, 2019

bradfitz commented Apr 29, 2019

bcmills commented May 16, 2019

cuonglm commented May 28, 2019

bcmills commented Jun 7, 2019

bcmills commented Jul 31, 2019

bcmills commented Aug 20, 2019

jtarchie commented Sep 11, 2019

jtarchie commented Sep 13, 2019

jtarchie commented Sep 13, 2019

bcmills commented Sep 16, 2019

bcmills commented Oct 4, 2019

jtarchie commented Oct 4, 2019 •

edited

bcmills commented Oct 21, 2019

bcmills commented Oct 21, 2019

bcmills commented Nov 21, 2019

bradfitz commented Nov 22, 2019

bcmills commented Dec 2, 2020

bcmills commented Feb 22, 2021

bcmills commented Mar 5, 2021

bcmills commented Apr 7, 2021

bcmills commented May 7, 2021

bcmills commented May 7, 2021

bcmills commented Aug 30, 2021

golang / go Public

net/http: TestRetryRequestsOnError is flaky #30938

net/http: TestRetryRequestsOnError is flaky #30938

Comments

bradfitz commented Mar 19, 2019 • edited

cuonglm commented Mar 20, 2019

bradfitz commented Mar 25, 2019

bradfitz commented Apr 29, 2019

bcmills commented May 16, 2019

cuonglm commented May 28, 2019

bcmills commented Jun 7, 2019

bcmills commented Jul 31, 2019

bcmills commented Aug 20, 2019

jtarchie commented Sep 11, 2019

jtarchie commented Sep 13, 2019

jtarchie commented Sep 13, 2019

bcmills commented Sep 16, 2019

bcmills commented Oct 4, 2019

jtarchie commented Oct 4, 2019 • edited

bcmills commented Oct 21, 2019

bcmills commented Oct 21, 2019

bcmills commented Nov 21, 2019

bradfitz commented Nov 22, 2019

bcmills commented Dec 2, 2020

bcmills commented Feb 22, 2021

bcmills commented Mar 5, 2021

bcmills commented Apr 7, 2021

bcmills commented May 7, 2021

bcmills commented May 7, 2021

bcmills commented Aug 30, 2021

bradfitz commented Mar 19, 2019 •

edited

jtarchie commented Oct 4, 2019 •

edited