Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

net/http: TestTransportMaxIdleConns failures on darwin and freebsd #56960

Open
gopherbot opened this issue Nov 28, 2022 · 13 comments
Open

net/http: TestTransportMaxIdleConns failures on darwin and freebsd #56960

gopherbot opened this issue Nov 28, 2022 · 13 comments
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Darwin OS-FreeBSD
Milestone
Backlog

Comments

@gopherbot
Copy link 

#!watchflakes
post <- pkg == "net/http" && test == "TestTransportMaxIdleConns"

Issue created automatically to collect these failures.

Example (log):

--- FAIL: TestTransportMaxIdleConns (0.00s)
    --- FAIL: TestTransportMaxIdleConns/h1 (0.18s)
        transport_test.go:4980: idle conn keys mismatch.
             got: ["|http|host-1.dns-is-faked.golang:29099" "|http|host-2.dns-is-faked.golang:29099" "|http|host-3.dns-is-faked.golang:29099"]
            want: ["|http|host-0.dns-is-faked.golang:29099" "|http|host-1.dns-is-faked.golang:29099" "|http|host-2.dns-is-faked.golang:29099" "|http|host-3.dns-is-faked.golang:29099"]

— watchflakes
@gopherbot gopherbot added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Nov 28, 2022
@gopherbot
Copy link
Author

Found new dashboard test flakes for:

#!watchflakes
post <- pkg == "net/http" && test == "TestTransportMaxIdleConns"
2022-11-23 18:23 freebsd-riscv64-unmatched go@4c0c0e5d net/http.TestTransportMaxIdleConns (log) 
--- FAIL: TestTransportMaxIdleConns (0.00s)
    --- FAIL: TestTransportMaxIdleConns/h1 (0.18s)
        transport_test.go:4980: idle conn keys mismatch.
             got: ["|http|host-1.dns-is-faked.golang:29099" "|http|host-2.dns-is-faked.golang:29099" "|http|host-3.dns-is-faked.golang:29099"]
            want: ["|http|host-0.dns-is-faked.golang:29099" "|http|host-1.dns-is-faked.golang:29099" "|http|host-2.dns-is-faked.golang:29099" "|http|host-3.dns-is-faked.golang:29099"]

— watchflakes

@bcmills
Copy link
Member

bcmills commented Nov 28, 2022

(CC @neild)

@prattmic
Copy link
Member

prattmic commented Dec 7, 2022

greplogs --dashboard -md -l -e 'FAIL: TestTransportMaxIdleConns'

2022-12-01T21:01:03-c71d3a0/freebsd-arm-paulzhol
2022-11-30T17:03:21-60525dc/freebsd-amd64-race
2022-11-29T22:59:16-7847260/darwin-amd64-13
2022-11-23T20:07:39-036696a/darwin-amd64-10_15-aws
2022-11-23T18:23:24-4c0c0e5/freebsd-riscv64-unmatched
2022-11-19T22:39:30-8205d83/darwin-amd64-aws-race
2022-11-18T08:04:52-205f636/freebsd-arm-paulzhol
2022-11-16T13:26:16-5947a07/darwin-amd64-aws-race
2022-11-08T14:54:25-6939659/darwin-amd64-12-aws
2022-11-04T22:09:36-d9cc39b/freebsd-arm-paulzhol
2022-10-26T05:51:33-939f9fd/freebsd-arm-paulzhol
2022-10-24T21:18:32-4319231/darwin-amd64-13-aws
2022-10-17T18:38:43-b65e259/freebsd-arm-paulzhol
2022-10-12T20:23:36-e0d01b8/darwin-amd64-11-aws
2022-10-11T12:40:42-1b316e3/freebsd-arm-paulzhol
2022-10-11T12:40:42-1b316e3/windows-amd64-newcc-race
2022-10-11T04:02:34-0ca3553/windows-amd64-race
2022-10-10T21:39:15-fce4496/freebsd-amd64-race

@bcmills
Copy link
Member

bcmills commented Dec 7, 2022 •
edited

@prattmic, that list is slightly overmatched. The oldest three -race failures were #56118, I think.

But it's close. The ones since Oct. 18 are mostly (or all?) this failure mode:

greplogs -l -e 'FAIL: TestTransportMaxIdleConns.*(\n\s+.*)*idle conn keys mismatch' --details

2022-12-01T21:01:03-c71d3a0/freebsd-arm-paulzhol
2022-11-30T17:03:21-60525dc/freebsd-amd64-race
2022-11-29T22:59:16-7847260/darwin-amd64-13
2022-11-23T20:07:39-036696a/darwin-amd64-10_15-aws
2022-11-23T18:23:24-4c0c0e5/freebsd-riscv64-unmatched
2022-11-19T22:39:30-8205d83/darwin-amd64-aws-race
2022-11-18T08:04:52-205f636/freebsd-arm-paulzhol
2022-11-16T23:17:58-6d0bf43/darwin-amd64-13
2022-11-16T13:26:16-5947a07/darwin-amd64-aws-race
2022-11-08T14:54:25-6939659/darwin-amd64-12-aws
2022-11-04T22:09:36-d9cc39b/freebsd-arm-paulzhol
2022-10-26T05:51:33-939f9fd/freebsd-arm-paulzhol
2022-10-24T21:18:32-4319231/darwin-amd64-13-aws
2022-10-17T18:38:43-b65e259/freebsd-arm-paulzhol
2022-10-12T20:23:36-e0d01b8/darwin-amd64-11-aws
2022-10-11T12:40:42-1b316e3/freebsd-arm-paulzhol

(16 matching logs)

@bcmills bcmills added this to the Backlog milestone Dec 8, 2022
@bcmills
Copy link
Member

bcmills commented Dec 8, 2022 •
edited

Intriguingly, all of those failures are on either freebsd or darwin, and parts of the Darwin platform (notably the network stack) are derived from FreeBSD.

Could be a (presumably very old) kernel or libc bug, or some other difference in the net stack where FreeBSD and macOS don't satisfy some assumption that happens to hold elsewhere.

(CC @golang/freebsd @golang/darwin)

@bcmills bcmills changed the title net/http: TestTransportMaxIdleConns failures net/http: TestTransportMaxIdleConns failures on darwin and freebsd Dec 8, 2022
@paulzhol
Copy link
Member

paulzhol commented Dec 9, 2022

Looking at both of these outputs:
2022-11-18T08:04:52-205f636/freebsd-arm-paulzhol
2022-11-04T22:09:36-d9cc39b/freebsd-arm-paulzhol

There seem to be another test running in parallel which opens several client TLS connections in the same port sequence range. Maybe it "hits" the non TLS server of TestTransportMaxIdleConns, causing it to be marked broken?
First log is missing "|http|host-0.dns-is-faked.golang:17796" .

2022/11/18 11:44:36 http: TLS handshake error from 127.0.0.1:17704: read tcp 127.0.0.1:17702->127.0.0.1:17704: read: connection reset by peer
2022/11/18 11:44:36 http: TLS handshake error from 127.0.0.1:17706: read tcp 127.0.0.1:17702->127.0.0.1:17706: read: connection reset by peer
...
2022/11/18 11:44:43 http: TLS handshake error from 127.0.0.1:17892: read tcp 127.0.0.1:17891->127.0.0.1:17892: read: connection reset by peer
2022/11/18 11:44:43 http: TLS handshake error from 127.0.0.1:17898: read tcp 127.0.0.1:17896->127.0.0.1:17898: read: connection reset by peer

The second log is missing "|http|host-4.dns-is-faked.golang:30467" for the evict case.

2022/11/05 13:12:35 http: TLS handshake error from 127.0.0.1:30346: write tcp 127.0.0.1:30337->127.0.0.1:30346: use of closed network connection
2022/11/05 13:12:35 http: TLS handshake error from 127.0.0.1:30345: write tcp 127.0.0.1:30337->127.0.0.1:30345: use of closed network connection
...
2022/11/05 13:13:18 http: TLS handshake error from 127.0.0.1:30766: EOF
2022/11/05 13:13:19 http: TLS handshake error from 127.0.0.1:30767: read tcp 127.0.0.1:30764->127.0.0.1:30767: read: connection reset by peer
2022/11/05 13:13:19 http: TLS handshake error from 127.0.0.1:30765: read tcp 127.0.0.1:30764->127.0.0.1:30765: read: connection reset by peer

@gopherbot
Copy link
Author

Found new dashboard test flakes for:

#!watchflakes
post <- pkg == "net/http" && test == "TestTransportMaxIdleConns"
2022-12-12 14:38 darwin-amd64-12_0 go@27301e82 net/http.TestTransportMaxIdleConns (log) 
--- FAIL: TestTransportMaxIdleConns (0.00s)
    --- FAIL: TestTransportMaxIdleConns/h1 (0.27s)
        transport_test.go:4992: idle conn keys mismatch after 5th host.
             got: ["|http|host-0.dns-is-faked.golang:51303" "|http|host-1.dns-is-faked.golang:51303" "|http|host-2.dns-is-faked.golang:51303" "|http|host-3.dns-is-faked.golang:51303"]
            want: ["|http|host-1.dns-is-faked.golang:51303" "|http|host-2.dns-is-faked.golang:51303" "|http|host-3.dns-is-faked.golang:51303" "|http|host-4.dns-is-faked.golang:51303"]

— watchflakes

@gopherbot
Copy link
Author

Found new dashboard test flakes for:

#!watchflakes
post <- pkg == "net/http" && test == "TestTransportMaxIdleConns"
2022-12-14 19:13 freebsd-arm-paulzhol go@5c682f94 net/http.TestTransportMaxIdleConns (log) 
--- FAIL: TestTransportMaxIdleConns (0.00s)
    --- FAIL: TestTransportMaxIdleConns/h1 (0.21s)
        transport_test.go:4980: idle conn keys mismatch.
             got: ["|http|host-0.dns-is-faked.golang:25512" "|http|host-2.dns-is-faked.golang:25512" "|http|host-3.dns-is-faked.golang:25512"]
            want: ["|http|host-0.dns-is-faked.golang:25512" "|http|host-1.dns-is-faked.golang:25512" "|http|host-2.dns-is-faked.golang:25512" "|http|host-3.dns-is-faked.golang:25512"]

— watchflakes

@gopherbot
Copy link
Author

Found new dashboard test flakes for:

#!watchflakes
post <- pkg == "net/http" && test == "TestTransportMaxIdleConns"
2022-12-22 04:34 freebsd-arm-paulzhol go@18baca67 net/http.TestTransportMaxIdleConns (log) 
--- FAIL: TestTransportMaxIdleConns (0.00s)
    --- FAIL: TestTransportMaxIdleConns/h1 (0.26s)
        transport_test.go:4980: idle conn keys mismatch.
             got: ["|http|host-1.dns-is-faked.golang:50742" "|http|host-2.dns-is-faked.golang:50742" "|http|host-3.dns-is-faked.golang:50742"]
            want: ["|http|host-0.dns-is-faked.golang:50742" "|http|host-1.dns-is-faked.golang:50742" "|http|host-2.dns-is-faked.golang:50742" "|http|host-3.dns-is-faked.golang:50742"]

— watchflakes

@gopherbot
Copy link
Author

Found new dashboard test flakes for:

#!watchflakes
post <- pkg == "net/http" && test == "TestTransportMaxIdleConns"
2022-11-30 17:03 freebsd-amd64-race go@60525dc3 net/http.TestTransportMaxIdleConns (log) 
--- FAIL: TestTransportMaxIdleConns (0.00s)
    --- FAIL: TestTransportMaxIdleConns/h1 (0.19s)
        transport_test.go:4980: idle conn keys mismatch.
             got: ["|http|host-0.dns-is-faked.golang:20502" "|http|host-1.dns-is-faked.golang:20502" "|http|host-2.dns-is-faked.golang:20502"]
            want: ["|http|host-0.dns-is-faked.golang:20502" "|http|host-1.dns-is-faked.golang:20502" "|http|host-2.dns-is-faked.golang:20502" "|http|host-3.dns-is-faked.golang:20502"]
2022-12-01 21:01 freebsd-arm-paulzhol go@c71d3a0f net/http.TestTransportMaxIdleConns (log) 
--- FAIL: TestTransportMaxIdleConns (0.00s)
    --- FAIL: TestTransportMaxIdleConns/h1 (0.30s)
        transport_test.go:4980: idle conn keys mismatch.
             got: ["|http|host-0.dns-is-faked.golang:43360" "|http|host-1.dns-is-faked.golang:43360" "|http|host-2.dns-is-faked.golang:43360"]
            want: ["|http|host-0.dns-is-faked.golang:43360" "|http|host-1.dns-is-faked.golang:43360" "|http|host-2.dns-is-faked.golang:43360" "|http|host-3.dns-is-faked.golang:43360"]

— watchflakes

@paulzhol paulzhol mentioned this issue Jan 8, 2023
Open
@gopherbot
Copy link
Author

Found new dashboard test flakes for:

#!watchflakes
post <- pkg == "net/http" && test == "TestTransportMaxIdleConns"
2023-01-06 18:22 darwin-amd64-nocgo go@f721fa3b net/http.TestTransportMaxIdleConns (log) 
--- FAIL: TestTransportMaxIdleConns (0.00s)
    --- FAIL: TestTransportMaxIdleConns/h1 (0.11s)
        transport_test.go:4992: idle conn keys mismatch after 5th host.
             got: ["|http|host-0.dns-is-faked.golang:50170" "|http|host-1.dns-is-faked.golang:50170" "|http|host-2.dns-is-faked.golang:50170" "|http|host-3.dns-is-faked.golang:50170"]
            want: ["|http|host-1.dns-is-faked.golang:50170" "|http|host-2.dns-is-faked.golang:50170" "|http|host-3.dns-is-faked.golang:50170" "|http|host-4.dns-is-faked.golang:50170"]

— watchflakes

@gopherbot
Copy link
Author

Found new dashboard test flakes for:

#!watchflakes
post <- pkg == "net/http" && test == "TestTransportMaxIdleConns"
2023-01-17 16:22 darwin-amd64-11_0 go@f375b305 net/http.TestTransportMaxIdleConns (log) 
--- FAIL: TestTransportMaxIdleConns (0.00s)
    --- FAIL: TestTransportMaxIdleConns/h1 (0.26s)
        transport_test.go:4980: idle conn keys mismatch.
             got: ["|http|host-0.dns-is-faked.golang:50893" "|http|host-1.dns-is-faked.golang:50893" "|http|host-3.dns-is-faked.golang:50893"]
            want: ["|http|host-0.dns-is-faked.golang:50893" "|http|host-1.dns-is-faked.golang:50893" "|http|host-2.dns-is-faked.golang:50893" "|http|host-3.dns-is-faked.golang:50893"]
2023-01-17 19:54 darwin-amd64-12_0 go@5cd805c6 net/http.TestTransportMaxIdleConns (log) 
--- FAIL: TestTransportMaxIdleConns (0.00s)
    --- FAIL: TestTransportMaxIdleConns/h1 (0.12s)
        transport_test.go:4980: idle conn keys mismatch.
             got: ["|http|host-0.dns-is-faked.golang:51242" "|http|host-1.dns-is-faked.golang:51242" "|http|host-2.dns-is-faked.golang:51242"]
            want: ["|http|host-0.dns-is-faked.golang:51242" "|http|host-1.dns-is-faked.golang:51242" "|http|host-2.dns-is-faked.golang:51242" "|http|host-3.dns-is-faked.golang:51242"]
2023-01-24 19:18 darwin-amd64-12_0 go@27500d88 net/http.TestTransportMaxIdleConns (log) 
--- FAIL: TestTransportMaxIdleConns (0.00s)
    --- FAIL: TestTransportMaxIdleConns/h1 (0.13s)
        transport_test.go:5031: idle conn keys mismatch after 5th host.
             got: ["|http|host-0.dns-is-faked.golang:56128" "|http|host-1.dns-is-faked.golang:56128" "|http|host-2.dns-is-faked.golang:56128" "|http|host-3.dns-is-faked.golang:56128"]
            want: ["|http|host-1.dns-is-faked.golang:56128" "|http|host-2.dns-is-faked.golang:56128" "|http|host-3.dns-is-faked.golang:56128" "|http|host-4.dns-is-faked.golang:56128"]

— watchflakes

@gopherbot
Copy link
Author

Found new dashboard test flakes for:

#!watchflakes
post <- pkg == "net/http" && test == "TestTransportMaxIdleConns"
2022-12-07 22:00 darwin-amd64-10_15 go@9f023421 net/http.TestTransportMaxIdleConns (log) 
--- FAIL: TestTransportMaxIdleConns (0.00s)
    --- FAIL: TestTransportMaxIdleConns/h1 (0.13s)
        transport_test.go:4980: idle conn keys mismatch.
             got: ["|http|host-1.dns-is-faked.golang:50370" "|http|host-2.dns-is-faked.golang:50370" "|http|host-3.dns-is-faked.golang:50370"]
            want: ["|http|host-0.dns-is-faked.golang:50370" "|http|host-1.dns-is-faked.golang:50370" "|http|host-2.dns-is-faked.golang:50370" "|http|host-3.dns-is-faked.golang:50370"]
2023-01-31 15:28 darwin-amd64-nocgo go@f298b90b net/http.TestTransportMaxIdleConns (log) 
--- FAIL: TestTransportMaxIdleConns (0.00s)
    --- FAIL: TestTransportMaxIdleConns/h1 (0.13s)
        transport_test.go:5019: idle conn keys mismatch.
             got: ["|http|host-0.dns-is-faked.golang:51061" "|http|host-1.dns-is-faked.golang:51061" "|http|host-2.dns-is-faked.golang:51061"]
            want: ["|http|host-0.dns-is-faked.golang:51061" "|http|host-1.dns-is-faked.golang:51061" "|http|host-2.dns-is-faked.golang:51061" "|http|host-3.dns-is-faked.golang:51061"]
2023-01-31 16:53 darwin-amd64-13 go@47e205c3 net/http.TestTransportMaxIdleConns (log) 
--- FAIL: TestTransportMaxIdleConns (0.00s)
    --- FAIL: TestTransportMaxIdleConns/h1 (0.17s)
        transport_test.go:5019: idle conn keys mismatch.
             got: ["|http|host-1.dns-is-faked.golang:50934" "|http|host-2.dns-is-faked.golang:50934" "|http|host-3.dns-is-faked.golang:50934"]
            want: ["|http|host-0.dns-is-faked.golang:50934" "|http|host-1.dns-is-faked.golang:50934" "|http|host-2.dns-is-faked.golang:50934" "|http|host-3.dns-is-faked.golang:50934"]
2023-02-01 21:30 darwin-amd64-race go@4b7f7eef net/http.TestTransportMaxIdleConns (log) 
--- FAIL: TestTransportMaxIdleConns (0.01s)
2023-02-02 14:52 darwin-amd64-12_0 go@d871f63b net/http.TestTransportMaxIdleConns (log) 
--- FAIL: TestTransportMaxIdleConns (0.00s)
    --- FAIL: TestTransportMaxIdleConns/h1 (0.28s)
        transport_test.go:5019: idle conn keys mismatch.
             got: ["|http|host-0.dns-is-faked.golang:50611" "|http|host-2.dns-is-faked.golang:50611" "|http|host-3.dns-is-faked.golang:50611"]
            want: ["|http|host-0.dns-is-faked.golang:50611" "|http|host-1.dns-is-faked.golang:50611" "|http|host-2.dns-is-faked.golang:50611" "|http|host-3.dns-is-faked.golang:50611"]

— watchflakes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Darwin OS-FreeBSD
Projects
Test Flakes
Status: No status
Milestone
Backlog
Development

No branches or pull requests
4 participants
@prattmic @paulzhol @bcmills @gopherbot