net: TestLookupDotsWithRemoteSource is flaky #27992

katiehockman · 2018-10-03T20:37:11Z

#!watchflakes
post <- pkg == "net" && test == "TestLookupDotsWithRemoteSource" && (`DNS server failure`||`no such host`||`server misbehaving`)

We are getting flaky test failures in net. This has been somewhat rare in the past few months, but it has failed the last 5 builds, so probably worth looking into now.

--- FAIL: TestLookupDotsWithRemoteSource (15.57s)
    lookup_test.go:694: LookupSRV(xmpp-server, tcp, google.com): lookup _xmpp-server._tcp.google.com on 8.8.8.8:53: read udp 10.50.0.170:50511->8.8.8.8:53: i/o timeout (mode=go)
FAIL
FAIL	net	30.123s

Looks very related to #16865 closed 2 years ago.

The text was updated successfully, but these errors were encountered:

katiehockman · 2018-10-03T20:39:09Z

/cc @mikioh @brad-burch @ianlancetaylor

ianlancetaylor · 2018-10-03T20:52:43Z

ianlancetaylor · 2018-10-03T20:57:36Z

Any chance there was a recent change to whether the builders set GO_BUILDER_FLAKY_NET in the environment? Because the test does not fail in that way if that environment variable is set.

mikioh · 2018-10-03T23:52:32Z

Once http://golang.org/cl/102875 lands, I will smash the use of external DNS speakers such as rsc.io or some public full resolver then flaky tests disappear. For now, I have no spare time to debug the circumstances running the tests, also not sure there's a way to sneak in the (protected) circumstances.

andybons · 2018-11-28T20:16:04Z

@mikioh should we just disable this test until 1.13?

mikioh · 2018-11-29T00:02:48Z

Yes, please. Will try to allocate my spare time for Go 1.13 devel cycle.

gopherbot · 2018-11-29T23:45:08Z

Change https://golang.org/cl/151799 mentions this issue: net: skip flaky TestLookupDotsWithRemoteSource for now

Updates #27992 Change-Id: Ic327df7cc5002a3d537f9117559c25f30e1eab9c Reviewed-on: https://go-review.googlesource.com/c/151799 Run-TryBot: Andrew Bonventre <andybons@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

andybons · 2018-11-30T22:25:34Z

@mikioh done. Assigning back to you. Thanks.

bcmills · 2019-05-15T13:44:01Z

Here's the same failure mode on dragonfly-amd64: https://build.golang.org/log/2f33b64e758b0fb89431e34f51f80580e5b56e59

--- FAIL: TestLookupDotsWithRemoteSource (10.85s)
    lookup_test.go:669: LookupCNAME(www.mit.edu, mode=go): lookup www.mit.edu on 192.168.1.100:53: read udp 192.168.1.2:54367->192.168.1.100:53: i/o timeout
FAIL
FAIL	net	68.224s

bcmills · 2019-08-20T18:12:53Z

freebsd-arm-paulzhol (https://build.golang.org/log/daa40802f66a0d79d4d356d551c94456f56785b1):

--- FAIL: TestLookupDotsWithRemoteSource (12.52s)
    lookup_test.go:656: LookupAddr(8.8.8.8): lookup 8.8.8.8.in-addr.arpa. on 192.168.1.253:53: read udp 192.168.1.101:38233->192.168.1.253:53: i/o timeout (mode=go)
FAIL
FAIL	net	44.919s

bcmills · 2020-01-27T14:00:34Z

2020-01-25T05:48:12-01df660/freebsd-arm-paulzhol
2019-11-22T16:04:32-1c5bd34/freebsd-arm-paulzhol
2019-10-26T05:40:14-c3e8a20/linux-s390x-ibm
2019-09-18T19:24:59-90f9426/freebsd-arm-paulzhol
2019-08-19T21:42:54-d6ffc1d/freebsd-arm-paulzhol
2019-07-29T16:52:01-5235501/freebsd-arm-paulzhol
2019-07-01T14:45:49-bc593ea/freebsd-arm-paulzhol
2019-06-19T18:19:01-b373d31/freebsd-arm-paulzhol
2019-05-14T16:36:38-a9e107c/dragonfly-amd64
2019-05-09T18:02:19-c514071/dragonfly-amd64
2019-05-02T04:44:36-74b3c50/dragonfly-amd64
2019-04-28T23:51:04-50ad094/dragonfly-amd64

bcmills · 2020-01-27T14:18:54Z

I did some digging, and I believe that this flakiness is caused by the hard-coded timeout and attempt limits in net/dnsconfig_unix.go:

go/src/net/dnsconfig_unix.go

Lines 41 to 45 in 825ff1e

    
           conf := &dnsConfig{ 
        
           	ndots:    1, 
        
           	timeout:  5 * time.Second, 
        
           	attempts: 2, 
        
           }

On my Linux workstation, man 5 resolv.conf reports these same defaults. However, given that the Go net.Resolver API accepts a context.Context which can provide its own timeout behavior, I think these hard-coded limits are out of place: they might make sense in the context of a legacy POSIX C API, but they don't really fit in a modern Go package.

I would recommend:

Complete this TODO to allow the Resolver to override the system-default timeout (and likely also the default limit on attempts).
Refactor the RemoteSource test to use a Resolver and explicit Context instead of the Context-less top-level LookupAddr function. Bound the timeout and number of retries to the test's deadline (testing: add (*T).Deadline #28135) instead of arbitrary constants.

bcmills · 2021-12-03T20:41:05Z

https://storage.googleapis.com/go-build-log/d0be7dc1/linux-riscv64-jsing_698793b2.log (a SlowBot on CL 369037)

alexbrainman · 2022-08-19T09:41:26Z

Another flake by standard try-bot

https://storage.googleapis.com/go-build-log/43f55574/windows-386-2008_d562679e.log

while testing

https://go-review.googlesource.com/c/go/+/416975

Alex

mknyszek · 2022-09-06T21:20:15Z

Another flake:

2022-09-02T06:22:23-ec2ea40/windows-386-2008

gopherbot · 2022-09-28T19:44:33Z

Found new dashboard test flakes for:

#!watchflakes
post <- pkg == "net" && test == "TestLookupDotsWithRemoteSource"

2022-08-02 18:51 windows-386-2008 go@e9d7f54a net.TestLookupDotsWithRemoteSource (log)

--- FAIL: TestLookupDotsWithRemoteSource (0.12s)
    lookup_test.go:663: LookupAddr(8.8.8.8): lookup 8.8.8.8: dnsquery: DNS server failure. (mode=go)

2022-08-11 20:03 windows-386-2012 go@72a857a2 net.TestLookupDotsWithRemoteSource (log)

--- FAIL: TestLookupDotsWithRemoteSource (0.03s)
    lookup_test.go:663: LookupAddr(8.8.8.8): lookup 8.8.8.8: dnsquery: DNS server failure. (mode=go)

2022-08-16 17:18 windows-amd64-2008 go@805305e2 net.TestLookupDotsWithRemoteSource (log)

--- FAIL: TestLookupDotsWithRemoteSource (0.08s)
    lookup_test.go:663: LookupAddr(8.8.8.8): lookup 8.8.8.8: dnsquery: DNS server failure. (mode=go)

2022-08-18 16:16 windows-amd64-2008 go@b6149224 net.TestLookupDotsWithRemoteSource (log)

--- FAIL: TestLookupDotsWithRemoteSource (0.02s)
    lookup_test.go:663: LookupAddr(8.8.8.8): lookup 8.8.8.8: dnsquery: DNS server failure. (mode=go)

2022-08-22 18:51 windows-amd64-2016 go@a726c9f6 net.TestLookupDotsWithRemoteSource (log)

--- FAIL: TestLookupDotsWithRemoteSource (0.02s)
    lookup_test.go:663: LookupAddr(8.8.8.8): lookup 8.8.8.8: dnsquery: DNS server failure. (mode=go)

2022-08-23 22:44 windows-amd64-2008 go@ab8a2c5e net.TestLookupDotsWithRemoteSource (log)

--- FAIL: TestLookupDotsWithRemoteSource (0.04s)
    lookup_test.go:663: LookupAddr(8.8.8.8): lookup 8.8.8.8: dnsquery: DNS server failure. (mode=go)

2022-08-31 20:05 windows-386-2008 go@9085ff58 net.TestLookupDotsWithRemoteSource (log)

--- FAIL: TestLookupDotsWithRemoteSource (0.04s)
    lookup_test.go:663: LookupAddr(8.8.8.8): lookup 8.8.8.8: dnsquery: DNS server failure. (mode=go)

2022-08-31 22:22 windows-amd64-race go@e4b624ea net.TestLookupDotsWithRemoteSource (log)

--- FAIL: TestLookupDotsWithRemoteSource (0.08s)
    lookup_test.go:663: LookupAddr(8.8.8.8): lookup 8.8.8.8: dnsquery: DNS server failure. (mode=go)

2022-09-02 02:09 windows-amd64-longtest go@a31c062c net.TestLookupDotsWithRemoteSource (log)

--- FAIL: TestLookupDotsWithRemoteSource (0.11s)
    lookup_test.go:663: LookupAddr(8.8.8.8): lookup 8.8.8.8: dnsquery: DNS server failure. (mode=go)

2022-09-08 14:57 windows-amd64-race go@530a2369 net.TestLookupDotsWithRemoteSource (log)

--- FAIL: TestLookupDotsWithRemoteSource (0.02s)
    lookup_test.go:662: LookupAddr(8.8.8.8): lookup 8.8.8.8: dnsquery: DNS server failure. (mode=go)

2022-09-08 21:18 windows-amd64-longtest go@e20106ac net.TestLookupDotsWithRemoteSource (log)

--- FAIL: TestLookupDotsWithRemoteSource (0.01s)
    lookup_test.go:662: LookupAddr(8.8.8.8): lookup 8.8.8.8: dnsquery: DNS server failure. (mode=go)

2022-09-21 01:56 windows-amd64-longtest go@d11c58ee net.TestLookupDotsWithRemoteSource (log)

--- FAIL: TestLookupDotsWithRemoteSource (0.05s)
    lookup_test.go:662: LookupAddr(8.8.8.8): lookup 8.8.8.8: dnsquery: DNS server failure. (mode=go)

— watchflakes

gopherbot · 2022-10-04T18:43:53Z

Found new dashboard test flakes for:

#!watchflakes
post <- pkg == "net" && test == "TestLookupDotsWithRemoteSource"

2022-09-29 14:51 windows-amd64-race go@9861e8b2 net.TestLookupDotsWithRemoteSource (log)

--- FAIL: TestLookupDotsWithRemoteSource (0.03s)
    lookup_test.go:662: LookupAddr(8.8.8.8): lookup 8.8.8.8: dnsquery: DNS server failure. (mode=go)

— watchflakes

gopherbot · 2022-11-04T17:59:38Z

Found new dashboard test flakes for:

#!watchflakes
post <- pkg == "net" && test == "TestLookupDotsWithRemoteSource"

2022-11-04 15:03 windows-386-2012 go@edfe0783 net.TestLookupDotsWithRemoteSource (log)

--- FAIL: TestLookupDotsWithRemoteSource (0.15s)
    lookup_test.go:663: LookupAddr(8.8.8.8): lookup 8.8.8.8: dnsquery: DNS server failure. (mode=go)

— watchflakes

gopherbot · 2022-11-17T16:06:58Z

Found new dashboard test flakes for:

#!watchflakes
post <- pkg == "net" && test == "TestLookupDotsWithRemoteSource"

2022-11-09 17:33 freebsd-arm64-dmgk go@7a92c4fc net.TestLookupDotsWithRemoteSource (log)

--- FAIL: TestLookupDotsWithRemoteSource (0.20s)
    lookup_test.go:708: LookupSRV(xmpp-server, tcp, google.com): lookup _xmpp-server._tcp.google.com on 10.0.0.2:53: no such host (mode=go)
    lookup_test.go:708: LookupSRV(xmpp-server, tcp, google.com): lookup _xmpp-server._tcp.google.com on 10.0.0.2:53: no such host (mode=cgo)

— watchflakes

gopherbot · 2023-01-04T21:16:43Z

Found new dashboard test flakes for:

#!watchflakes
post <- pkg == "net" && test == "TestLookupDotsWithRemoteSource"

2022-11-08 14:52 freebsd-arm64-dmgk go@a1c31d68 net.TestLookupDotsWithRemoteSource (log)

--- FAIL: TestLookupDotsWithRemoteSource (0.07s)
    lookup_test.go:706: LookupSRV(xmpp-server, tcp, google.com): lookup _xmpp-server._tcp.google.com on 10.0.0.2:53: no such host (mode=go)
    lookup_test.go:706: LookupSRV(xmpp-server, tcp, google.com): lookup _xmpp-server._tcp.google.com on 10.0.0.2:53: no such host (mode=cgo)

2022-11-09 17:39 aix-ppc64 go@cd8d1bca net.TestLookupDotsWithRemoteSource (log)

--- FAIL: TestLookupDotsWithRemoteSource (0.16s)
    lookup_test.go:708: LookupSRV(xmpp-server, tcp, google.com): lookup _xmpp-server._tcp.google.com on 140.211.166.130:53: no such host (mode=go)
    lookup_test.go:708: LookupSRV(xmpp-server, tcp, google.com): lookup _xmpp-server._tcp.google.com on 140.211.166.130:53: no such host (mode=cgo)

2022-11-09 17:39 freebsd-arm64-dmgk go@cd8d1bca net.TestLookupDotsWithRemoteSource (log)

--- FAIL: TestLookupDotsWithRemoteSource (0.06s)
    lookup_test.go:708: LookupSRV(xmpp-server, tcp, google.com): lookup _xmpp-server._tcp.google.com on 10.0.0.2:53: no such host (mode=go)
    lookup_test.go:708: LookupSRV(xmpp-server, tcp, google.com): lookup _xmpp-server._tcp.google.com on 10.0.0.2:53: no such host (mode=cgo)

2022-11-09 18:44 freebsd-arm64-dmgk go@9e0b6c11 net.TestLookupDotsWithRemoteSource (log)

--- FAIL: TestLookupDotsWithRemoteSource (0.12s)
    lookup_test.go:706: LookupSRV(xmpp-server, tcp, google.com): lookup _xmpp-server._tcp.google.com on 10.0.0.2:53: no such host (mode=go)
    lookup_test.go:706: LookupSRV(xmpp-server, tcp, google.com): lookup _xmpp-server._tcp.google.com on 10.0.0.2:53: no such host (mode=cgo)

2022-11-10 18:18 aix-ppc64 go@fd0c0db4 net.TestLookupDotsWithRemoteSource (log)

--- FAIL: TestLookupDotsWithRemoteSource (0.14s)
    lookup_test.go:708: LookupSRV(xmpp-server, tcp, google.com): lookup _xmpp-server._tcp.google.com on 140.211.166.130:53: no such host (mode=go)
    lookup_test.go:708: LookupSRV(xmpp-server, tcp, google.com): lookup _xmpp-server._tcp.google.com on 140.211.166.130:53: no such host (mode=cgo)

2022-11-10 18:18 freebsd-arm64-dmgk go@fd0c0db4 net.TestLookupDotsWithRemoteSource (log)

--- FAIL: TestLookupDotsWithRemoteSource (0.11s)
    lookup_test.go:708: LookupSRV(xmpp-server, tcp, google.com): lookup _xmpp-server._tcp.google.com on 10.0.0.2:53: no such host (mode=go)
    lookup_test.go:708: LookupSRV(xmpp-server, tcp, google.com): lookup _xmpp-server._tcp.google.com on 10.0.0.2:53: no such host (mode=cgo)

2022-11-10 18:29 aix-ppc64 go@d931b3b7 net.TestLookupDotsWithRemoteSource (log)

--- FAIL: TestLookupDotsWithRemoteSource (0.13s)
    lookup_test.go:708: LookupSRV(xmpp-server, tcp, google.com): lookup _xmpp-server._tcp.google.com on 140.211.166.130:53: no such host (mode=go)
    lookup_test.go:708: LookupSRV(xmpp-server, tcp, google.com): lookup _xmpp-server._tcp.google.com on 140.211.166.130:53: no such host (mode=cgo)

— watchflakes

gopherbot · 2023-01-25T16:37:04Z

Found new dashboard test flakes for:

#!watchflakes
post <- pkg == "net" && test == "TestLookupDotsWithRemoteSource"

2023-01-17 19:56 windows-amd64-race go@354c8fb4 net.TestLookupDotsWithRemoteSource (log)

--- FAIL: TestLookupDotsWithRemoteSource (0.05s)
    lookup_test.go:665: LookupAddr(8.8.8.8): lookup 8.8.8.8: dnsquery: DNS server failure. (mode=go)

— watchflakes

gopherbot · 2023-04-13T04:02:28Z

Found new dashboard test flakes for:

#!watchflakes
post <- pkg == "net" && test == "TestLookupDotsWithRemoteSource"

2023-04-12 20:54 linux-amd64-longtest-race go@f58c6ccc net.TestLookupDotsWithRemoteSource (log)

--- FAIL: TestLookupDotsWithRemoteSource (10.49s)
    lookup_test.go:666: LookupAddr(8.8.8.8): lookup 8.8.8.8.in-addr.arpa. on 169.254.169.254:53: server misbehaving (mode=go)
    lookup_test.go:666: LookupAddr(8.8.8.8): lookup 8.8.8.8.in-addr.arpa. on 169.254.169.254:53: server misbehaving (mode=cgo)
2023/04/12 21:27:45 killing splice client after 5 second shutdown timeout

— watchflakes

gopherbot · 2023-05-10T23:00:20Z

Found new dashboard test flakes for:

#!watchflakes
post <- pkg == "net" && test == "TestLookupDotsWithRemoteSource"

2023-05-10 22:43 freebsd-amd64-12_3 go@639957eb net.TestLookupDotsWithRemoteSource (log)

SIGSEGV: segmentation violation
PC=0x8009f9f1c m=0 sigcode=1
signal arrived during cgo execution

rax    0xc2c3
rbx    0x437e
rcx    0xc2c3
rdx    0xffffffffffffffff
rdi    0x0
rsi    0x0
...
	/tmp/workdir/go/src/net/lookup_unix.go:96 +0xa5 fp=0xc00032bd98 sp=0xc00032bd38 pc=0x561325
net.(*Resolver).LookupCNAME(0xc000392310?, {0x6dcc10?, 0x879980?}, {0x685212, 0xb})
	/tmp/workdir/go/src/net/lookup.go:472 +0x2b fp=0xc00032bde0 sp=0xc00032bd98 pc=0x55deeb
net.LookupCNAME(...)
	/tmp/workdir/go/src/net/lookup.go:455
net.testDots(0xc0001df380, {0x682ee0, 0x3})
	/tmp/workdir/go/src/net/lookup_test.go:676 +0x12e fp=0xc00032bef0 sp=0xc00032bde0 pc=0x5b918e
net.TestLookupDotsWithRemoteSource(0xc0001df380)
	/tmp/workdir/go/src/net/lookup_test.go:658 +0x157 fp=0xc00032bf70 sp=0xc00032bef0 pc=0x5b8fb7
testing.tRunner(0xc0001df380, 0x699c48)

— watchflakes

katiehockman added OS-Darwin Testing An issue that has been verified to require only test changes, not just a test failure. labels Oct 3, 2018

ianlancetaylor added the release-blocker label Oct 3, 2018

ianlancetaylor added this to the Go1.12 milestone Oct 3, 2018

katiehockman changed the title ~~net: Flaky test failing in darwin-amd64-10_10~~ net: flaky test failing in darwin-amd64-10_10 Oct 3, 2018

bradfitz mentioned this issue Oct 4, 2018

net: localhost pipes cause test hangs on macOS Sierra #25696

Closed

andybons self-assigned this Nov 29, 2018

andybons assigned mikioh and unassigned andybons Nov 30, 2018

andybons removed the release-blocker label Nov 30, 2018

andybons modified the milestones: Go1.12, Go1.13 Nov 30, 2018

mikioh removed their assignment Dec 4, 2018

bcmills changed the title ~~net: flaky test failing in darwin-amd64-10_10~~ net: TestLookupDotsWithRemoteSource is flaky May 15, 2019

bcmills added the NeedsFix The path to resolution is known, but the work has not been done. label May 15, 2019

andybons modified the milestones: Go1.13, Go1.14 Jul 8, 2019

bcmills removed the OS-Darwin label Aug 20, 2019

rsc removed this from the Go1.14 milestone Oct 9, 2019

rsc added this to the Backlog milestone Oct 9, 2019

dmitshur mentioned this issue Aug 5, 2022

build dashboard triage log #52653

Closed

bcmills mentioned this issue Sep 28, 2022

all: test failures due to windows dns errors #55165

Closed

bcmills mentioned this issue Nov 9, 2022

net: Resolver errors on Windows with code WSATRY_AGAIN #55050

Closed

bcmills mentioned this issue May 11, 2023

net: failures with signal arrived during cgo execution #60132

Open

This comment was marked as off-topic.

Sign in to view

net: TestLookupDotsWithRemoteSource is flaky #27992

net: TestLookupDotsWithRemoteSource is flaky #27992

katiehockman commented Oct 3, 2018 •

edited by bcmills

katiehockman commented Oct 3, 2018

ianlancetaylor commented Oct 3, 2018

ianlancetaylor commented Oct 3, 2018

mikioh commented Oct 3, 2018

andybons commented Nov 28, 2018

mikioh commented Nov 29, 2018

gopherbot commented Nov 29, 2018

andybons commented Nov 30, 2018

bcmills commented May 15, 2019

bcmills commented Aug 20, 2019

bcmills commented Jan 27, 2020 •

edited

bcmills commented Jan 27, 2020

bcmills commented Dec 3, 2021

alexbrainman commented Aug 19, 2022

mknyszek commented Sep 6, 2022

gopherbot commented Sep 28, 2022

gopherbot commented Oct 4, 2022

gopherbot commented Nov 4, 2022

gopherbot commented Nov 17, 2022

gopherbot commented Jan 4, 2023

gopherbot commented Jan 25, 2023

gopherbot commented Apr 13, 2023

gopherbot commented May 10, 2023

This comment was marked as off-topic.

net: TestLookupDotsWithRemoteSource is flaky #27992

net: TestLookupDotsWithRemoteSource is flaky #27992

Comments

katiehockman commented Oct 3, 2018 • edited by bcmills

katiehockman commented Oct 3, 2018

ianlancetaylor commented Oct 3, 2018

ianlancetaylor commented Oct 3, 2018

mikioh commented Oct 3, 2018

andybons commented Nov 28, 2018

mikioh commented Nov 29, 2018

gopherbot commented Nov 29, 2018

andybons commented Nov 30, 2018

bcmills commented May 15, 2019

bcmills commented Aug 20, 2019

bcmills commented Jan 27, 2020 • edited

bcmills commented Jan 27, 2020

bcmills commented Dec 3, 2021

alexbrainman commented Aug 19, 2022

mknyszek commented Sep 6, 2022

gopherbot commented Sep 28, 2022

gopherbot commented Oct 4, 2022

gopherbot commented Nov 4, 2022

gopherbot commented Nov 17, 2022

gopherbot commented Jan 4, 2023

gopherbot commented Jan 25, 2023

gopherbot commented Apr 13, 2023

gopherbot commented May 10, 2023

This comment was marked as off-topic.

katiehockman commented Oct 3, 2018 •

edited by bcmills

bcmills commented Jan 27, 2020 •

edited