net: TestLookupDotsWithRemoteSource is flaky #27992

katiehockman · 2018-10-03T20:37:11Z

We are getting flaky test failures in net. This has been somewhat rare in the past few months, but it has failed the last 5 builds, so probably worth looking into now.

--- FAIL: TestLookupDotsWithRemoteSource (15.57s)
    lookup_test.go:694: LookupSRV(xmpp-server, tcp, google.com): lookup _xmpp-server._tcp.google.com on 8.8.8.8:53: read udp 10.50.0.170:50511->8.8.8.8:53: i/o timeout (mode=go)
FAIL
FAIL	net	30.123s

Looks very related to #16865 closed 2 years ago.

The text was updated successfully, but these errors were encountered:

katiehockman · 2018-10-03T20:39:09Z

/cc @mikioh @brad-burch @ianlancetaylor

ianlancetaylor · 2018-10-03T20:52:43Z

ianlancetaylor · 2018-10-03T20:57:36Z

Any chance there was a recent change to whether the builders set GO_BUILDER_FLAKY_NET in the environment? Because the test does not fail in that way if that environment variable is set.

mikioh · 2018-10-03T23:52:32Z

Once http://golang.org/cl/102875 lands, I will smash the use of external DNS speakers such as rsc.io or some public full resolver then flaky tests disappear. For now, I have no spare time to debug the circumstances running the tests, also not sure there's a way to sneak in the (protected) circumstances.

andybons · 2018-11-28T20:16:04Z

@mikioh should we just disable this test until 1.13?

mikioh · 2018-11-29T00:02:48Z

Yes, please. Will try to allocate my spare time for Go 1.13 devel cycle.

gopherbot · 2018-11-29T23:45:08Z

Change https://golang.org/cl/151799 mentions this issue: net: skip flaky TestLookupDotsWithRemoteSource for now

Updates #27992 Change-Id: Ic327df7cc5002a3d537f9117559c25f30e1eab9c Reviewed-on: https://go-review.googlesource.com/c/151799 Run-TryBot: Andrew Bonventre <andybons@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

andybons · 2018-11-30T22:25:34Z

@mikioh done. Assigning back to you. Thanks.

bcmills · 2019-05-15T13:44:01Z

Here's the same failure mode on dragonfly-amd64: https://build.golang.org/log/2f33b64e758b0fb89431e34f51f80580e5b56e59

--- FAIL: TestLookupDotsWithRemoteSource (10.85s)
    lookup_test.go:669: LookupCNAME(www.mit.edu, mode=go): lookup www.mit.edu on 192.168.1.100:53: read udp 192.168.1.2:54367->192.168.1.100:53: i/o timeout
FAIL
FAIL	net	68.224s

bcmills · 2019-08-20T18:12:53Z

freebsd-arm-paulzhol (https://build.golang.org/log/daa40802f66a0d79d4d356d551c94456f56785b1):

--- FAIL: TestLookupDotsWithRemoteSource (12.52s)
    lookup_test.go:656: LookupAddr(8.8.8.8): lookup 8.8.8.8.in-addr.arpa. on 192.168.1.253:53: read udp 192.168.1.101:38233->192.168.1.253:53: i/o timeout (mode=go)
FAIL
FAIL	net	44.919s

bcmills · 2020-01-27T14:00:34Z

2020-01-25T05:48:12-01df660/freebsd-arm-paulzhol
2019-11-22T16:04:32-1c5bd34/freebsd-arm-paulzhol
2019-10-26T05:40:14-c3e8a20/linux-s390x-ibm
2019-09-18T19:24:59-90f9426/freebsd-arm-paulzhol
2019-08-19T21:42:54-d6ffc1d/freebsd-arm-paulzhol
2019-07-29T16:52:01-5235501/freebsd-arm-paulzhol
2019-07-01T14:45:49-bc593ea/freebsd-arm-paulzhol
2019-06-19T18:19:01-b373d31/freebsd-arm-paulzhol
2019-05-14T16:36:38-a9e107c/dragonfly-amd64
2019-05-09T18:02:19-c514071/dragonfly-amd64
2019-05-02T04:44:36-74b3c50/dragonfly-amd64
2019-04-28T23:51:04-50ad094/dragonfly-amd64

bcmills · 2020-01-27T14:18:54Z

I did some digging, and I believe that this flakiness is caused by the hard-coded timeout and attempt limits in net/dnsconfig_unix.go:

go/src/net/dnsconfig_unix.go

Lines 41 to 45 in 825ff1e

    
           conf := &dnsConfig{ 
        
           	ndots:    1, 
        
           	timeout:  5 * time.Second, 
        
           	attempts: 2, 
        
           }

On my Linux workstation, man 5 resolv.conf reports these same defaults. However, given that the Go net.Resolver API accepts a context.Context which can provide its own timeout behavior, I think these hard-coded limits are out of place: they might make sense in the context of a legacy POSIX C API, but they don't really fit in a modern Go package.

I would recommend:

Complete this TODO to allow the Resolver to override the system-default timeout (and likely also the default limit on attempts).
Refactor the RemoteSource test to use a Resolver and explicit Context instead of the Context-less top-level LookupAddr function. Bound the timeout and number of retries to the test's deadline (#28135) instead of arbitrary constants.

katiehockman added OS-Darwin Testing labels Oct 3, 2018

ianlancetaylor added the release-blocker label Oct 3, 2018

ianlancetaylor added this to the Go1.12 milestone Oct 3, 2018

katiehockman changed the title ~~net: Flaky test failing in darwin-amd64-10_10~~ net: flaky test failing in darwin-amd64-10_10 Oct 3, 2018

bradfitz mentioned this issue Oct 4, 2018

net: localhost pipes cause test hangs on macOS Sierra #25696

Closed

andybons self-assigned this Nov 29, 2018

andybons assigned mikioh and unassigned andybons Nov 30, 2018

andybons removed the release-blocker label Nov 30, 2018

andybons modified the milestones: Go1.12, Go1.13 Nov 30, 2018

mikioh removed their assignment Dec 4, 2018

bcmills changed the title ~~net: flaky test failing in darwin-amd64-10_10~~ net: TestLookupDotsWithRemoteSource is flaky May 15, 2019

bcmills added the NeedsFix label May 15, 2019

andybons modified the milestones: Go1.13, Go1.14 Jul 8, 2019

bcmills removed the OS-Darwin label Aug 20, 2019

rsc removed this from the Go1.14 milestone Oct 9, 2019

rsc added this to the Backlog milestone Oct 9, 2019

golang / go

net: TestLookupDotsWithRemoteSource is flaky #27992

net: TestLookupDotsWithRemoteSource is flaky #27992

katiehockman commented Oct 3, 2018

katiehockman commented Oct 3, 2018

ianlancetaylor commented Oct 3, 2018

ianlancetaylor commented Oct 3, 2018

mikioh commented Oct 3, 2018

andybons commented Nov 28, 2018

mikioh commented Nov 29, 2018

gopherbot commented Nov 29, 2018

andybons commented Nov 30, 2018

bcmills commented May 15, 2019

bcmills commented Aug 20, 2019

bcmills commented Jan 27, 2020 •

edited

bcmills commented Jan 27, 2020

golang / go

net: TestLookupDotsWithRemoteSource is flaky #27992

net: TestLookupDotsWithRemoteSource is flaky #27992

Comments

katiehockman commented Oct 3, 2018

katiehockman commented Oct 3, 2018

ianlancetaylor commented Oct 3, 2018

ianlancetaylor commented Oct 3, 2018

mikioh commented Oct 3, 2018

andybons commented Nov 28, 2018

mikioh commented Nov 29, 2018

gopherbot commented Nov 29, 2018

andybons commented Nov 30, 2018

bcmills commented May 15, 2019

bcmills commented Aug 20, 2019

bcmills commented Jan 27, 2020 • edited

bcmills commented Jan 27, 2020

bcmills commented Jan 27, 2020 •

edited