test: Use longer timeout for ginkgo DNS lookups

[jrajahalme]

FQDN tests have been failing due to DNS lookups from ginkgo not
succeeding in 30 seconds. Use the longer HelperTimeout (4 minutes)
instead.

Also split the two lookups into two separate WithTimeout()
invocations, so that we do not need to repeat the 1st if the 2nd
fails.

Related: #11848
Related: #10538
Signed-off-by: Jarno Rajahalme jarno@covalent.io

[jrajahalme]

test-me-please

[coveralls]

Coverage decreased (-0.001%) to 37.035% when pulling ae31e75 on pr/jrajahalme/test-ginkgo-longer-dns-timeout into a8066ce on master.

[pchaigno]

Do we really expect a DNS request to take more than 30 seconds? Maybe we could add retries instead?

What's the risk of letting the tests proceed if the domains could not be resolved? We already have the IPs above and they don't seem that unstable. Would we risk missing actual test failures if we issue a warning instead of an error when the preparatory DNS request fails?

[jrajahalme]

@pchaigno WithTimeout() retries the command by default every 5 seconds already.

Based on the test run of this PR vagrant-cache.ci.cilium.io is still 147.75.38.95, but http://jenkins.cilium.io is 54.148.123.155 rather than 104.198.14.52.

[errordeveloper]

Why not use google.com or something similar with high availability guarantees?

[pchaigno]

I don't think changing the domain name we request will have any impact on the success of DNS requests. We already use Google's DNS, if I recall correctly, to resolve domain names.
The problem is likely Packet's connection to the outside, so any DNS resolver will have the same issues.

[jrajahalme]

Hence a longer timeout may help to carry over intermittent connectivity issues.

[pchaigno]

That DNS resolution relies on getaddrinfo(3) under the hood. I don't think getaddrinfo(3) implements any retry. I couldn't find any retry mechanism at the Golang layer either.

In case of intermittent connectivity issues, the request packet won't stay around, it will be dropped. So I don't think waiting longer for an answer is going to solve this.

[joestringer]

@pchaigno WithTimeout() retries the command by default every 5 seconds already.

How many attempts are actually made? The Go net library doesn't document how long the net.LookupHost() request actually takes so probably needs a bit more digging to find out. Basically I'm wondering how many requests the 30s timeout corresponds to and why we think that number of requests are failing (and hence why a longer timeout would improve the behaviour)

[jrajahalme]

@pchaigno @joestringer The default call interval of helpers.RepeatUntilTrue() that helpers.WithTimeout() uses, is 5 seconds, so you would get 5 or 6 repeated calls to net.LookupHost() during the 30 second timeout.

[pchaigno]

Right, I keep forgetting that WithTimeout implements retries :/

LGTM!

[joestringer]

@pchaigno @joestringer The default call interval of helpers.RepeatUntilTrue() that helpers.WithTimeout() uses, is 5 seconds, so you would get 5 or 6 repeated calls to net.LookupHost() during the 30 second timeout.

This assumes that net.LookupHost() returns immediately. Is that true?

[jrajahalme]

This assumes that net.LookupHost() returns immediately. Is that true?

Good question! According to Go docs net.LookupHost() can be backed by a cgo call to getaddrinfo() or by an internal Go resolver, depending on many different considerations. Tried looking into potential latency of getaddrinfo(), and found mentions of multiple seconds at the worst, so I don't know if it is bounded at all. Don't know the internals of the Go implementation.