"context deadline exceeded" - Continuous delay when running with (broken?) DNS settings #289
Comments
|
On a related note, we might want to rephrase the error above ( |
fgsch
changed the title
|
I can reproduce this issue outside the CLI so it looks like a cross compiling problem. |
|
Everything indicates this is because cgo is disabled during cross compilation. Quoting from https://golang.org/pkg/net/#hdr-Name_Resolution:
Running the binary built natively under macos and setting |
Can you confirm this is a function of the resolver used by the CLI, and not your system's IPv6 setup? What hostname/IP is the CLI trying to resolve? If you resolve that manually, does it work quickly? |
|
@peterbourgon Yes, I can confirm it works using e.g. curl or running a binary built under macos from the same tag as the one released. |
|
So, in summary, it appears that your system DNS resolver works correctly when you have IPv6 enabled, but Go's DNS resolver does not, and triggers this timeout? To confirm (again 😉) could you try the following commands with the official version of the CLI? (The +1 adds DNS debug information, which may be useful.) I took those env vars from the package net docs, which also tell us that
So, if that ^^ does confirm the issue, it would suggest that the DNS servers in your /etc/resolv.conf aren't resolving IPv6 addresses correctly. The debug information provided by the above commands may be helpful in confirming. If that's true, what should we do? It's possible to build the CLI in a way that forces the cgo-based resolver in all circumstances, but it's not obvious to me that that's the right solution. |
|
The second command times out as well with the official version, I assume because the cross compiled version does not have the cgo codepath builtin - as with the first command it shows If I try with a natively compiled version (which should include both the cgo and the go variants), this works as expected. The same with a test program cross compiled (added below).
Maybe, but then
To me the right solution is to use whatever it works regardless of the OS or particular settings, in this case the native (cgo-based) resolution mechanism. After all this works for natively built binaries, and AFAICT it is a side effect of cross compiling the macos binary without CGO available. |
|
Repro: package main
import (
"context"
"fmt"
"net/http"
"time"
)
func main() {
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
req, err := http.NewRequestWithContext(ctx, http.MethodGet, "http://v6.testmyipv6.com/", nil)
if err != nil {
panic(err)
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
panic(err)
}
fmt.Println(resp)
} |
The fact that this is the preferred choice for you does not by itself mean it will be the preferred choice for everyone. It might mean that, but we don't have enough information right now to reach that conclusion. Is there any indication that many/most/all macOS systems won't be able to resolve IPv6 lookups to api.fastly.com correctly using the Go DNS resolver? It seems to me this is a function of your ISP and/or network settings, and not the OS itself. (I could be wrong!)
That comment seems to be descriptive, not normative. It's not saying this file is or isn't preferred, it's saying most processes don't use it. |
|
As another data point, my current ISP doesn't offer IPv6, but when I use my VPN, both pure-Go and CGO resolvers work fine. package main
import (
"context"
"fmt"
"log"
"net/http"
"time"
)
func main() {
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
req, err := http.NewRequestWithContext(ctx, http.MethodGet, "http://v6.testmyipv6.com/", nil)
if err != nil {
log.Fatal(err)
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
log.Fatal(err)
}
fmt.Println(resp.Status)
} |
You make it sound like it's a personal preference. It is not. I'm really having a hard time understanding why you are so adamant about this.
When it comes to local network settings, under macOS these are quite limited AFAICT, and other than enabling or disabling IPv6, or changing your DNSs manually, there is no much you can do. My router is limited to these options as well. (In both cases I'm using the default / automatic settings). Beyond that it is the ISP, and it'd be hard to tell what is going on.
api.fastly.com does not have an IPv6 address :)
Maybe but I have not encountered any issues with my ISP and/or network settings outside the CLI.
Quoting my original comment "I suspect it's both a network and a cross-compiling issue". |
|
With help from @peterbourgon we got to the bottom of the problem. My access point is providing a non working primary nameserver and a working secondary one. When the pure Go resolver is used, iterating through the list of nameservers and resolving the hostname takes more than 5 seconds ( When the CGO based (native) resolver is used, iterating through the list and resolving the hostname is much quicker (a few orders of magnitude faster than the pure Go implementation) and the CLI manages to complete the resolution and fetch within the allotted time. |
|
For completion, in my box successfully resolving Code: package main
import "net"
func main() {
_, err := net.LookupHost("developer.fastly.com")
if err != nil {
panic(err)
}
} |
|
+1 I'm hitting this issue as well. I've worked around it by hard coding the IPv4 address for |
|
> dig AAAA developer.fastly.com
; <<>> DiG 9.10.6 <<>> AAAA developer.fastly.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 28204
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;developer.fastly.com. IN AAAA
;; ANSWER SECTION:
developer.fastly.com. 1800 IN CNAME devhub.map.fastly.net.
devhub.map.fastly.net. 30 IN AAAA 2a04:4e42::313
devhub.map.fastly.net. 30 IN AAAA 2a04:4e42:200::313
devhub.map.fastly.net. 30 IN AAAA 2a04:4e42:400::313
devhub.map.fastly.net. 30 IN AAAA 2a04:4e42:600::313
;; Query time: 27 msec
;; SERVER: 192.168.1.122#53(192.168.1.122)
;; WHEN: Tue Jun 01 14:22:53 PDT 2021
;; MSG SIZE rcvd: 196I have a dual stack, is anyone exclusively using IPv6? wget is able to resolve and connect using IPv6 > wget "https://developer.fastly.com/api/internal/cli-config"
--2021-06-01 14:26:36-- https://developer.fastly.com/api/internal/cli-config
Resolving developer.fastly.com (developer.fastly.com)... 2a04:4e42::313, 2a04:4e42:200::313, 2a04:4e42:400::313, ...
Connecting to developer.fastly.com (developer.fastly.com)|2a04:4e42::313|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 855 [application/toml]
Saving to: ‘cli-config’
cli-config 100%[==========================================>] 855 --.-KB/s in 0s
2021-06-01 14:26:37 (22.6 MB/s) - ‘cli-config’ saved [855/855] |
|
@ckaznocha What are the entries in your /etc/resolv.conf, and what happens if you query them directly with dig? So far "it works with X but fails with Y" hasn't been high-value signal because it's at least one degree removed from the actual issue: the pure Go resolver queries the nameservers in your /etc/resolv.conf, so if they don't work then you have a problem. But they should work! (Right?) |
Just to be clear, the one that does not have an IPv6 address is |
|
Ran into this repeatably here. Disabling ipv6 works around it. Happy to try any experiments. |
|
These work: These hang: /etc/resolv.conf contains ifconfig says: |
|
Based on that data, it seems that |
|
So, it seems my router is advertising itself as a DNS server; that's its address. Will investigate further... |
kailan
changed the title
|
Still running into this here, yecch. Since the broken DNS is from a large ISP that serves many people, perhaps we should add a check |
fgsch
changed the title
fgsch
changed the title
|
Hello! Just confirming that I am also seeing this context deadline exceeded error when publishing/deploying even just updating the fastly cli app. Once I turn off ipv6 at my end it works no problems though. |
|
What ISP? |
|
One more data point: I experienced exactly this behavior today when tethering my MacBook Pro to my iPhone. The iPhone sets its IP4 and IP6 addresses as DNS servers, but does not respond to IP6 DNS requests. |
|
I believe #528 is the best we can do without adding external dependencies or cross compiling. |
|
It might be worth issuing a warning if the connection can't be made. Users will otherwise be surprised their cli is out of date. |
|
You do get a warning: |
|
Good, then let's close. I didn't see the warning myself, but maybe I blinked. |
Version:
Description:
For some strange reason, when I have ipv6 enabled, updating the configuration fails in my box with a time out.
This causes a 5 seconds delay every time I run the CLI. This does not seem to happen if I build e.g. the v0.30.0 tag natively.
The output when this happens:
When the update fails
last_checkedis not updated, so the next time the CLI is run the update process is triggered.In my case, this means that every time I run the CLI I experience the delay as the update process keeps timing out.
Besides trying to find out why the update is timing out (I suspect it's both a network and a cross-compiling issue) it might be desirable to update the
last_checkedtimestamp to avoid triggering the update continuously when there is a network issue.The text was updated successfully, but these errors were encountered: