Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CNAME multiple identical lines in output #21

Open
uggyuggy opened this issue Apr 21, 2022 · 4 comments
Open

CNAME multiple identical lines in output #21

uggyuggy opened this issue Apr 21, 2022 · 4 comments

Comments

@uggyuggy
Copy link

Hi,
Thank's for sharing q

I understand not putting a record type, test/output the results for MX TXT NS, A AAAA CNAME .. (which is nice default)

But in some cases, this will report the identical CNAME answer for each of those records types.
Which may be quite confusing into the output as all exact same CNAME line.

$ q -V
https://github.com/natesales/q version 0.5.8 (e84c6bbbfaa762f904ffbbd402f83630fa67589e 2022-04-18T20:59:46Z)
$
$ q www.eff.org
www.eff.org. 2h0m0s CNAME eff.map.fastly.net.
www.eff.org. 2h0m0s CNAME eff.map.fastly.net.
www.eff.org. 2h0m0s CNAME eff.map.fastly.net.
www.eff.org. 2h0m0s CNAME eff.map.fastly.net.
www.eff.org. 2h0m0s CNAME eff.map.fastly.net.
www.eff.org. 2h0m0s CNAME eff.map.fastly.net.
eff.map.fastly.net. 30s AAAA 2a04:4e42:1d::201
eff.map.fastly.net. 30s A 151.101.120.201
$
$ q www.eff.org | sort -u
eff.map.fastly.net. 30s A 151.101.120.201
eff.map.fastly.net. 30s AAAA 2a04:4e42:1d::201
www.eff.org. 1h48m16s CNAME eff.map.fastly.net.
@natesales
Copy link
Owner

This should be a trivial fix, but I wanted to confirm what you think the default behavior should be. What do you think about only removing duplicate answers when using the default query types?

@uggyuggy
Copy link
Author

Hi @natesales,

Thank's for your work on this tool.

Yes, I think still asking all 6 types, and removing only duplicates is the best default behavior.
The TTL may possibly slightly different in some replies

www.eff.org. 2h0m0s CNAME eff.map.fastly.net.
www.eff.org. 1h58m36s CNAME eff.map.fastly.net.
www.eff.org. 1h57m46s CNAME eff.map.fastly.net.
...

So I would ignore this TTL value before removing duplicates, and I would report in output the longest TTL found

www.eff.org. 2h0m0s CNAME eff.map.fastly.net.

Thank's !

@smlx
Copy link

smlx commented Dec 21, 2023

This remains a problem in other formatting modes e.g. column.

@natesales natesales reopened this Dec 21, 2023
@polarathene polarathene mentioned this issue May 12, 2024
Open
@polarathene
Copy link

TL;DR (UPDATE 3):

  • This duplicate behaviour with CNAME still occurs even with pretty, it's not limited to column.
    • Number of records to query for appears to affect the requests / duplicates?
    • The same query seems to return a higher number of CNAME duplicates when the q command is repeated again.
      • Sometimes I would get a timeout error with @1.1.1.1 (Presumably too many queries at once? Potentially hit a rate limit)
    • Content to show in output is likely to only differ by TTL
      • Although sometimes an FQDN has multiple CNAME associated to it, it appears q will choose one to resolve, but that choice is independent for IPv4 and IPv6 (possibly does both CNAME and takes the first response for each?).
      • Thus you may still have two separate CNAME entries, when resolving other record types, but for just CNAME it may only output the first one found (returned?) rather than all.
  • For output, could filter out duplicates by ignoring the TTL?
    • I've proposed alternative --format at the end of this comment that is configurable with which columns/fields to show.
    • Even without de-duplicate logic in q, the user could easily remove duplicates via sort -u / uniq provided TTL can be omitted from results?
Update prior to TLDR (collapsed for brevity)

UPDATE: The CNAME records TTL while inconsistent across repeating the q command, introduces the duplicates after the initial query (like how A records show a decrease in TTL).

  • The duplicates appear to be multiplied by the number of record types being queried for the resolved CNAME record (not only types returned via response).
  • If the CNAME resolves to another CNAME record, you'll get repetition there too, so both CNAME will be duplicated 3 times for CNAME A AAAA (may require repeating q a few times).

I am not familiar with DNS enough to know, but assume that the only real difference here is TTL? Below I have noted a query that sometimes responded with a result with two CNAME records to different CDNs, but the 2nd CDN CNAME was sporadic in results and it did not appear to be resolved to the other queried records, only the first CNAME (while q CNAME fqdn-here would return either CNAME, but not both, first in first-served?).

UPDATE 2: With CNAME => CNAME => A / AAAA, this observation didn't quite behave the same. I've not inspected the YAML/JSON output, duplication was minimized:

Same request, variety of responses (Click to expand) 
# Different CNAME resolved for IPv4 vs IPv6
# x1 static.crates.io => cloudfront-static.crates.io
# x1 cloudfront-static.crates.io => d19xqa3lc3clo8.cloudfront.net
# x2 static.crates.io => fastly-static.crates.io
# x2 fastly-static.crates.io => dualstack.k.sni.global.fastly.net
$ q @1.1.1.1 CNAME A NS TXT AAAA static.crates.io
static.crates.io. 5m CNAME fastly-static.crates.io.
d19xqa3lc3clo8.cloudfront.net. 18s A 65.9.141.105
d19xqa3lc3clo8.cloudfront.net. 18s A 65.9.141.122
d19xqa3lc3clo8.cloudfront.net. 18s A 65.9.141.40
d19xqa3lc3clo8.cloudfront.net. 18s A 65.9.141.65
static.crates.io. 4m18s CNAME cloudfront-static.crates.io.
cloudfront-static.crates.io. 4m18s CNAME d19xqa3lc3clo8.cloudfront.net.
fastly-static.crates.io. 1m CNAME dualstack.k.sni.global.fastly.net.
fastly-static.crates.io. 54s CNAME dualstack.k.sni.global.fastly.net.
static.crates.io. 4m54s CNAME fastly-static.crates.io.
dualstack.k.sni.global.fastly.net. 30s AAAA 2a04:4e42:200::649
dualstack.k.sni.global.fastly.net. 30s AAAA 2a04:4e42:400::649
dualstack.k.sni.global.fastly.net. 30s AAAA 2a04:4e42:600::649
dualstack.k.sni.global.fastly.net. 30s AAAA 2a04:4e42::649

# This time IPv4 and IPv6 hosts switched, and CNAME duplicates for fastly differ:
# x1 static.crates.io => cloudfront-static.crates.io
# x1 cloudfront-static.crates.io => d19xqa3lc3clo8.cloudfront.net
# x3 static.crates.io => fastly-static.crates.io
# x1 fastly-static.crates.io => dualstack.k.sni.global.fastly.net
$ q @1.1.1.1 CNAME A NS TXT AAAA static.crates.io
d19xqa3lc3clo8.cloudfront.net. 46s AAAA 2600:9000:204b:4000:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 46s AAAA 2600:9000:204b:4c00:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 46s AAAA 2600:9000:204b:8200:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 46s AAAA 2600:9000:204b:c000:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 46s AAAA 2600:9000:204b:c400:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 46s AAAA 2600:9000:204b:d000:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 46s AAAA 2600:9000:204b:f600:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 46s AAAA 2600:9000:204b:f800:1b:d7a6:ee00:93a1
static.crates.io. 2m CNAME cloudfront-static.crates.io.
cloudfront-static.crates.io. 2m CNAME d19xqa3lc3clo8.cloudfront.net.
static.crates.io. 2m43s CNAME fastly-static.crates.io.
dualstack.k.sni.global.fastly.net. 30s A 151.101.130.137
dualstack.k.sni.global.fastly.net. 30s A 151.101.194.137
dualstack.k.sni.global.fastly.net. 30s A 151.101.2.137
dualstack.k.sni.global.fastly.net. 30s A 151.101.66.137
fastly-static.crates.io. 1m CNAME dualstack.k.sni.global.fastly.net.
static.crates.io. 1m59s CNAME fastly-static.crates.io.
static.crates.io. 5m CNAME fastly-static.crates.io.

# No cloudfront CNAME thus less duplicates, all queries to fastly (1 per IPv4 + IPv6?):
# x2 static.crates.io => fastly-static.crates.io
# x2 fastly-static.crates.io => dualstack.k.sni.global.fastly.net
$ q @1.1.1.1 CNAME A NS TXT AAAA static.crates.io
static.crates.io. 5m CNAME fastly-static.crates.io.
dualstack.k.sni.global.fastly.net. 30s A 151.101.130.137
dualstack.k.sni.global.fastly.net. 30s A 151.101.194.137
dualstack.k.sni.global.fastly.net. 30s A 151.101.2.137
dualstack.k.sni.global.fastly.net. 30s A 151.101.66.137
fastly-static.crates.io. 1m CNAME dualstack.k.sni.global.fastly.net.
static.crates.io. 1m20s CNAME fastly-static.crates.io.
fastly-static.crates.io. 57s CNAME dualstack.k.sni.global.fastly.net.
static.crates.io. 4m57s CNAME fastly-static.crates.io.
dualstack.k.sni.global.fastly.net. 30s AAAA 2a04:4e42:200::649
dualstack.k.sni.global.fastly.net. 30s AAAA 2a04:4e42:400::649
dualstack.k.sni.global.fastly.net. 30s AAAA 2a04:4e42:600::649
dualstack.k.sni.global.fastly.net. 30s AAAA 2a04:4e42::649

# 1 less layer of CNAME indirection:
# x5 cloudfront-static.crates.io => d19xqa3lc3clo8.cloudfront.net
$ q @1.1.1.1 CNAME A NS TXT AAAA cloudfront-static.crates.io
cloudfront-static.crates.io. 5m CNAME d19xqa3lc3clo8.cloudfront.net.
d19xqa3lc3clo8.cloudfront.net. 1s A 3.163.43.100
d19xqa3lc3clo8.cloudfront.net. 1s A 3.163.43.21
d19xqa3lc3clo8.cloudfront.net. 1s A 3.163.43.28
d19xqa3lc3clo8.cloudfront.net. 1s A 3.163.43.37
cloudfront-static.crates.io. 3m59s CNAME d19xqa3lc3clo8.cloudfront.net.
cloudfront-static.crates.io. 3m14s CNAME d19xqa3lc3clo8.cloudfront.net.
d19xqa3lc3clo8.cloudfront.net. 47h58m14s NS ns-1118.awsdns-11.org.
d19xqa3lc3clo8.cloudfront.net. 47h58m14s NS ns-1570.awsdns-04.co.uk.
d19xqa3lc3clo8.cloudfront.net. 47h58m14s NS ns-246.awsdns-30.com.
d19xqa3lc3clo8.cloudfront.net. 47h58m14s NS ns-793.awsdns-35.net.
cloudfront-static.crates.io. 3m42s CNAME d19xqa3lc3clo8.cloudfront.net.
d19xqa3lc3clo8.cloudfront.net. 1m AAAA 2600:9000:204b:2000:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 1m AAAA 2600:9000:204b:2600:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 1m AAAA 2600:9000:204b:8800:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 1m AAAA 2600:9000:204b:8e00:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 1m AAAA 2600:9000:204b:9c00:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 1m AAAA 2600:9000:204b:a400:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 1m AAAA 2600:9000:204b:c000:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 1m AAAA 2600:9000:204b:d600:1b:d7a6:ee00:93a1
cloudfront-static.crates.io. 3m11s CNAME d19xqa3lc3clo8.cloudfront.net.

# When TTL cache is cold, no duplicates:
# x1 static.crates.io => fastly-static.crates.io
# x1 fastly-static.crates.io => dualstack.k.sni.global.fastly.net
$ q @1.1.1.1 CNAME A NS TXT AAAA static.crates.io
static.crates.io. 5m CNAME fastly-static.crates.io.
dualstack.k.sni.global.fastly.net. 30s A 151.101.130.137
dualstack.k.sni.global.fastly.net. 30s A 151.101.194.137
dualstack.k.sni.global.fastly.net. 30s A 151.101.2.137
dualstack.k.sni.global.fastly.net. 30s A 151.101.66.137
fastly-static.crates.io. 1m CNAME dualstack.k.sni.global.fastly.net.
dualstack.k.sni.global.fastly.net. 30s AAAA 2a04:4e42:200::649
dualstack.k.sni.global.fastly.net. 30s AAAA 2a04:4e42:400::649
dualstack.k.sni.global.fastly.net. 30s AAAA 2a04:4e42:600::649
dualstack.k.sni.global.fastly.net. 30s AAAA 2a04:4e42::649

# Attempt to opt-out of possible influences?:
# x3 static.crates.io => fastly-static.crates.io
# x4 fastly-static.crates.io => dualstack.k.sni.global.fastly.net
$ q @1.1.1.1 CNAME A NS TXT AAAA static.crates.io +noreuse-conn +noid-check
static.crates.io. 5m CNAME fastly-static.crates.io.
dualstack.k.sni.global.fastly.net. 30s A 151.101.130.137
dualstack.k.sni.global.fastly.net. 30s A 151.101.194.137
dualstack.k.sni.global.fastly.net. 30s A 151.101.2.137
dualstack.k.sni.global.fastly.net. 30s A 151.101.66.137
fastly-static.crates.io. 24s CNAME dualstack.k.sni.global.fastly.net.
static.crates.io. 4m24s CNAME fastly-static.crates.io.
fastly-static.crates.io. 10s CNAME dualstack.k.sni.global.fastly.net.
static.crates.io. 4m10s CNAME fastly-static.crates.io.
dualstack.k.sni.global.fastly.net. 6s AAAA 2a04:4e42:200::649
dualstack.k.sni.global.fastly.net. 6s AAAA 2a04:4e42:400::649
dualstack.k.sni.global.fastly.net. 6s AAAA 2a04:4e42:600::649
dualstack.k.sni.global.fastly.net. 6s AAAA 2a04:4e42::649
fastly-static.crates.io. 36s CNAME dualstack.k.sni.global.fastly.net.
static.crates.io. 4m36s CNAME fastly-static.crates.io.
Original response

This remains a problem in other formatting modes like column

For reference:

$ q @1.1.1.1 static.crates.io --format column
    A 30s 151.101.130.137
    A 30s 151.101.194.137
    A 30s 151.101.2.137
    A 30s 151.101.66.137
 AAAA 30s 2a04:4e42:200::649
 AAAA 30s 2a04:4e42:400::649
 AAAA 30s 2a04:4e42:600::649
 AAAA 30s 2a04:4e42::649
CNAME 1m  dualstack.k.sni.global.fastly.net.
CNAME 5m  fastly-static.crates.io.

# Again shortly afterwards (16s):
$ q @1.1.1.1 static.crates.io --format column
    A 14s   151.101.130.137
    A 14s   151.101.194.137
    A 14s   151.101.2.137
    A 14s   151.101.66.137
 AAAA 30s   2a04:4e42:200::649
 AAAA 30s   2a04:4e42:400::649
 AAAA 30s   2a04:4e42:600::649
 AAAA 30s   2a04:4e42::649
CNAME 44s   dualstack.k.sni.global.fastly.net.
CNAME 1m    dualstack.k.sni.global.fastly.net.
CNAME 4m44s fastly-static.crates.io.
CNAME 3m31s fastly-static.crates.io.
CNAME 5m    fastly-static.crates.io.

The CNAME records appear to be duplicated due to the TTL variance? Personally I'm not interested in the TTL most of the time, but there doesn't appear to be an easy way to filter that out.

UPDATE: This actually occurs with the default pretty output too:

$ q @1.1.1.1 static.crates.io

dualstack.k.sni.global.fastly.net. 30s A 151.101.130.137
dualstack.k.sni.global.fastly.net. 30s A 151.101.194.137
dualstack.k.sni.global.fastly.net. 30s A 151.101.2.137
dualstack.k.sni.global.fastly.net. 30s A 151.101.66.137
fastly-static.crates.io. 1m CNAME dualstack.k.sni.global.fastly.net.
static.crates.io. 5m CNAME fastly-static.crates.io.
dualstack.k.sni.global.fastly.net. 30s AAAA 2a04:4e42:200::649
dualstack.k.sni.global.fastly.net. 30s AAAA 2a04:4e42:400::649
dualstack.k.sni.global.fastly.net. 30s AAAA 2a04:4e42:600::649
dualstack.k.sni.global.fastly.net. 30s AAAA 2a04:4e42::649

$ q @1.1.1.1 static.crates.io
fastly-static.crates.io. 30s CNAME dualstack.k.sni.global.fastly.net.
static.crates.io. 4m30s CNAME fastly-static.crates.io.
fastly-static.crates.io. 1m CNAME dualstack.k.sni.global.fastly.net.
static.crates.io. 5m CNAME fastly-static.crates.io.
dualstack.k.sni.global.fastly.net. 30s A 151.101.130.137
dualstack.k.sni.global.fastly.net. 30s A 151.101.194.137
dualstack.k.sni.global.fastly.net. 30s A 151.101.2.137
dualstack.k.sni.global.fastly.net. 30s A 151.101.66.137
dualstack.k.sni.global.fastly.net. 30s AAAA 2a04:4e42:200::649
dualstack.k.sni.global.fastly.net. 30s AAAA 2a04:4e42:400::649
dualstack.k.sni.global.fastly.net. 30s AAAA 2a04:4e42:600::649
dualstack.k.sni.global.fastly.net. 30s AAAA 2a04:4e42::649
fastly-static.crates.io. 29s CNAME dualstack.k.sni.global.fastly.net.
static.crates.io. 4m29s CNAME fastly-static.crates.io.

Potentially related, when I queried again after the 2nd time it stalled and I got a failure, presumably the number of DNS requests is growing?:

q @1.1.1.1 static.crates.io
FATA[0010] read udp 172.17.0.9:60735->1.1.1.1:53: i/o timeout

172.17.0.9 is the IP of the docker container I'm running q in. It's /etc/resolv.conf DNS nameserver is configured via Docker Desktop (Windows 11), which differs from the one within the Docker host (WSL2), and the Windows host itself (1.1.1.1). So a bit of indirection going on if I don't specify @1.1.1.1 directly, but in this case it's unclear why q had trouble unless it's trying to query a larger request each time?

Possibly related to the above error, which may be related to the duplicates (click to expand)

I know presently without using 1.1.1.1, the router DNS from /etc/resolv.conf misbehaves with some DNS queries (it cannot handle rDNS queries for example). I used q --verbose to troubleshoot, however in this case q has no issue with this query to static.crates.io (regardless of DNS server used), but doggo without 1.1.1.1 failed with:

$ doggo static.crates.io
ERROR[2024-05-12T01:04:49Z] error looking up DNS records                  error="dns: buffer size too small"

That seems like it might be related to CNAME records as other domains I've tried (google.com and github.com, both only appear to have a single A record) don't appear to have CNAME records, while static.crates.io won't fail with doggo like above when using 1.1.1.1 - it returns all 4 A records after resolving the double CNAME indirection:

$ doggo @1.1.1.1 static.crates.io

NAME                                    TYPE    CLASS   TTL     ADDRESS                                 NAMESERVER
static.crates.io.                       CNAME   IN      171s    fastly-static.crates.io.                1.1.1.1:53
fastly-static.crates.io.                CNAME   IN      56s     dualstack.k.sni.global.fastly.net.      1.1.1.1:53
dualstack.k.sni.global.fastly.net.      A       IN      26s     151.101.2.137                           1.1.1.1:53
dualstack.k.sni.global.fastly.net.      A       IN      26s     151.101.130.137                         1.1.1.1:53
dualstack.k.sni.global.fastly.net.      A       IN      26s     151.101.194.137                         1.1.1.1:53
dualstack.k.sni.global.fastly.net.      A       IN      26s     151.101.66.137                          1.1.1.1:53

but is fine resolving just the CNAME record without 1.1.1.1:

$ doggo CNAME static.crates.io
NAME                    TYPE    CLASS   TTL     ADDRESS                         NAMESERVER
static.crates.io.       CNAME   IN      300s    fastly-static.crates.io.        192.168.65.7:53

Thus doggo buffer error is related to resolving a much larger response? Perhaps similar is happening with the q error due to the increased records (duplicates) in the response? Especially since q seems to discover more than doggo, but it is unclear if it's following both CNAME or just resolving the first one (which seems to be the case).

Maybe it's the CNAME => CNAME => A / AAAA resolution, doggo had no problems resolving CNAME => A:

# When using the DNS server from the router / ISP, duplicate records are also returned by doggo in this case:
# EDIT: Cannot reproduce after trying again, some sort of cache/issue presumably with this service / router handling responses.
$ doggo cloudfront-static.crates.io
NAME                            TYPE    CLASS   TTL     ADDRESS                         NAMESERVER
cloudfront-static.crates.io.    CNAME   IN      22s     d19xqa3lc3clo8.cloudfront.net.  192.168.65.7:53
d19xqa3lc3clo8.cloudfront.net.  A       IN      44s     65.9.141.40                     192.168.65.7:53
d19xqa3lc3clo8.cloudfront.net.  A       IN      44s     65.9.141.122                    192.168.65.7:53
d19xqa3lc3clo8.cloudfront.net.  A       IN      44s     65.9.141.105                    192.168.65.7:53
d19xqa3lc3clo8.cloudfront.net.  A       IN      44s     65.9.141.65                     192.168.65.7:53
d19xqa3lc3clo8.cloudfront.net.  A       IN      22s     65.9.141.122                    192.168.65.7:53
d19xqa3lc3clo8.cloudfront.net.  A       IN      22s     65.9.141.40                     192.168.65.7:53
d19xqa3lc3clo8.cloudfront.net.  A       IN      22s     65.9.141.105                    192.168.65.7:53
d19xqa3lc3clo8.cloudfront.net.  A       IN      22s     65.9.141.65                     192.168.65.7:53

# However, when querying directly instead of via the CNAME, no duplicates:
$ doggo d19xqa3lc3clo8.cloudfront.net
NAME                            TYPE    CLASS   TTL     ADDRESS         NAMESERVER
d19xqa3lc3clo8.cloudfront.net.  A       IN      49s     65.9.141.122    192.168.65.7:53
d19xqa3lc3clo8.cloudfront.net.  A       IN      49s     65.9.141.65     192.168.65.7:53
d19xqa3lc3clo8.cloudfront.net.  A       IN      49s     65.9.141.40     192.168.65.7:53
d19xqa3lc3clo8.cloudfront.net.  A       IN      49s     65.9.141.105    192.168.65.7:53

# Unique despite indirect (EDIT: Appears to be due to defaults):
$ doggo @1.1.1.1 cloudfront-static.crates.io
NAME                            TYPE    CLASS   TTL     ADDRESS                         NAMESERVER
cloudfront-static.crates.io.    CNAME   IN      251s    d19xqa3lc3clo8.cloudfront.net.  1.1.1.1:53
d19xqa3lc3clo8.cloudfront.net.  A       IN      11s     65.9.141.105                    1.1.1.1:53
d19xqa3lc3clo8.cloudfront.net.  A       IN      11s     65.9.141.65                     1.1.1.1:53
d19xqa3lc3clo8.cloudfront.net.  A       IN      11s     65.9.141.40                     1.1.1.1:53
d19xqa3lc3clo8.cloudfront.net.  A       IN      11s     65.9.141.122                    1.1.1.1:53

# Not unique when specifically requesting both record types:
$ doggo @1.1.1.1 CNAME A cloudfront-static.crates.io
NAME                            TYPE    CLASS   TTL     ADDRESS                         NAMESERVER
cloudfront-static.crates.io.    CNAME   IN      300s    d19xqa3lc3clo8.cloudfront.net.  1.1.1.1:53
cloudfront-static.crates.io.    CNAME   IN      271s    d19xqa3lc3clo8.cloudfront.net.  1.1.1.1:53
d19xqa3lc3clo8.cloudfront.net.  A       IN      31s     65.9.141.40                     1.1.1.1:53
d19xqa3lc3clo8.cloudfront.net.  A       IN      31s     65.9.141.105                    1.1.1.1:53
d19xqa3lc3clo8.cloudfront.net.  A       IN      31s     65.9.141.122                    1.1.1.1:53
d19xqa3lc3clo8.cloudfront.net.  A       IN      31s     65.9.141.65                     1.1.1.1:53

# But works fine when only requesting single record type:
$ doggo @1.1.1.1 CNAME cloudfront-static.crates.io
NAME                            TYPE    CLASS   TTL     ADDRESS                         NAMESERVER
cloudfront-static.crates.io.    CNAME   IN      186s    d19xqa3lc3clo8.cloudfront.net.  1.1.1.1:53
$ doggo @1.1.1.1 A cloudfront-static.crates.io
NAME                            TYPE    CLASS   TTL     ADDRESS                         NAMESERVER
cloudfront-static.crates.io.    CNAME   IN      153s    d19xqa3lc3clo8.cloudfront.net.  1.1.1.1:53
d19xqa3lc3clo8.cloudfront.net.  A       IN      60s     65.9.141.40                     1.1.1.1:53
d19xqa3lc3clo8.cloudfront.net.  A       IN      60s     65.9.141.65                     1.1.1.1:53
d19xqa3lc3clo8.cloudfront.net.  A       IN      60s     65.9.141.122                    1.1.1.1:53
d19xqa3lc3clo8.cloudfront.net.  A       IN      60s     65.9.141.105                    1.1.1.1:53

I observed the same with q:

# CNAME repeated 4 times (per A?)
$ q @1.1.1.1 cloudfront-static.crates.io
cloudfront-static.crates.io. 2m56s CNAME d19xqa3lc3clo8.cloudfront.net.
cloudfront-static.crates.io. 3m11s CNAME d19xqa3lc3clo8.cloudfront.net.
cloudfront-static.crates.io. 2m54s CNAME d19xqa3lc3clo8.cloudfront.net.
d19xqa3lc3clo8.cloudfront.net. 39s A 65.9.141.105
d19xqa3lc3clo8.cloudfront.net. 39s A 65.9.141.122
d19xqa3lc3clo8.cloudfront.net. 39s A 65.9.141.40
d19xqa3lc3clo8.cloudfront.net. 39s A 65.9.141.65
d19xqa3lc3clo8.cloudfront.net. 1m AAAA 2600:9000:2212:3e00:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 1m AAAA 2600:9000:2212:4800:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 1m AAAA 2600:9000:2212:6e00:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 1m AAAA 2600:9000:2212:7200:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 1m AAAA 2600:9000:2212:7e00:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 1m AAAA 2600:9000:2212:9200:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 1m AAAA 2600:9000:2212:d200:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 1m AAAA 2600:9000:2212:de00:1b:d7a6:ee00:93a1
cloudfront-static.crates.io. 5m CNAME d19xqa3lc3clo8.cloudfront.net.
d19xqa3lc3clo8.cloudfront.net. 47h57m54s NS ns-1118.awsdns-11.org.
d19xqa3lc3clo8.cloudfront.net. 47h57m54s NS ns-1570.awsdns-04.co.uk.
d19xqa3lc3clo8.cloudfront.net. 47h57m54s NS ns-246.awsdns-30.com.
d19xqa3lc3clo8.cloudfront.net. 47h57m54s NS ns-793.awsdns-35.net.

# Without 1.1.1.1, duplicate A / AAAA + NS records are also returned:
$ q cloudfront-static.crates.io

d19xqa3lc3clo8.cloudfront.net. 20s A 18.67.93.107
d19xqa3lc3clo8.cloudfront.net. 20s A 18.67.93.30
d19xqa3lc3clo8.cloudfront.net. 20s A 18.67.93.39
d19xqa3lc3clo8.cloudfront.net. 20s A 18.67.93.48
d19xqa3lc3clo8.cloudfront.net. 11s A 65.9.141.105
d19xqa3lc3clo8.cloudfront.net. 11s A 65.9.141.122
d19xqa3lc3clo8.cloudfront.net. 11s A 65.9.141.40
d19xqa3lc3clo8.cloudfront.net. 11s A 65.9.141.65
cloudfront-static.crates.io. 20s CNAME d19xqa3lc3clo8.cloudfront.net.
d19xqa3lc3clo8.cloudfront.net. 20s AAAA 2600:9000:204b:1400:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 41s AAAA 2600:9000:204b:2000:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 41s AAAA 2600:9000:204b:2e00:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 41s AAAA 2600:9000:204b:3000:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 41s AAAA 2600:9000:204b:6200:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 20s AAAA 2600:9000:204b:7800:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 20s AAAA 2600:9000:204b:7a00:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 20s AAAA 2600:9000:204b:7e00:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 41s AAAA 2600:9000:204b:8200:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 41s AAAA 2600:9000:204b:9200:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 41s AAAA 2600:9000:204b:9600:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 20s AAAA 2600:9000:204b:a600:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 20s AAAA 2600:9000:204b:d600:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 20s AAAA 2600:9000:204b:d800:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 20s AAAA 2600:9000:204b:dc00:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 41s AAAA 2600:9000:204b:ee00:1b:d7a6:ee00:93a1
cloudfront-static.crates.io. 2m48s CNAME d19xqa3lc3clo8.cloudfront.net.
d19xqa3lc3clo8.cloudfront.net. 21h7m8s NS ns-1118.awsdns-11.org.
d19xqa3lc3clo8.cloudfront.net. 2m48s NS ns-1118.awsdns-11.org.
d19xqa3lc3clo8.cloudfront.net. 21h7m8s NS ns-1570.awsdns-04.co.uk.
d19xqa3lc3clo8.cloudfront.net. 2m48s NS ns-1570.awsdns-04.co.uk.
d19xqa3lc3clo8.cloudfront.net. 21h7m8s NS ns-246.awsdns-30.com.
d19xqa3lc3clo8.cloudfront.net. 2m48s NS ns-246.awsdns-30.com.
d19xqa3lc3clo8.cloudfront.net. 21h7m8s NS ns-793.awsdns-35.net.
d19xqa3lc3clo8.cloudfront.net. 2m48s NS ns-793.awsdns-35.net.
cloudfront-static.crates.io. 2m52s CNAME d19xqa3lc3clo8.cloudfront.net.
cloudfront-static.crates.io. 3m7s CNAME d19xqa3lc3clo8.cloudfront.net.
cloudfront-static.crates.io. 2m49s CNAME d19xqa3lc3clo8.cloudfront.net.

# Direct without 1.1.1.1, duplicate A / AAAA records no longer present:
$ q d19xqa3lc3clo8.cloudfront.net

d19xqa3lc3clo8.cloudfront.net. 21h4m23s NS ns-1118.awsdns-11.org.
d19xqa3lc3clo8.cloudfront.net. 21h4m23s NS ns-1570.awsdns-04.co.uk.
d19xqa3lc3clo8.cloudfront.net. 21h4m23s NS ns-246.awsdns-30.com.
d19xqa3lc3clo8.cloudfront.net. 21h4m23s NS ns-793.awsdns-35.net.
d19xqa3lc3clo8.cloudfront.net. 58s A 18.67.93.107
d19xqa3lc3clo8.cloudfront.net. 58s A 18.67.93.30
d19xqa3lc3clo8.cloudfront.net. 58s A 18.67.93.39
d19xqa3lc3clo8.cloudfront.net. 58s A 18.67.93.48
d19xqa3lc3clo8.cloudfront.net. 58s AAAA 2600:9000:204b:1000:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 58s AAAA 2600:9000:204b:1a00:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 58s AAAA 2600:9000:204b:5c00:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 58s AAAA 2600:9000:204b:8800:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 58s AAAA 2600:9000:204b:9400:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 58s AAAA 2600:9000:204b:a800:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 58s AAAA 2600:9000:204b:d800:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 58s AAAA 2600:9000:204b:fa00:1b:d7a6:ee00:93a1

# Thus no difference with 1.1.1.1:
$ q @1.1.1.1 d19xqa3lc3clo8.cloudfront.net

d19xqa3lc3clo8.cloudfront.net. 13s A 65.9.141.105
d19xqa3lc3clo8.cloudfront.net. 13s A 65.9.141.122
d19xqa3lc3clo8.cloudfront.net. 13s A 65.9.141.40
d19xqa3lc3clo8.cloudfront.net. 13s A 65.9.141.65
d19xqa3lc3clo8.cloudfront.net. 13s AAAA 2600:9000:204b:1000:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 13s AAAA 2600:9000:204b:1a00:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 13s AAAA 2600:9000:204b:5c00:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 13s AAAA 2600:9000:204b:8800:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 13s AAAA 2600:9000:204b:9400:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 13s AAAA 2600:9000:204b:a800:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 13s AAAA 2600:9000:204b:d800:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 13s AAAA 2600:9000:204b:fa00:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 48h NS ns-1118.awsdns-11.org.
d19xqa3lc3clo8.cloudfront.net. 48h NS ns-1570.awsdns-04.co.uk.
d19xqa3lc3clo8.cloudfront.net. 48h NS ns-246.awsdns-30.com.
d19xqa3lc3clo8.cloudfront.net. 48h NS ns-793.awsdns-35.net.

# Timeout error was encountered after a repeat check (rate limited?)
$ q @1.1.1.1 d19xqa3lc3clo8.cloudfront.net
FATA[0010] read udp 172.17.0.9:51513->1.1.1.1:53: i/o timeout

Issue identified as due to multiple record types requested:

$ q CNAME cloudfront-static.crates.io
cloudfront-static.crates.io. 1m3s CNAME d19xqa3lc3clo8.cloudfront.net.

$ q CNAME A cloudfront-static.crates.io
cloudfront-static.crates.io. 58s CNAME d19xqa3lc3clo8.cloudfront.net.
d19xqa3lc3clo8.cloudfront.net. 52s A 18.67.93.107
d19xqa3lc3clo8.cloudfront.net. 52s A 18.67.93.30
d19xqa3lc3clo8.cloudfront.net. 52s A 18.67.93.39
d19xqa3lc3clo8.cloudfront.net. 52s A 18.67.93.48
d19xqa3lc3clo8.cloudfront.net. 51s A 65.9.141.105
d19xqa3lc3clo8.cloudfront.net. 51s A 65.9.141.122
d19xqa3lc3clo8.cloudfront.net. 51s A 65.9.141.40
d19xqa3lc3clo8.cloudfront.net. 51s A 65.9.141.65
cloudfront-static.crates.io. 51s CNAME d19xqa3lc3clo8.cloudfront.net.

# Now using 1.1.1.1 to workaround duplicate A records with local DNS server responses:
# NOTE: Must repeat the same request to get the unique TTLs (prior one since, and new one)
$ q @1.1.1.1 CNAME A cloudfront-static.crates.io
cloudfront-static.crates.io. 4m49s CNAME d19xqa3lc3clo8.cloudfront.net.
d19xqa3lc3clo8.cloudfront.net. 1m A 65.9.141.105
d19xqa3lc3clo8.cloudfront.net. 1m A 65.9.141.122
d19xqa3lc3clo8.cloudfront.net. 1m A 65.9.141.40
d19xqa3lc3clo8.cloudfront.net. 1m A 65.9.141.65
cloudfront-static.crates.io. 5m CNAME d19xqa3lc3clo8.cloudfront.net.

# Three CNAME, one per record type resolved from it?
# NOTE: TTLs for CNAME returned vary across runs, they don't decrement like A
# NS also resets TTL in a much shorter time span
$ q @1.1.1.1 CNAME A NS cloudfront-static.crates.io
cloudfront-static.crates.io. 46s CNAME d19xqa3lc3clo8.cloudfront.net.
d19xqa3lc3clo8.cloudfront.net. 39s A 65.9.141.105
d19xqa3lc3clo8.cloudfront.net. 39s A 65.9.141.122
d19xqa3lc3clo8.cloudfront.net. 39s A 65.9.141.40
d19xqa3lc3clo8.cloudfront.net. 39s A 65.9.141.65
cloudfront-static.crates.io. 47s CNAME d19xqa3lc3clo8.cloudfront.net.
cloudfront-static.crates.io. 5m CNAME d19xqa3lc3clo8.cloudfront.net.
d19xqa3lc3clo8.cloudfront.net. 48h NS ns-1118.awsdns-11.org.
d19xqa3lc3clo8.cloudfront.net. 48h NS ns-1570.awsdns-04.co.uk.
d19xqa3lc3clo8.cloudfront.net. 48h NS ns-246.awsdns-30.com.
d19xqa3lc3clo8.cloudfront.net. 48h NS ns-793.awsdns-35.net.

Might be due to the default --reuse-conn? (Nope)

Output improvements?

As can be seen above, the column layout loses a bit of context with the mapping for CNAME, it could perhaps display like MX does with preference express the mapping (not entirely accurate equivalent, see alternative proposal that follows):

Feature request here (with some examples if implemented), to support --format tsv so that it can be used with with other tooling. Easily filter out TTL and do dedupe via external tools if fixing it directly in q is tricky 👍

Full original response with examples 
# Better clarity of the CNAME record indirection that the A/AAAA records are for
$ q @1.1.1.1 static.crates.io --format column

    A 30s 151.101.130.137
    A 30s 151.101.194.137
    A 30s 151.101.2.137
    A 30s 151.101.66.137
 AAAA 30s 2a04:4e42:200::649
 AAAA 30s 2a04:4e42:400::649
 AAAA 30s 2a04:4e42:600::649
 AAAA 30s 2a04:4e42::649
CNAME 1m  fastly-static.crates.io. => dualstack.k.sni.global.fastly.net.
CNAME 5m  static.crates.io. => fastly-static.crates.io.

Or alternatively, as per my feature request (with control over what "columns" / record fields are displayed):

$ q @1.1.1.1 static.crates.io --format column --columns name,type,data

dualstack.k.sni.global.fastly.net. A     151.101.130.137                    
dualstack.k.sni.global.fastly.net. A     151.101.194.137                    
dualstack.k.sni.global.fastly.net. A     151.101.2.137                      
dualstack.k.sni.global.fastly.net. A     151.101.66.137                     
dualstack.k.sni.global.fastly.net. AAAA  2a04:4e42:200::649                 
dualstack.k.sni.global.fastly.net. AAAA  2a04:4e42:400::649                 
dualstack.k.sni.global.fastly.net. AAAA  2a04:4e42:600::649                 
dualstack.k.sni.global.fastly.net. AAAA  2a04:4e42::649                     
fastly-static.crates.io.           CNAME dualstack.k.sni.global.fastly.net. 
static.crates.io.                  CNAME fastly-static.crates.io.        
$ q @1.1.1.1 static.crates.io --format column --columns type,name,data

A     dualstack.k.sni.global.fastly.net. 151.101.130.137                    
A     dualstack.k.sni.global.fastly.net. 151.101.194.137                    
A     dualstack.k.sni.global.fastly.net. 151.101.2.137                      
A     dualstack.k.sni.global.fastly.net. 151.101.66.137                     
AAAA  dualstack.k.sni.global.fastly.net. 2a04:4e42:200::649                 
AAAA  dualstack.k.sni.global.fastly.net. 2a04:4e42:400::649                 
AAAA  dualstack.k.sni.global.fastly.net. 2a04:4e42:600::649                 
AAAA  dualstack.k.sni.global.fastly.net. 2a04:4e42::649                     
CNAME fastly-static.crates.io.           dualstack.k.sni.global.fastly.net. 
CNAME static.crates.io.                  fastly-static.crates.io.   
# Default format is presently `type,ttl,data` columns, thus omitting a column also works:
$ q @1.1.1.1 static.crates.io --format column +nocolumns ttl

A     151.101.130.137                    
A     151.101.194.137                    
A     151.101.2.137                      
A     151.101.66.137                     
AAAA  2a04:4e42:200::649                 
AAAA  2a04:4e42:400::649                 
AAAA  2a04:4e42:600::649                 
AAAA  2a04:4e42::649                     
CNAME dualstack.k.sni.global.fastly.net. 
CNAME fastly-static.crates.io.

UPDATE: Not long after the output above, the domain has updated DNS records, so that the CNAME resolves to two different CDNs:

# Only showing the newly received records in addition to those above
$ q @1.1.1.1 static.crates.io

static.crates.io. 5m CNAME cloudfront-static.crates.io.
cloudfront-static.crates.io. 5m CNAME d19xqa3lc3clo8.cloudfront.net.
# Results that aren't included in main query (only one CNAME was followed?)
$ q @1.1.1.1 cloudfront-static.crates.io

d19xqa3lc3clo8.cloudfront.net. 1m AAAA 2600:9000:204b:5e00:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 1m AAAA 2600:9000:204b:6e00:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 1m AAAA 2600:9000:204b:800:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 1m AAAA 2600:9000:204b:a400:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 1m AAAA 2600:9000:204b:a600:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 1m AAAA 2600:9000:204b:ae00:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 1m AAAA 2600:9000:204b:dc00:1b:d7a6:ee00:93a1
d19xqa3lc3clo8.cloudfront.net. 1m AAAA 2600:9000:204b:ec00:1b:d7a6:ee00:93a1
cloudfront-static.crates.io. 5m CNAME d19xqa3lc3clo8.cloudfront.net.
d19xqa3lc3clo8.cloudfront.net. 48h NS ns-1118.awsdns-11.org.
d19xqa3lc3clo8.cloudfront.net. 48h NS ns-1570.awsdns-04.co.uk.
d19xqa3lc3clo8.cloudfront.net. 48h NS ns-246.awsdns-30.com.
d19xqa3lc3clo8.cloudfront.net. 48h NS ns-793.awsdns-35.net.
d19xqa3lc3clo8.cloudfront.net. 1m A 65.9.141.105
d19xqa3lc3clo8.cloudfront.net. 1m A 65.9.141.122
d19xqa3lc3clo8.cloudfront.net. 1m A 65.9.141.40
d19xqa3lc3clo8.cloudfront.net. 1m A 65.9.141.65

Due to the above, I noticed if I specifically query the CNAME record instead of the implicit default, I only get one CNAME back:

# The result returned varies (by TTL cache?)
$ q CNAME static.crates.io
static.crates.io. 2m50s CNAME cloudfront-static.crates.io.

$ q @1.1.1.1 CNAME static.crates.io
static.crates.io. 4m23s CNAME fastly-static.crates.io.

Meanwhile, doggo isn't aware of the new cloudfront-static.crates.io record for some reason:

$ doggo @1.1.1.1 static.crates.io

NAME                                    TYPE    CLASS   TTL     ADDRESS                                 NAMESERVER
static.crates.io.                       CNAME   IN      288s    fastly-static.crates.io.                1.1.1.1:53
fastly-static.crates.io.                CNAME   IN      48s     dualstack.k.sni.global.fastly.net.      1.1.1.1:53
dualstack.k.sni.global.fastly.net.      A       IN      18s     151.101.66.137                          1.1.1.1:53
dualstack.k.sni.global.fastly.net.      A       IN      18s     151.101.194.137                         1.1.1.1:53
dualstack.k.sni.global.fastly.net.      A       IN      18s     151.101.2.137                           1.1.1.1:53
dualstack.k.sni.global.fastly.net.      A       IN      18s     151.101.130.137                         1.1.1.1:53

EDIT: Now q results returned aren't showing the extra CNAME for CloudFront... so that seems like it was temporarily configured (perhaps by some automation in CI or active devops 🤷‍♂️ ) ...and it's back again but only with the direct CNAME mapping, not resolving / listing the next CNAME mapping it points to... and now it does. So I have no clue, it's inconsistent.

I had a quick glance over the project source and I think q delegates a bit of the DNS querying to another package, but processes the response, so I'm not sure if it can address the failure I encountered or what might be an increasing amount of queries on repeated requests (at least when CNAME record is present) which may be an upstream bug?

q could still filter/dedupe the response when it receives one to process for display. By omitting the TTL (opt-in) you should have unique records quite easily, or can pipe through sort -u / uniq commands without having to omit the TTL (that can differ as shown above).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@smlx @uggyuggy @polarathene @natesales