nic.storage whoisServer issue

[case]

Lately, the daily autopull PRs tend to include a change for the .storage "whoisServer" toggling back and forth between whois.nic.storage and whois.afilias-srs.net.

Here's one, for example: https://github.com/zonedb/zonedb/pull/574/files.

[case]

It switched back again in today's autopull PR.

[case]

@ydnar worked on this a bit in #565.

[philpennock]

Ultimately, whois-servers.net seems to be run by CenterGate LLC and is entirely unofficial, just widely used. Since nothing in zonedb repo hard-codes afilias-srs as a fallback, it must be coming from https://github.com/weppos/whois/raw/master/data/tld.json.

When I check, whois-servers.net seems to blacklist my home IP rather rapidly. They don't seem to appreciate someone looking up every record in the domain.

So I think we're just hitting rate-limits and falling back to the Ruby data when we don't get a DNS response. And the Ruby data is so very stale that the server returned from there doesn't work.

This needs a rethink as to how often we want to revalidate working entries, whether we want to switch sources when one query fails, etc. Is it worth reaching out to CenterGate and asking them what they consider reasonable and looking for other options?

For instance, https://whois.icann.org/en/lookup?name=storage shows that ICANN have the whois server in their records. Can we get this direct from ICANN?

[philpennock]

Looks like IANA has canonically correct information: https://www.iana.org/domains/root/db/storage.html

[philpennock]

whois -h whois.iana.org storage is nice and fast, but I don't know about abuse limits, might be worth asking IANA?

[philpennock]

At this point, I'm thinking a one-off run through whois.iana.org and then in future, only re-query that server for any new zones or zones where the set of DNS servers has changed in any way.

Otherwise, we assume that if the DNS servers are unchanged, then whois server probably is too.

The code to do a one-off run would still be in the repo, so could be used again if we think it needed.

I can't see this dataset in the IANA rsync server modules, or I'd say to just use that. :(

Thoughts?

[philpennock]

https://data.iana.org/TLD/tlds-alpha-by-domain.txt might also be worth comparing to the ICANN data.

[case]

@ydnar how would you like to proceed here?

[ydnar]

We could use IANA as canonical, and fall back to whois-servers.net and/or ruby-whois as a fallback for when we don’t have a whois server.

[case]

Haven't seen this in a while, so closing this.