Handle unicode/punycode

[sshine]

TL;DR: Is there any interest in conversion to punycode in the whois program?

---

The lead-up to this question:

When I whois ål.dk, DK-Hostmaster's WHOIS server sends back

No entries found for the selected source.

even though their web-interface says that the domain is taken.

There are three ways I should be able to get a positive result here:

1. Had I converted to punycode, whois xn--l-1fa.dk does work.

2. Had I used ISO-8859-15, e.g.

   $ whois "$(echo -n "ål.dk" | iconv -f UTF-8 -t ISO-8859-15)"
   

   that would have worked as well.

3. (Doesn't work, notified.) Following DK-Hostmaster's own UTF-8 recommendations,

   # Assuming the 'å' is made with UTF-8, this should work but doesn't
   $ whois " --charset=utf8 ål.dk"
   

Now, I really don't know if --charset is common in WHOIS servers. In fact, since .dk is one of the very few TLDs that have any special rules in whois (--show-handles), I actually doubt it. I assume we're in absolutely-no-standards-land and any general support here is futile.

So without having done extensive surveys (we could), an alternative to sending arbitrary unicodey bytestrings and hope for the best, one could punycode them. This is much easier in my Haskell library since I can assume UTF-8, and not so easy for the present whois program, since we also have to figure out the calling terminal's encoding first (and possibly forego conversion if we can't.)

Some TLDs limit allowable non-standard extended characters which makes guessing without knowing the encoding easier. For example, if .dk only allows æøåöäüé, I doubt there is any overlap in the way those letters are encoded. Still, I'd prefer a generalised method over any TLD-specific knowledge, since there are so many TLDs with special behavior to keep track of.

[rfc1036]

whois has supported IDN since 2003. I am quite sure that .dk used to work and indeed the code does contain "special rules" for it, so maybe they changed something on their side.

Feel free to investigate, or else I will get to this later. Check the source and use "whois --verbose".

[sshine]

I'll investigate. Thanks for your quick response.

[rfc1036]

Works for me. Probably you compiled whois without IDN support.

[sshine]

Yes, apparently Ubuntu 18.04 does. The one I have available on Debian 10 is fine.