The domain names with the spacing modifier letters are misrecognized

[noraworld]

For example, the following URL is recognized correctly, but it's not on Mastodon.

https://www.ᴳoogle.com

We copy the above link as a string, paste it to Mastodon, and post the status. Then, the link is recognized as a punycode like the following, and we cannot access to the true server.

https://www.xn--oogle-r89a.com

---

• I searched or browsed the repo’s other issues to ensure this is not a duplicate.
• This bug happens on a tagged release and not on master (If you're a user, don't worry about this).

[noraworld]

Perhaps, is this problem the same as #4837?

[nightpool]

Is there a standard that says were not supposed to recognize this as the literal form? In my view this is working as designed—google.com and ᴳoogle.com are two different URLs

[noraworld]

Do you mean that the link "ᴳoogle.com" leads you to a server that is different from google.com? In other words, does your browser recognize the following two links as the same URL in the end?

• https://www.ᴳoogle.com
• https://www.xn--oogle-r89a.com

[unarist]

ᴳ is disallowed in IDNA2008, but mapped to normal character in UTS#46. Browsers behavior would be due to UTS#46.

https://www.unicode.org/cldr/utility/idna.jsp?a=%E1%B4%B3oogle.com
http://unicode.org/faq/idn.html#10

Mastodon uses addressable for Punycode encoding, and addressable uses lbidn which only supports IDNA2003. So using it should be an error due to unassigned code points, but it's allowed now by ALLOW_UNASSIGNED option (c.f. #4496).

Options:

• Don't care: Twitter also encodes it to https://www.xn--oogle-r89a.com.
• Use UTS#46: libidn2 with IDN2_TRANSITIONAL option does this, although I don't know Ruby bindings for libidn2.
• Don't convert by ourselves: GitHub handles like this and also fixes Don't normalize URIs in Unicode NFKC #4837 (I'm not sure about side effects on this)

[noraworld]

Then, can't Mastodon follow the same behavior as the browser one (UTS46) so far?