IDN support

[Johann150]

IDN (International Domain Name) is not yet implemented in parsing domain names.

However, there is the concern that it can break existing posts for example the following could be parsed incorrectly:

@syuilo@misskey.ioありがとう 

An idea would be to recognize IDNs only if there is a space after it, but it might not always work.

I am not sure how to best solve this, but I think not implementing IDNs is a very inelegant solution that needs to be fixed. I have already seen a few people that were irritated by the missing proper support for IDNs.

[Johann150]

see also misskey-dev/misskey#5826

[marihachi]

今の仕様ではメンションのホスト名には非ASCII文字は直接使用できません。
非ASCII文字を表現するにはPunycodeに変換することが必要になります。

ホスト名の部分をUnicodeでも記述できるように変更するということですかね？

[Johann150]

Yes, when parsing Unicode should be understood. But I think it would make sense if the output contains punycoded domains if necessary.
So, for example:

@somebody@みすきー.テスト

⇓

MENTION('somebody', 'xn--w8jxa7itv.xn--zckzah', '@somebody@みすきー.テスト')

[Johann150]

I looked to Mastodon how they are doing it, and these are the regular expressions they use to recognize a mention:

USERNAME_RE   = /[a-z0-9_]+([a-z0-9_\.-]+[a-z0-9_]+)?/i
MENTION_RE    = /(?<=^|[^\/[:word:]])@((#{USERNAME_RE})(?:@[[:word:]\.\-]+[[:word:]]+)?)/i

https://github.com/mastodon/mastodon/blob/1114935e6486caaae6e4ba98b51ab803317acb03/app/models/account.rb#L61-L62

Since pegjs does not support \p{L} or \p{N} which would be needed to represent the same meaning as [:word:] in Ruby, it might be simpler to handle mentions with IDNs before the parser starts and convert them into punycoded domains. Then the parser itself would not have to be changed.

Mastodon's regular expression for mentions translates to Javascript as (replacing [:word:] with the Javascript equivalent \p{L}\p{N}_, adding the u Unicode flag and removing unnecessary parentheses)

/(?<=^|[^\/\p{L}\p{N}_])@[a-z0-9_]+(?:[a-z0-9_\.-]+[a-z0-9_]+)?(?:@[\p{L}\p{N}_\.-]+[\p{L}\p{N}_]+)?/iu

[marihachi]

https://github.com/mathiasbynens/idn-allowed-code-points-regex
IDNAの実装見つけた