Lots of IDN::Idna::IdnaError in puma log #4496

TheKinrar · Aug 2, 2017

Users of the instance report 500 error codes and issues with the local timeline only.
Nothing wrong on my side so it seems like it does not impact every user.

In puma's log I have a lot of errors like these:

Aug 02 10:06:06 mastodon bundle[14168]: [d98891f1-1f2d-4549-a211-bbca351fc409] IDN::Idna::IdnaError (String preparation failed (1)):
Aug 02 10:06:06 mastodon bundle[14168]: [d98891f1-1f2d-4549-a211-bbca351fc409]
Aug 02 10:06:06 mastodon bundle[14168]: [d98891f1-1f2d-4549-a211-bbca351fc409] app/lib/formatter.rb:103:in `link_to_url'
Aug 02 10:06:06 mastodon bundle[14168]: [d98891f1-1f2d-4549-a211-bbca351fc409] app/lib/formatter.rb:70:in `block in encode_and_link_urls'
Aug 02 10:06:06 mastodon bundle[14168]: [d98891f1-1f2d-4549-a211-bbca351fc409] app/lib/formatter.rb:93:in `block in rewrite'
Aug 02 10:06:06 mastodon bundle[14168]: [d98891f1-1f2d-4549-a211-bbca351fc409] app/lib/formatter.rb:90:in `each'
Aug 02 10:06:06 mastodon bundle[14168]: [d98891f1-1f2d-4549-a211-bbca351fc409] app/lib/formatter.rb:90:in `reduce'
Aug 02 10:06:06 mastodon bundle[14168]: [d98891f1-1f2d-4549-a211-bbca351fc409] app/lib/formatter.rb:90:in `rewrite'
Aug 02 10:06:06 mastodon bundle[14168]: [d98891f1-1f2d-4549-a211-bbca351fc409] app/lib/formatter.rb:68:in `encode_and_link_urls'
Aug 02 10:06:06 mastodon bundle[14168]: [d98891f1-1f2d-4549-a211-bbca351fc409] app/lib/formatter.rb:48:in `simplified_format'
Aug 02 10:06:06 mastodon bundle[14168]: [d98891f1-1f2d-4549-a211-bbca351fc409] app/serializers/rest/account_serializer.rb:11:in `note'
Aug 02 10:06:06 mastodon bundle[14168]: [d98891f1-1f2d-4549-a211-bbca351fc409] app/controllers/api/v1/notifications_controller.rb:14:in `index'
Aug 02 10:06:50 mastodon bundle[14168]: [ea010a38-5fc4-45a1-b32c-93cfd1d52a6f]
Aug 02 10:06:50 mastodon bundle[14168]: [ea010a38-5fc4-45a1-b32c-93cfd1d52a6f] IDN::Idna::IdnaError (String preparation failed (1)):
Aug 02 10:06:50 mastodon bundle[14168]: [ea010a38-5fc4-45a1-b32c-93cfd1d52a6f]
Aug 02 10:06:50 mastodon bundle[14168]: [ea010a38-5fc4-45a1-b32c-93cfd1d52a6f] app/lib/formatter.rb:103:in `link_to_url'
Aug 02 10:06:50 mastodon bundle[14168]: [ea010a38-5fc4-45a1-b32c-93cfd1d52a6f] app/lib/formatter.rb:70:in `block in encode_and_link_urls'
Aug 02 10:06:50 mastodon bundle[14168]: [ea010a38-5fc4-45a1-b32c-93cfd1d52a6f] app/lib/formatter.rb:93:in `block in rewrite'
Aug 02 10:06:50 mastodon bundle[14168]: [ea010a38-5fc4-45a1-b32c-93cfd1d52a6f] app/lib/formatter.rb:90:in `each'
Aug 02 10:06:50 mastodon bundle[14168]: [ea010a38-5fc4-45a1-b32c-93cfd1d52a6f] app/lib/formatter.rb:90:in `reduce'
Aug 02 10:06:50 mastodon bundle[14168]: [ea010a38-5fc4-45a1-b32c-93cfd1d52a6f] app/lib/formatter.rb:90:in `rewrite'
Aug 02 10:06:50 mastodon bundle[14168]: [ea010a38-5fc4-45a1-b32c-93cfd1d52a6f] app/lib/formatter.rb:68:in `encode_and_link_urls'
Aug 02 10:06:50 mastodon bundle[14168]: [ea010a38-5fc4-45a1-b32c-93cfd1d52a6f] app/lib/formatter.rb:48:in `simplified_format'
Aug 02 10:06:50 mastodon bundle[14168]: [ea010a38-5fc4-45a1-b32c-93cfd1d52a6f] app/serializers/rest/account_serializer.rb:11:in `note'
Aug 02 10:06:50 mastodon bundle[14168]: [ea010a38-5fc4-45a1-b32c-93cfd1d52a6f] app/controllers/api/v1/timelines/public_controller.rb:10:in `show'
Aug 02 10:07:13 mastodon bundle[14168]: [2e6bf225-5fa5-4249-849d-f74f89580169]
Aug 02 10:07:13 mastodon bundle[14168]: [2e6bf225-5fa5-4249-849d-f74f89580169] IDN::Idna::IdnaError (String preparation failed (1)):
Aug 02 10:07:13 mastodon bundle[14168]: [2e6bf225-5fa5-4249-849d-f74f89580169]
Aug 02 10:07:13 mastodon bundle[14168]: [2e6bf225-5fa5-4249-849d-f74f89580169] app/lib/formatter.rb:103:in `link_to_url'
Aug 02 10:07:13 mastodon bundle[14168]: [2e6bf225-5fa5-4249-849d-f74f89580169] app/lib/formatter.rb:70:in `block in encode_and_link_urls'
Aug 02 10:07:13 mastodon bundle[14168]: [2e6bf225-5fa5-4249-849d-f74f89580169] app/lib/formatter.rb:93:in `block in rewrite'
Aug 02 10:07:13 mastodon bundle[14168]: [2e6bf225-5fa5-4249-849d-f74f89580169] app/lib/formatter.rb:90:in `each'
Aug 02 10:07:13 mastodon bundle[14168]: [2e6bf225-5fa5-4249-849d-f74f89580169] app/lib/formatter.rb:90:in `reduce'
Aug 02 10:07:13 mastodon bundle[14168]: [2e6bf225-5fa5-4249-849d-f74f89580169] app/lib/formatter.rb:90:in `rewrite'
Aug 02 10:07:13 mastodon bundle[14168]: [2e6bf225-5fa5-4249-849d-f74f89580169] app/lib/formatter.rb:68:in `encode_and_link_urls'
Aug 02 10:07:13 mastodon bundle[14168]: [2e6bf225-5fa5-4249-849d-f74f89580169] app/lib/formatter.rb:48:in `simplified_format'
Aug 02 10:07:13 mastodon bundle[14168]: [2e6bf225-5fa5-4249-849d-f74f89580169] app/serializers/rest/account_serializer.rb:11:in `note'
Aug 02 10:07:13 mastodon bundle[14168]: [2e6bf225-5fa5-4249-849d-f74f89580169] app/controllers/api/v1/timelines/public_controller.rb:10:in `show'

I searched or browsed the repo’s other issues to ensure this is not a duplicate.
This bug happens on a tagged release and not on master (If you're a user, don't worry about this).

I've made a temporary "fix" by rescuing IDN::Idna::IdnaError at https://github.com/tootsuite/mastodon/blob/v1.5.0/app/lib/formatter.rb#L107

  def link_to_url(entity)
    normalized_url = Addressable::URI.parse(entity[:url]).normalize
    html_attrs     = { target: '_blank', rel: 'nofollow noopener' }

    Twitter::Autolink.send(:link_to_text, entity, link_html(entity[:url]), normalized_url, html_attrs)
  rescue Addressable::URI::InvalidURIError, IDN::Idna::IdnaError
    encode(entity[:url])
  end

Local timeline loads now but I have still no idea what's causing this error

TheKinrar · Aug 2, 2017

I've made a temporary "fix" by rescuing IDN::Idna::IdnaError at https://github.com/tootsuite/mastodon/blob/v1.5.0/app/lib/formatter.rb#L107

  def link_to_url(entity)
    normalized_url = Addressable::URI.parse(entity[:url]).normalize
    html_attrs     = { target: '_blank', rel: 'nofollow noopener' }

    Twitter::Autolink.send(:link_to_text, entity, link_html(entity[:url]), normalized_url, html_attrs)
  rescue Addressable::URI::InvalidURIError, IDN::Idna::IdnaError
    encode(entity[:url])
  end

Local timeline loads now but I have still no idea what's causing this error

Do you know what URL causes that problem?

Also, I think your solution is very good. Please send a pull request if you do not mind.

ykzts · Aug 2, 2017

Do you know what URL causes that problem?

Also, I think your solution is very good. Please send a pull request if you do not mind.

http://🌹🌹🌹.ws is one of the URL causing problem

So I suppose this is because of the characters inside?
I'm okay with sending a PR but I'm not sure this cannot be fixed by an other, better, way.

TheKinrar · Aug 2, 2017

http://🌹🌹🌹.ws is one of the URL causing problem

So I suppose this is because of the characters inside?
I'm okay with sending a PR but I'm not sure this cannot be fixed by an other, better, way.

I see. It seems that this is happening because libidn does not support Unicode containing emoji.

ykzts · Aug 2, 2017

I see. It seems that this is happening because libidn does not support Unicode containing emoji.

I tried to encode 🌹🌹🌹.ws using idn command and it throws an error during string preparation, too. So, it's definitely libidn's problem.

$ idn
libidn 1.33
Copyright 2002-2015 Simon Josefsson.
GNU Libidn is free software with ABSOLUTELY NO WARRANTY.  For more
information about these matters, see <http://www.gnu.org/licenses/>.
Type each input string on a line by itself, terminated by a newline character.
🌹🌹🌹.ws
idn: idna_to_ascii_4z: String preparation failed

nullkal · Aug 2, 2017

I tried to encode 🌹🌹🌹.ws using idn command and it throws an error during string preparation, too. So, it's definitely libidn's problem.

$ idn
libidn 1.33
Copyright 2002-2015 Simon Josefsson.
GNU Libidn is free software with ABSOLUTELY NO WARRANTY.  For more
information about these matters, see <http://www.gnu.org/licenses/>.
Type each input string on a line by itself, terminated by a newline character.
🌹🌹🌹.ws
idn: idna_to_ascii_4z: String preparation failed

So, should I make a PR anyway if this is not caused by Mastodon itself?

TheKinrar · Aug 2, 2017

So, should I make a PR anyway if this is not caused by Mastodon itself?

With --allow-unassigned I can successfully encode 🌹🌹🌹.ws.

$ idn 🌹🌹🌹.ws --allow-unassigned
xn--2h8haa.ws

nullkal · Aug 2, 2017

With --allow-unassigned I can successfully encode 🌹🌹🌹.ws.

$ idn 🌹🌹🌹.ws --allow-unassigned
xn--2h8haa.ws

This bug is starting from #4212.

ykzts · Aug 2, 2017

This bug is starting from #4212.

🌹🌹🌹.ws should not be a valid domain name in fact as it contains unassigned code points.

I'm not sure we should care about handling invalid domain names.
On the other part, the strange thing is I can resolve this domain. But it should not be valid.

TheKinrar · Aug 2, 2017

🌹🌹🌹.ws should not be a valid domain name in fact as it contains unassigned code points.

I'm not sure we should care about handling invalid domain names.
On the other part, the strange thing is I can resolve this domain. But it should not be valid.

It seems libidn 1.x, which ruby-idn uses, treats characters which is not specified in RFC3454, as unassigned code points. The most ideal solution is to use libidn2, which regards IDNA2008, instead of libidn 1.x.

nullkal · Aug 2, 2017

It seems libidn 1.x, which ruby-idn uses, treats characters which is not specified in RFC3454, as unassigned code points. The most ideal solution is to use libidn2, which regards IDNA2008, instead of libidn 1.x.

Strictly speaking, symbols and emojis including 🌹 are "disallowed" in IDNA2008, but allowed in TR46 for compatibility during the transition phase.

http://unicode.org/faq/idn.html

So IDN2_TRANSITIONAL flag will be needed even if it uses libidn2.

unarist · Aug 6, 2017

Strictly speaking, symbols and emojis including 🌹 are "disallowed" in IDNA2008, but allowed in TR46 for compatibility during the transition phase.

http://unicode.org/faq/idn.html

So IDN2_TRANSITIONAL flag will be needed even if it uses libidn2.

ykzts added the bug label Aug 2, 2017

nullkal referenced this issue in sporkmonger/addressable Aug 2, 2017
Merged
Specify ALLOW_UNASSIGNED flags in Addressable::IDNA.to_ascii/to_unicode #266

TheKinrar added a commit to TheKinrar/mastodon that referenced this issue Aug 2, 2017

Don't normalize invalid domain names
Fixes #4496

6d301e2

TheKinrar referenced this issue Aug 2, 2017
Merged
Don't throw while normalizing invalid domain names #4499

Gargron closed this in #4499 Aug 2, 2017

Gargron added a commit that referenced this issue Aug 2, 2017

Don't normalize invalid domain names (#4499)
Fixes #4496

9ba7d52

mosaxiv added a commit to mosaxiv/mastodon that referenced this issue Aug 6, 2017

Don't normalize invalid domain names (#4499)
Fixes #4496

a096ab6

wxcafe added a commit to wxcafe/mastodon that referenced this issue Aug 7, 2017

Don't normalize invalid domain names (#4499)
Fixes #4496

4a00714

wxcafe added a commit to wxcafe/mastodon that referenced this issue Aug 7, 2017

Don't normalize invalid domain names (#4499)
Fixes #4496

e37d70a

ykzts referenced this issue Aug 25, 2017
Merged
Update addressable to version 2.5.2 #4686

YaQ00 added a commit to YaQ00/mastodon that referenced this issue Sep 5, 2017

Don't normalize invalid domain names (#4499)
Fixes #4496

aec8f93

unarist referenced this issue Nov 4, 2017
Open
The domain names with the spacing modifier letters are misrecognized #5585
2 of 2 tasks complete

tootsuite/mastodon

Join GitHub today

Lots of IDN::Idna::IdnaError in puma log #4496

Comments

TheKinrar commented Aug 2, 2017 • edited Edited 1 time TheKinrar edited Aug 2, 2017 (most recent)

This comment has been minimized.

TheKinrar commented Aug 2, 2017

This comment has been minimized.

ykzts commented Aug 2, 2017 • edited Edited 1 time ykzts edited Aug 2, 2017 (most recent)

This comment has been minimized.

TheKinrar commented Aug 2, 2017 • edited Edited 1 time TheKinrar edited Aug 2, 2017 (most recent)

This comment has been minimized.

ykzts commented Aug 2, 2017

This comment has been minimized.

nullkal commented Aug 2, 2017

This comment has been minimized.

TheKinrar commented Aug 2, 2017

ykzts added the bug label Aug 2, 2017

This comment has been minimized.

nullkal commented Aug 2, 2017

This comment has been minimized.

ykzts commented Aug 2, 2017

This comment has been minimized.

TheKinrar commented Aug 2, 2017

nullkal referenced this issue in sporkmonger/addressable Aug 2, 2017

TheKinrar added a commit to TheKinrar/mastodon that referenced this issue Aug 2, 2017

TheKinrar referenced this issue Aug 2, 2017

This comment has been minimized.

nullkal commented Aug 2, 2017

Gargron closed this in #4499 Aug 2, 2017

Gargron added a commit that referenced this issue Aug 2, 2017

mosaxiv added a commit to mosaxiv/mastodon that referenced this issue Aug 6, 2017

This comment has been minimized.

unarist commented Aug 6, 2017

wxcafe added a commit to wxcafe/mastodon that referenced this issue Aug 7, 2017

wxcafe added a commit to wxcafe/mastodon that referenced this issue Aug 7, 2017

ykzts referenced this issue Aug 25, 2017

YaQ00 added a commit to YaQ00/mastodon that referenced this issue Sep 5, 2017

unarist referenced this issue Nov 4, 2017

TheKinrar commented Aug 2, 2017 •

edited

Edited 1 time

TheKinrar edited Aug 2, 2017 (most recent)

ykzts commented Aug 2, 2017 •

edited

Edited 1 time

ykzts edited Aug 2, 2017 (most recent)

TheKinrar commented Aug 2, 2017 •

edited

Edited 1 time

TheKinrar edited Aug 2, 2017 (most recent)