New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lots of IDN::Idna::IdnaError in puma log #4496

Closed
TheKinrar opened this Issue Aug 2, 2017 · 11 comments

Comments

Projects
None yet
4 participants
@TheKinrar
Collaborator

TheKinrar commented Aug 2, 2017

Users of the instance report 500 error codes and issues with the local timeline only.
Nothing wrong on my side so it seems like it does not impact every user.

In puma's log I have a lot of errors like these:

Aug 02 10:06:06 mastodon bundle[14168]: [d98891f1-1f2d-4549-a211-bbca351fc409] IDN::Idna::IdnaError (String preparation failed (1)):
Aug 02 10:06:06 mastodon bundle[14168]: [d98891f1-1f2d-4549-a211-bbca351fc409]
Aug 02 10:06:06 mastodon bundle[14168]: [d98891f1-1f2d-4549-a211-bbca351fc409] app/lib/formatter.rb:103:in `link_to_url'
Aug 02 10:06:06 mastodon bundle[14168]: [d98891f1-1f2d-4549-a211-bbca351fc409] app/lib/formatter.rb:70:in `block in encode_and_link_urls'
Aug 02 10:06:06 mastodon bundle[14168]: [d98891f1-1f2d-4549-a211-bbca351fc409] app/lib/formatter.rb:93:in `block in rewrite'
Aug 02 10:06:06 mastodon bundle[14168]: [d98891f1-1f2d-4549-a211-bbca351fc409] app/lib/formatter.rb:90:in `each'
Aug 02 10:06:06 mastodon bundle[14168]: [d98891f1-1f2d-4549-a211-bbca351fc409] app/lib/formatter.rb:90:in `reduce'
Aug 02 10:06:06 mastodon bundle[14168]: [d98891f1-1f2d-4549-a211-bbca351fc409] app/lib/formatter.rb:90:in `rewrite'
Aug 02 10:06:06 mastodon bundle[14168]: [d98891f1-1f2d-4549-a211-bbca351fc409] app/lib/formatter.rb:68:in `encode_and_link_urls'
Aug 02 10:06:06 mastodon bundle[14168]: [d98891f1-1f2d-4549-a211-bbca351fc409] app/lib/formatter.rb:48:in `simplified_format'
Aug 02 10:06:06 mastodon bundle[14168]: [d98891f1-1f2d-4549-a211-bbca351fc409] app/serializers/rest/account_serializer.rb:11:in `note'
Aug 02 10:06:06 mastodon bundle[14168]: [d98891f1-1f2d-4549-a211-bbca351fc409] app/controllers/api/v1/notifications_controller.rb:14:in `index'
Aug 02 10:06:50 mastodon bundle[14168]: [ea010a38-5fc4-45a1-b32c-93cfd1d52a6f]
Aug 02 10:06:50 mastodon bundle[14168]: [ea010a38-5fc4-45a1-b32c-93cfd1d52a6f] IDN::Idna::IdnaError (String preparation failed (1)):
Aug 02 10:06:50 mastodon bundle[14168]: [ea010a38-5fc4-45a1-b32c-93cfd1d52a6f]
Aug 02 10:06:50 mastodon bundle[14168]: [ea010a38-5fc4-45a1-b32c-93cfd1d52a6f] app/lib/formatter.rb:103:in `link_to_url'
Aug 02 10:06:50 mastodon bundle[14168]: [ea010a38-5fc4-45a1-b32c-93cfd1d52a6f] app/lib/formatter.rb:70:in `block in encode_and_link_urls'
Aug 02 10:06:50 mastodon bundle[14168]: [ea010a38-5fc4-45a1-b32c-93cfd1d52a6f] app/lib/formatter.rb:93:in `block in rewrite'
Aug 02 10:06:50 mastodon bundle[14168]: [ea010a38-5fc4-45a1-b32c-93cfd1d52a6f] app/lib/formatter.rb:90:in `each'
Aug 02 10:06:50 mastodon bundle[14168]: [ea010a38-5fc4-45a1-b32c-93cfd1d52a6f] app/lib/formatter.rb:90:in `reduce'
Aug 02 10:06:50 mastodon bundle[14168]: [ea010a38-5fc4-45a1-b32c-93cfd1d52a6f] app/lib/formatter.rb:90:in `rewrite'
Aug 02 10:06:50 mastodon bundle[14168]: [ea010a38-5fc4-45a1-b32c-93cfd1d52a6f] app/lib/formatter.rb:68:in `encode_and_link_urls'
Aug 02 10:06:50 mastodon bundle[14168]: [ea010a38-5fc4-45a1-b32c-93cfd1d52a6f] app/lib/formatter.rb:48:in `simplified_format'
Aug 02 10:06:50 mastodon bundle[14168]: [ea010a38-5fc4-45a1-b32c-93cfd1d52a6f] app/serializers/rest/account_serializer.rb:11:in `note'
Aug 02 10:06:50 mastodon bundle[14168]: [ea010a38-5fc4-45a1-b32c-93cfd1d52a6f] app/controllers/api/v1/timelines/public_controller.rb:10:in `show'
Aug 02 10:07:13 mastodon bundle[14168]: [2e6bf225-5fa5-4249-849d-f74f89580169]
Aug 02 10:07:13 mastodon bundle[14168]: [2e6bf225-5fa5-4249-849d-f74f89580169] IDN::Idna::IdnaError (String preparation failed (1)):
Aug 02 10:07:13 mastodon bundle[14168]: [2e6bf225-5fa5-4249-849d-f74f89580169]
Aug 02 10:07:13 mastodon bundle[14168]: [2e6bf225-5fa5-4249-849d-f74f89580169] app/lib/formatter.rb:103:in `link_to_url'
Aug 02 10:07:13 mastodon bundle[14168]: [2e6bf225-5fa5-4249-849d-f74f89580169] app/lib/formatter.rb:70:in `block in encode_and_link_urls'
Aug 02 10:07:13 mastodon bundle[14168]: [2e6bf225-5fa5-4249-849d-f74f89580169] app/lib/formatter.rb:93:in `block in rewrite'
Aug 02 10:07:13 mastodon bundle[14168]: [2e6bf225-5fa5-4249-849d-f74f89580169] app/lib/formatter.rb:90:in `each'
Aug 02 10:07:13 mastodon bundle[14168]: [2e6bf225-5fa5-4249-849d-f74f89580169] app/lib/formatter.rb:90:in `reduce'
Aug 02 10:07:13 mastodon bundle[14168]: [2e6bf225-5fa5-4249-849d-f74f89580169] app/lib/formatter.rb:90:in `rewrite'
Aug 02 10:07:13 mastodon bundle[14168]: [2e6bf225-5fa5-4249-849d-f74f89580169] app/lib/formatter.rb:68:in `encode_and_link_urls'
Aug 02 10:07:13 mastodon bundle[14168]: [2e6bf225-5fa5-4249-849d-f74f89580169] app/lib/formatter.rb:48:in `simplified_format'
Aug 02 10:07:13 mastodon bundle[14168]: [2e6bf225-5fa5-4249-849d-f74f89580169] app/serializers/rest/account_serializer.rb:11:in `note'
Aug 02 10:07:13 mastodon bundle[14168]: [2e6bf225-5fa5-4249-849d-f74f89580169] app/controllers/api/v1/timelines/public_controller.rb:10:in `show'

  • I searched or browsed the repo’s other issues to ensure this is not a duplicate.
  • This bug happens on a tagged release and not on master (If you're a user, don't worry about this).
@TheKinrar

This comment has been minimized.

Show comment
Hide comment
@TheKinrar

TheKinrar Aug 2, 2017

Collaborator

I've made a temporary "fix" by rescuing IDN::Idna::IdnaError at https://github.com/tootsuite/mastodon/blob/v1.5.0/app/lib/formatter.rb#L107

  def link_to_url(entity)
    normalized_url = Addressable::URI.parse(entity[:url]).normalize
    html_attrs     = { target: '_blank', rel: 'nofollow noopener' }

    Twitter::Autolink.send(:link_to_text, entity, link_html(entity[:url]), normalized_url, html_attrs)
  rescue Addressable::URI::InvalidURIError, IDN::Idna::IdnaError
    encode(entity[:url])
  end

Local timeline loads now but I have still no idea what's causing this error

Collaborator

TheKinrar commented Aug 2, 2017

I've made a temporary "fix" by rescuing IDN::Idna::IdnaError at https://github.com/tootsuite/mastodon/blob/v1.5.0/app/lib/formatter.rb#L107

  def link_to_url(entity)
    normalized_url = Addressable::URI.parse(entity[:url]).normalize
    html_attrs     = { target: '_blank', rel: 'nofollow noopener' }

    Twitter::Autolink.send(:link_to_text, entity, link_html(entity[:url]), normalized_url, html_attrs)
  rescue Addressable::URI::InvalidURIError, IDN::Idna::IdnaError
    encode(entity[:url])
  end

Local timeline loads now but I have still no idea what's causing this error

@ykzts

This comment has been minimized.

Show comment
Hide comment
@ykzts

ykzts Aug 2, 2017

Collaborator

Do you know what URL causes that problem?

Also, I think your solution is very good. Please send a pull request if you do not mind.

Collaborator

ykzts commented Aug 2, 2017

Do you know what URL causes that problem?

Also, I think your solution is very good. Please send a pull request if you do not mind.

@TheKinrar

This comment has been minimized.

Show comment
Hide comment
@TheKinrar

TheKinrar Aug 2, 2017

Collaborator

http://🌹🌹🌹.ws is one of the URL causing problem

So I suppose this is because of the characters inside?
I'm okay with sending a PR but I'm not sure this cannot be fixed by an other, better, way.

Collaborator

TheKinrar commented Aug 2, 2017

http://🌹🌹🌹.ws is one of the URL causing problem

So I suppose this is because of the characters inside?
I'm okay with sending a PR but I'm not sure this cannot be fixed by an other, better, way.

@ykzts

This comment has been minimized.

Show comment
Hide comment
@ykzts

ykzts Aug 2, 2017

Collaborator

I see. It seems that this is happening because libidn does not support Unicode containing emoji.

Collaborator

ykzts commented Aug 2, 2017

I see. It seems that this is happening because libidn does not support Unicode containing emoji.

@nullkal

This comment has been minimized.

Show comment
Hide comment
@nullkal

nullkal Aug 2, 2017

Collaborator

I tried to encode 🌹🌹🌹.ws using idn command and it throws an error during string preparation, too. So, it's definitely libidn's problem.

$ idn
libidn 1.33
Copyright 2002-2015 Simon Josefsson.
GNU Libidn is free software with ABSOLUTELY NO WARRANTY.  For more
information about these matters, see <http://www.gnu.org/licenses/>.
Type each input string on a line by itself, terminated by a newline character.
🌹🌹🌹.ws
idn: idna_to_ascii_4z: String preparation failed
Collaborator

nullkal commented Aug 2, 2017

I tried to encode 🌹🌹🌹.ws using idn command and it throws an error during string preparation, too. So, it's definitely libidn's problem.

$ idn
libidn 1.33
Copyright 2002-2015 Simon Josefsson.
GNU Libidn is free software with ABSOLUTELY NO WARRANTY.  For more
information about these matters, see <http://www.gnu.org/licenses/>.
Type each input string on a line by itself, terminated by a newline character.
🌹🌹🌹.ws
idn: idna_to_ascii_4z: String preparation failed
@TheKinrar

This comment has been minimized.

Show comment
Hide comment
@TheKinrar

TheKinrar Aug 2, 2017

Collaborator

So, should I make a PR anyway if this is not caused by Mastodon itself?

Collaborator

TheKinrar commented Aug 2, 2017

So, should I make a PR anyway if this is not caused by Mastodon itself?

@ykzts ykzts added the bug label Aug 2, 2017

@nullkal

This comment has been minimized.

Show comment
Hide comment
@nullkal

nullkal Aug 2, 2017

Collaborator

With --allow-unassigned I can successfully encode 🌹🌹🌹.ws.

$ idn 🌹🌹🌹.ws --allow-unassigned
xn--2h8haa.ws
Collaborator

nullkal commented Aug 2, 2017

With --allow-unassigned I can successfully encode 🌹🌹🌹.ws.

$ idn 🌹🌹🌹.ws --allow-unassigned
xn--2h8haa.ws
@ykzts

This comment has been minimized.

Show comment
Hide comment
@ykzts

ykzts Aug 2, 2017

Collaborator

This bug is starting from #4212.

Collaborator

ykzts commented Aug 2, 2017

This bug is starting from #4212.

@TheKinrar

This comment has been minimized.

Show comment
Hide comment
@TheKinrar

TheKinrar Aug 2, 2017

Collaborator

🌹🌹🌹.ws should not be a valid domain name in fact as it contains unassigned code points.

I'm not sure we should care about handling invalid domain names.
On the other part, the strange thing is I can resolve this domain. But it should not be valid.

Collaborator

TheKinrar commented Aug 2, 2017

🌹🌹🌹.ws should not be a valid domain name in fact as it contains unassigned code points.

I'm not sure we should care about handling invalid domain names.
On the other part, the strange thing is I can resolve this domain. But it should not be valid.

@nullkal

This comment has been minimized.

Show comment
Hide comment
@nullkal

nullkal Aug 2, 2017

Collaborator

It seems libidn 1.x, which ruby-idn uses, treats characters which is not specified in RFC3454, as unassigned code points. The most ideal solution is to use libidn2, which regards IDNA2008, instead of libidn 1.x.

Collaborator

nullkal commented Aug 2, 2017

It seems libidn 1.x, which ruby-idn uses, treats characters which is not specified in RFC3454, as unassigned code points. The most ideal solution is to use libidn2, which regards IDNA2008, instead of libidn 1.x.

@Gargron Gargron closed this in #4499 Aug 2, 2017

Gargron added a commit that referenced this issue Aug 2, 2017

mosaxiv added a commit to mosaxiv/mastodon that referenced this issue Aug 6, 2017

@unarist

This comment has been minimized.

Show comment
Hide comment
@unarist

unarist Aug 6, 2017

Collaborator

Strictly speaking, symbols and emojis including 🌹 are "disallowed" in IDNA2008, but allowed in TR46 for compatibility during the transition phase.

http://unicode.org/faq/idn.html

So IDN2_TRANSITIONAL flag will be needed even if it uses libidn2.

Collaborator

unarist commented Aug 6, 2017

Strictly speaking, symbols and emojis including 🌹 are "disallowed" in IDNA2008, but allowed in TR46 for compatibility during the transition phase.

http://unicode.org/faq/idn.html

So IDN2_TRANSITIONAL flag will be needed even if it uses libidn2.

wxcafe added a commit to wxcafe/mastodon that referenced this issue Aug 7, 2017

wxcafe added a commit to wxcafe/mastodon that referenced this issue Aug 7, 2017

YaQ00 added a commit to YaQ00/mastodon that referenced this issue Sep 5, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment