Use charlock_holmes instead of nkf at FetchLinkCardService

[nullkal]

I suspect NKF is only for Japanese encodings, so we might have to use more generic way for language detection.

After applying this PR, we use charlock_holmes instead of nkf to detect charset.
In addition to it, the code tries to use the value of charset specified in Content-Type HTTP response header, and use the charset detection only when charset is not be specified in HTTP response header or failed to parse the response body due to mis-specifying of the charset in HTTP response header.

Pros

• charlock_holmes uses ICU to detecting charset, so it is reliable for non-Japanese encodings.

Cons

• charlock_holmes depends on ICU, and we need to install icu4c to the system via each environment's package manager.

[akihikodaki]

The spec succeeds even with nkf. Spec the parsed card and you'll find it returns windows-1252 for sjis and ISO-8859-1 for koi8-r. It is a regression and does not solve the problem you expected the change to solve.
charlock_holmes uses ICU as its backend. Show ICU beats NKF to adopt this change.

[akihikodaki]

Also, give response.charset to CharlockHolmes::EncodingDetector.detect as hint_enc instead of a fallback.

[nightpool]

from a more general perspective, i'm very wary about adding new external, non-Gemfile dependencies after the problems many admins had deploying language detection.

[akihikodaki]

I'm not really worrying about the external dependency. People have managed to get cld3-ruby work anyway. (I was expecting more fatal problems at the time, by the way. It was surprising for me that the library, tested only on my environment, worked on lots of computers. A more popular library would certainly work.)

[nullkal]

As @akihikodaki advised I modified to pass response.charset to CharlockHolmes::EncodingDetector as hint_enc. And I also set CharlockHolmes::EncodingDetector#strip_tags to true.

Now koi8-r is detected correctly, but sjis is still detected as windows-1252. Maybe it is trade-off for adopting international charset detector instead of Japanese-encoding-specific implementation.

[akihikodaki]

As @akihikodaki advised I modified to pass response.charset to CharlockHolmes::EncodingDetector as hint_enc. And I also set CharlockHolmes::EncodingDetector#strip_tags to true.

👍

Now koi8-r is detected correctly, but sjis is still detected as windows-1252. Maybe it is trade-off for adopting international charset detector instead of Japanese-encoding-specific implementation.

The string may be too short to infer the encoding. Please test with different texts and evaluate whether its accuracy is acceptable. Also, do not forget to spec the inferred results.

[nullkal]

OK, I improved the test.

[akihikodaki]

Thanks for the series of fixes.

[abcang]

Probably, I think Docker will need to install icu.

[nullkal]

I added icu-dev and checked docker-compose build ran successfully.

[Gargron]

At the Interop Tokyo conference, one of the comments I got was that upgrading Mastodon was too bothersome. I would like to not add system-level dependencies unless absolutely necessary. This is used only for FetchLinkCardService? It could be better to just fail for unknown encodings. Aren't all modern pages served in UTF8?

[nullkal]

Newly created pages should be served in UTF-8 for sure, but there are many websites which are created before UTF-8 became dominant. attempt_opengraph gets information also from obsolete pages, not only from modern pages which contain <meta property="og:*">.

I think users will not suffer from adding a dependency on ICU so much, because It seems Ubuntu Server 16.04 and 14.04 contains ICU by default.

[Gargron]

Paging @Jehops who is maintaining the FreeBSD package for Mastodon - would this be another PITA?

[Jehops]

No. Unless I'm missing something, it would be straightforward to add the dependency to the FreeBSD package.