encoding issues

[Haroenv]

At some point the description is parsed as something else than utf-8.

this will make emoji show up as something other than their actual code.

example of that error happening at floating.js

[vvo]

Indeed inside the data it's already badly encoded, so it's an issue in the current indexer

[Haroenv]

A description that has html escape codes like > will also show them escaped once too much (e.g. emoji-images)

[vvo]

I think this is necessary, npm is not doing it but I don't think that's right:

(from https://www.npmjs.com/package/emoji-images)

[Haroenv]

Yep saw that npm is quite wrong on that, but wouldn't a better solution be one that showed it as <img>?

[vvo]

That would be nicer yes, we have seen packages with weird content in it that would display it better this way yes, like yarn:

[vvo]

I don't know if those two things have the same root cause, I would say it's different, one is bad encoding the other is "bad" escaping (dumb one)

[Haroenv]

Issue is happening because backend strings are being ported to UCS-2, but then when highlighted being parsed into utf8 again, which causes problem with multi-byte characters (like emoji) cc @redox

[Haroenv]

close in favour of internal issue