Non-Latin sentences downloaded from list are not displayed correctly

[RyckRichards]

By Selena777 - https://tatoeba.org/eng/wall/show_message/24550#message_24550

I can't download any list that isn't in Latin script correctly. There are only strange symbols instead of normal letters. Is it a bug?

By wells https://tatoeba.org/eng/user/profile/wells

Can you add to the bug that having a byte-order mark might fix it? (Needs some testing though.)

[alanfgh]

This is not a bug in Tatoeba, but a consequence of the assumptions that some applications make about the encoding of the files they open. Excel seems to make bad assumptions. However, other applications (for instance, OpenOffice Calc) always allow the user to specify an encoding when opening a CSV file that contains non-ASCII characters, and even Excel allows the user convoluted ways to specify the encoding, though this is not the default behavior.

It's possible that adding a byte-order mark to a CSV file might cause Excel to open the file with the correct encoding, but this would cause problems with other applications that don't strip it out. I think we should not make a change on our side. It might be a good idea to write up on the wiki ways to open a non-ASCII file in various applications, however.