UTF-8 Java Properties upload broken

[Nick-kel]

If I upload something to a language in Weblate, the source language works very well in UTF-8 (e.g. a § is displayed correctly). But if I upload a translation to a "normal" language, it isn't in UTF-8 and in front of a § there's a weird character:

I'm using the newest version of Weblate

[github-actions]

This issue looks more like a support question than an issue. We strive to answer these reasonably fast, but purchasing the support subscription is not only more responsible and faster for your business but also makes Weblate stronger. In case your question is already answered, making a donation is the right way to say thank you!

[nijel]

Maybe the existing file is iso-8859-1 encoded? What file format do you use? How do you upload?

[Nick-kel]

The existing file is UTF-8 encoded and I'm using a Java Properties (UTF-8) format. I'm uploading the translation using the "Files" > "Upload translation" button in a language.

[nijel]

I mean which upload method do you use...

[Nick-kel]

The upload method is "add as translation"

[nijel]

Can you please share the uploaded file? It seems that it's detected as iso-8859-1 instead of UTF-8. Maybe it has encoding error or there is something wrong with the detection.

[Nick-kel]

@nijel https://www.file-upload.net/download-14404874/test.properties.html

[nijel]

There is no unicode char in there which could break...

[Nick-kel]

@nijel ah sorry: https://www.file-upload.net/download-14405120/en_US.properties.html

[nijel]

That one works fine in my tests. The content looks like something (mis-)detectcs it as iso-8859-1. It might be a bug in chardet on the specific file you are using.

[Nick-kel]

@nijel how to fix this?

[nijel]

How to reproduce that?

[GoneTone]

I also encountered the same problem. When uploading a new Chinese translation, it will be garbled.

File is UTF-8 encoded.

[Nick-kel]

@nijel I don't know ... I only uploaded the translations, these are my settings:

In the source language there's no problem with the encoding, only in other languages

[nijel]

The most likely suspect is still chardet here as the detection is always in place. Unfortunately, the file in #5083 (comment) doesn't show it. Can reproduce it with that file? Having the actual uploaded file to reproduce this would help...

[GoneTone]

@nijel Here contains the demo video and translation files:
https://drive.google.com/drive/folders/1pY2AEZgkvsgW-DOVRN9tN5u4Z_fd99Vn?usp=sharing

[nijel]

@GoneTone I can reproduce this if the file format is configured to ISO-8859-1, and it's expected in this case - the replace upload method does no processing of the file. When I switch file format in Weblate to UTF-8, it is shown correctly and the replace upload works fine. What still seems to be broken is regular upload though, what probably matches behavior @Nick-kel was describing initially.

[github-actions]

Thank you for your report, the issue you have reported has just been fixed.

• In case you see a problem with the fix, please comment on this issue.
• In case you see a similar problem, please open a separate issue.
• If you are happy with the outcome, don’t hesitate to support Weblate by making a donation.