Issues with file encoding

[tsaglam]

JPlag is currently built for UTF-8 encoding and may reject submissions that have improper encoding (e.g. the Java language module and ANSI files). Maybe we can come up with a more convenient system that is more tolerant.

• First priority is to find and test a solution for the java language.
• After that, we could think of language-independent solutions.

We could use heuristics like https://github.com/albfernandez/juniversalchardet, for an example, see:

https://github.com/Feuermagier/autograder/blob/1a47ee7ea91d3d2f7563053528124fd18c2c037f/autograder-core/src/main/java/de/firemage/autograder/core/file/UploadedFile.java#L41-L56

[dfuchss]

In general this might be not possible easily.
Nevertheless, I found this library : https://unicode-org.github.io/icu/userguide/conversion/detection

[tsaglam]

See also past discussions on this topic: #115.

[alberth]

It's theoretically not decidable to obtain the encoding from file content. When you get a file with bytes between 0 and 127, you may conclude it's ASCII, but it could be UTF-8 as well, and if it has an even length, UCS-16 would work too.

You can make a guess, but nothing prevents you from wrongly interpreting bytes in the wrong way.

One way to "solve" this is to move the problem entirely to the language. The language parser has to make sense of the input anyway. Currently the tricky part is that JPlag also reads the text-file to display it to the user. If you could move that part to the language as well, JPlag has no direct contact with the input text anymore.

[sebinside]

Fixed in #1026