500 Internal Error when jury view code with non-UTF-8 encoding code.

[cubercsl]

Description of the problem

Jury can not view code with non-UTF-8 encoding code (and diff with it)

The string to escape is not a valid UTF-8 string in "jury/partials/submission_diff.html.twig" at line 72.

Your environment

• DOMjudge version 9.0 or later (with Monaco Editor)
• Debian 13
• Nginx

Steps to reproduce

1. Team or jury submit a code with non-UTF-8 encoded source code. (This may happen in some local contest, some team may type some non-ascii character in their source code)
2. Jury view the code.

Expected behaviour

It should be shown either in the correct encoding or force display in UTF-8 encoding.
At the very least, it shouldn't be a 500 error; the code can't be displayed at all.

Actual behaviour

Jury can view the code.

Any other information that you want to share?

I have repro this in demoweb now. See https://www.domjudge.org/demoweb/jury/submissions/9677/source
It is also happened in our last weekend ICPC Regional Contest (only one jury test submission with Chinese GBK-encoded comment in his code and submit from his Windows laptop), and I have a db-dump.
If more information is needed, I can provide it via slack DM.

[vmcj]

@cubercsl can you enclose the file here? We reset the database every night so your upload will get lost. I now tried to extract it from the database but having the original is better.

[cubercsl]

non-utf8-test.cpp

[Kevinjil]

Fix is included in #3132

[HeRaNO]

Fix is included in #3132

I've checked the fix (in 43f4b58 and 5e22362). In these commits, the editor rejects all codes without UTF-8 encoding. However, in our online contest, many participants use GBK encoding, and quite a number of codes contain Chinese comments in GBK encoding. These commits will prevent the jury from checking these codes backstage. Would you consider relaxing the encoding rule to allow more encoding? At least we can ignore the GBK encoding part (which can be shown as strange characters) and ensure the ASCII part is shown correctly. Some libraries, like jschardet, will also help.

[Kevinjil]

@HeRaNO Could you test #3221, as you seem to have a lot of non-UTF8 submissions. I have never encountered non-UTF8 source code files.

[HeRaNO]

@HeRaNO Could you test #3221, as you seem to have a lot of non-UTF8 submissions. I have never encountered non-UTF8 source code files.

@Kevinjil Sure, but I only have time to test the PR this weekend. You can submit the file @cubercsl provides, which is in GBK encoding.

[cubercsl]

Before: (current main branch)

After: (cherry-pick PR #3221)

I have tested in my db-dump with the main branch and cherry-pick #3221.

Although some non-UTF-8 encoded comments were displayed as gibberish, this did not prevent the judges from viewing the core of the code. So this is fine from my side.

[Kevinjil]

@cubercsl Thank you for testing it.
I highly suspect the source file to be a valid file for multiple different character encodings, resulting in mb_detect_encoding picking "the wrong one". Without additional information about the file, I'm not sure if there are better ways to find the actual encoding of a file.

[HeRaNO]

I think it's fine. These characters are usually harmless, and we just need to make sure we don't block the jury from checking the code. Before the modification, the jury can only download the code to check it.

[Kevinjil]

I updated the PR to respect the system defined encoding detection order, and fall back to all available encodings. If you configure mbstring.detect_order correctly for your expected submission encodings, it should work.