Non-ASCII characters in files

[leiflindholm]

Expanding the conversation from #5840

Background:

• Passing an ECC check is required for a commit to pass CI and be possible to merge through mergify.
• Some people don't have a way to write their name that fits into 7-bit ASCII.
• Copyright statements.

@mdkinney in #5840

If non-ASCII is required in a source file comment, then we need to decide the file format and encodings supported and we need to make sure every tool that may open a source file is compatible with that encoding. I know over the years, we have had both commercial tools and custom tools that would fail if the source file type was not ASCII. Compilers, static analysis, document generators, and statistics collection are just a few examples of the types of tools that open/read source files. We also have to make sure all the editors that the community is using supports the select file type/encodings.

@leiflindholm here

1. The proposed ECC change does not flip to "permit ASCII in comments", it introduces support for matching specific strings as exceptions. Through the CI system, this ends up being a per-package exception.
2. There is only one format to support, UTF-8. Which is the charset we already support - only in source files we are currently manually restricting ourselves to code points in the range 0-127.
3. Many of the submodules contain source files with non-ASCII characters, and a few files have snuck through before ECC was enabled in ci (I presume).
4. The only way to determine whether any tools are still incompatible with the modern world is to test, and the only way to test is to merge some source files with non-ASCII in them (OvmfPkg/Include/Library/AcpiPlatformLib.h already has a non-ASCII char in copyright, since June 2023).
5. The strict ban on non-ASCII also extends to things like https://github.com/tianocore/edk2/blob/master/EmbeddedPkg/Drivers/VirtualKeyboardDxe/ComponentName.c#L37, which feels wrong.

Now, fundamentally, what I really want is an override label that lets me ignore ECC when it's being unhelpful. But the ASCII-question still needs resolving.

[mdkinney]

I am not in favor of a way to ignore/bypass all ECC checks. Let's resolve the ASCII topic and fix ECC to follow the approved changes to the ASCII checks.

[mdkinney]

The ComponentName example is interesting. In many places, Unicode strings associated with UI are implemented in .UNI files and are referenced as tokens from C/H files as part of HII. ComponentName has no relation to HII. And UEFI Specification supports Unicode strings for consoles. So use of Unicode strings needs to be supported in general for any UEFI API that supports CHAR16 * parameters.
Does ECC currently allow non-ASCII in Unicode L"" strings in C/H files? If so, then perhaps those exception need to be extended to Copyright statements in a file header comment block?

[leiflindholm]

Does ECC currently allow non-ASCII in Unicode L"" strings in C/H files? If so, then perhaps those exception need to be extended to Copyright statements in a file header comment block?

No. ECC's current non-ASCII check is GeneralCheckNonAcsii (), which runs before the content-aware tests and just checks that all bytes on every line are not > 126 (apparently DEL isn't good enough 7-bit).

My change rejigs this slightly, treating each line as a UTF-8 string, then attempting to convert it to ASCII and if that fails, checks if the word it failed in was a configured exception, and if so reattempts to convert the rest of the line, after the exception.

If we wanted to be able to support non-ASCII in a content-aware fashion, we'd need to drop GeneralCheckNonAcsii () completely and do something more complicated later on.

[leiflindholm]

Digging a bit further, if we restricted ourselves to restricting (!) identifier names and possibly path names, the best I can come up with is we could hook someting into the various individual functions called from NamingConventionCheck () in Check.py.

This would mean dropping GeneralCheckNonAcsii () and would completely ignore the contents of comments and string literals (so getting rid of the ComponentName problem).