Add support for specifying response file encoding

[juj]

Add support for specifying response file encoding using the suffix of the response file name, and autodetect the response file encoding using the suffix of the response file name if one is specified there.

Fixes #15292.

[sbc100]

lgtm.. with a couple of commend.

I still think it would be good to see who llvm does this without resorting to such measures (I assume clang accepts all these different encodings without issue?)

[dschuff]

This new test seems to be failing on the emscripten-releases roller: https://logs.chromium.org/logs/emscripten-releases/buildbucket/cr-buildbucket/8831614067975027953/+/steps/Emscripten_testsuite__upstream__other_/0/stdout

specifically:

Traceback (most recent call last):
  File "C:\b\s\w\ir\k\install\emscripten\tools\response_file.py", line 90, in read_response_file
    args = f.read()
  File "C:\b\s\w\ir\cache\vpython\4adb0b\lib\codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe4 in position 0: invalid continuation byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\b\s\w\ir\k\install\emscripten\\emcc.py", line 3852, in <module>
    sys.exit(main(sys.argv))
  File "C:\b\s\w\ir\k\install\emscripten\\emcc.py", line 3845, in main
    ret = run(args)
  File "C:\b\s\w\ir\k\install\emscripten\\emcc.py", line 980, in run
    args = substitute_response_files(args)
  File "C:\b\s\w\ir\k\install\emscripten\tools\response_file.py", line 111, in substitute_response_files
    new_args += read_response_file(arg)
  File "C:\b\s\w\ir\k\install\emscripten\tools\response_file.py", line 96, in read_response_file
    args = f.read()
  File "C:\b\s\w\ir\cache\vpython\4adb0b\lib\codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe4 in position 0: invalid continuation byte
None
None
test_response_file_encoding (test_other.other) ... FAIL

[sbc100]

I would guess because of "This test assumes that the default system encoding is CP-1252."?

I think this PR was submitted via auto-submit by mistake.. I suggest we revert it for now and re-land with comments addressed? (Unless @juj is around right now and wants to fix forward?)

[juj]

This test assumes that the default system encoding is CP-1252.
Does this assumption hold on all windows machines? (or at least all that might want to run this test?)

Yes, it should, although I now understand the term "default system encoding" I used there is wrong. It is the "default Python encoding on Windows" and not the system's default encoding. Python documentation for this can be found here:

https://docs.python.org/3/glossary.html#term-locale-encoding

where it states

On Windows, it is the ANSI code page (ex: cp1252).

However this is not the default Windows encoding - that is at least for me cp437, as detected by

E:\code\emsdk\emscripten\main>chcp
Active code page: 437

The above wording suggests that Python does not make a system-specific choice, however:

This new test seems to be failing on the emscripten-releases roller
I would guess because of "This test assumes that the default system encoding is CP-1252."?

In Python docs for locale.getpreferredencoding() at https://docs.python.org/3/library/locale.html#locale.getpreferredencoding it says

On Android or if the Python UTF-8 Mode is enabled, always return 'UTF-8', the locale encoding and the do_setlocale argument are ignored.

so I presume that this Python UTF-8 Mode must be in effect on the autoroller. There's this comment also in the docs about the Python UTF-8 Mode:

It can be enabled or disabled using the -X utf8 command line option and the PYTHONUTF8 environment variable.

[sbc100]

so I presume that this Python UTF-8 Mode must be in effect on the autoroller. There's this comment also in the docs about the Python UTF-8 Mode:

It can be enabled or disabled using the -X utf8 command line option and the `PYTHONUTF8` environment variable.

Interestingly we set this in github CI also:

emscripten/.circleci/config.yml

Lines 409 to 411 in 0d24418

	# Override the default locale to use UTF-8 encoding for all files and stdio
	# streams (see PEP540), since oure test files are encoded with UTF-8.
	PYTHONUTF8: "1"

Which I guess means we are not running this tests on windows CI. The list of tests we currently run on windows during github CI is:

emscripten/.circleci/config.yml

Line 431 in 0d24418

test_targets: "other.test_emcc_cflags other.test_stdin other.test_bad_triple wasm2.test_sse1 wasm2.test_ccall other.test_closure_externs other.test_binaryen_debug other.test_js_optimizer_parse_error other.test_output_to_nowhere other.test_emcc_dev_null other.test_cmake* other.test_system_include_paths other.test_emar_response_file wasm2.test_utf16 other.test_special_chars_in_arguments other.test_toolchain_profiler"

. So I guess this new test should be added to that list when you re-upload this PR.

emscripten-releases also looks like it sets PYTHONUTF8: https://chromium.googlesource.com/emscripten-releases/+/refs/heads/main/src/build.py#1777.

I think it should be possible to "unset PYTHONUTF8" (using with modify_env .. :) during that particular test in order run emcc without that environment variable.

[juj]

I rewrote the test to work in either case when PYTHONUTF8 is 0 or 1.