[WIP] Fixes to get unit tests working for Arm64 and Visual Studio 2022 Preview

[Blackhex]

This PR is heavily WIP. I am just wondering whether the test\runner.py tests are runing for Windows on the CI as they seem quite outdated.

[sbc100]

You can use the shared.bat_suffix helper here perhaps?

[sbc100]

Do you know why these changes would be needed? Isn't python supposed to include the script directory implicitly? What version of python are you using here?

[Blackhex]

Currently, I am testing it with the official Arm64 Python binaries (https://www.python.org/ftp/python/3.11.1/python-3.11.1-embed-arm64.zip) installed and activated by emsdk.py. But I am experiencing the same issue on a x64 machine with Python interpreter obtained from emsdk install latest. It's also a mystery to me that I am currently trying to solve. For some reason, the %EMSDK_PYTHON% Python interpreter is ignoring PYTHONPATH variable (even when -E argument is removed from bat files) and it does not contain current working dir in sys.path by default.

[sbc100]

But you shouldn't need to set PYTHONPATH at all should you?

The current working directory is not normally part of sys.path but the script directory certainly should be.

e.g.:

$ cat emscripten/test.py 
import sys
print(sys.path)
$ pwd
/usr/local/google/home/sbc/dev/wasm
$ python3 emscripten/test.py
['/usr/local/google/home/sbc/dev/wasm/emscripten', '/usr/local/buildtools/current/sitecustomize', '/usr/lib/python310.zip', '/usr/lib/python3.10', '/usr/lib/python3.10/lib-dynload', '/usr/local/google/home/sbc/.local/lib/python3.10/site-packages', '/usr/local/lib/python3.10/dist-packages', '/usr/lib/python3/dist-packages', '/usr/lib/python3.10/dist-packages']

Note that pwd does not appear in the path, but that script directory does.

Is there some reason you want to remove the -E from the python command?

[Blackhex]

Your are right in every point. I am just not sure what you mean by "script directory"? site-packages? Nevertheless, currently I just need to find a simple workaround to continue working on making unit tests passing on Arm64 with Visual Studio 2022 and I'll figure out what is wrong with the Python interpreted acquired by emdsk.py later. With the system-wide Python-interpreter installed from EXE installer, the runner.py is loading the Python local scripts just fine.

[Blackhex]

@sbc100, JFYI, the workarounds for sys.path imports are related to this Python issue python/cpython#101867

[sbc100]

We intentionally don't use the embed versions of python for this (and perhaps other) reasons.

How are you installing python? Why not just use the distro package of python? this is what we normally use/recommend on linux. (Note that while the mac and windows versions of emsdk install python themselves the linux version expects you to have it installed already).

[sbc100]

Oh I see, you are trying to install on windows, not linux. See #16466 for when we switched to the nuget packages.

[sbc100]

See scripts/update_python.py for how to build the patched nuget installer. I think you would need to modify that script to support non x86_64 arch.

[Blackhex]

See scripts/update_python.py for how to build the patched nuget installer. I think you would need to modify that script to support non x86_64 arch.

Great, thank you. That's actually what we are currently working on. Btw, why some Python packages listed in the emsdk_manifest.json contains "embed" in its name, e. g. https://storage.googleapis.com/webassembly/emscripten-releases-builds/deps/python-3.9.2-1-embed-amd64+pywin32.zip?