Cross compilation to WebAssembly / WASI

[syrusakbary]

This PR aims to support to enable cross-compilation to other architectures, and more namely WebAssembly and WASI.
We've detailed a bit more in the announcement blogpost (which should be released soon: https://wasmer.io/posts/py2wasm-a-python-to-wasm-compiler )

Ideally, we would like most of the changes to be in Nuitka, and py2wasm being a very thin layer on top of that.
More patches will be needed but this PR more or less what enables the bulk work (I do not want to overwhelm the maintainer with too many changes at once).

To summarize, this PR should achieve the following:

• Being able to cross-compile / serialize&desserialize to 32 bit architecture (Wasm32, WASI)
• Added support for some hacks required to compile to WASI

[sourcery-ai]

Hey @syrusakbary - I've reviewed your changes and they look great!

Here's what I looked at during the review

• 🟡 General issues: 1 issue found
• 🟢 Security: all looks good
• 🟢 Testing: all looks good
• 🟢 Complexity: all looks good
• 🟢 Docstrings: all looks good

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment to tell me if it was helpful.}

[kayhayen]

This looks like it's not a lot yet, I hope there is instructions somewhere how to reproduce it, I am confused about the representation changes and the hard coded long change, that Python doesn't know about in ctypes, is that also unsupported as a module?

[kayhayen]

That looks wrong to me, hardcoding that, just as an experiment?

[kayhayen]

That's going to cause issues with much of standalone, etc. but I guess, that's not a topic for that platform?

[kayhayen]

So, cross compilation has not been supported for a long time, however, this looks like a good case for getting it started, esp. if you are willing to help. I invite you to reach out to me in private email if you can allocate a bit of resources to it, and we can split to work.

In my plan, it takes a slave Python process, that is capable of answering questions. Things like os.name and so on, that are compile time evaluated, are currently all hidden behind "hard imports" that e.g. indicate it to be constant value, and then there are modules, allow listed to be used with getattr that turn the result into compile time values.

There are also functions like

def getComputationResult(node, computation, description, user_provided):
    """With a computation function, execute it and return constant result or
    exception node.

    """

that are used by the core of Nuitka optimization a bunch. Other places that you found, would e.g. have to cover encoding a value for the target platform with struct.pack and the like. Many places are already allowing for proper abstraction of things, and the challenge would be to find ways to not loose a bunch of compile time performance for the normal case.

The plan would be to make an interface, that gets asked questions, and responds with either a slave Python used, which in your case would be a WASI one, where computations are actually executed. Communication to this slave Python (in my goal set) is like is in the transport level SSH and/or subprocess input/output, and cached. This interface can then be chosen. You can lie in that interface all you want, or hard code things if you feel the need for it, but I would prefer caching to be used over hard coding, if we discover this to be slow. For the standard case, said interface would be trivial, but it would provide the hooks needed, to know what WASI gives for os.name without us knowing what it does.

This interface could then be implemented by something that implements more or less things. For Nuitka, I am also interested in cross version compilation. Like being able to compile Python 3.7 as a target Python on 3.13 with no-GIL threads and being able to move the Nuitka core from 2.6 / 3.x dual lingo, to more modern Python.

Another side of the interface would be to also plug Scons C compiler selection and configuration.

I would envision that Nuitka plugins end up doing these things, with the limitation probably that only one plugin can handle the slave Python interface. The plugins interface of Nuitka however, should probably provide the infrastructure for communication between the two Python interpreters. The pickle might be a good start, but I seem to recall that it is incapable of doing everything, we might have to do something, that allows for the host Python to speak of values and types, that only the slave Python actually knows for real, but this can be expanded over time. I am assuming, that you can live with matching the Python version, and I can later figure out how to unlock current code to produce 3.13 programs when Nuitka is running on 2.6 and vice versa.

[syrusakbary]

Awesome @kayhayen . I'll send you an email soon!

[jedisct1]

dlopen() is always going to fail, so I'd suggest adding something like:

--- a/nuitka/build/static_src/MetaPathBasedLoader.c
+++ b/nuitka/build/static_src/MetaPathBasedLoader.c
@@ -1181,6 +1181,7 @@ static PyObject *_nuitka_loader_load_module(PyObject *self, PyObject *args, PyOb
     PyThreadState *tstate = PyThreadState_GET();

 #ifndef _NUITKA_STANDALONE
+# ifndef __wasi__
     if (installed_extension_modules != NULL) {
         PyObject *extension_module_filename = DICT_GET_ITEM0(tstate, installed_extension_modules, module_name);

@@ -1189,6 +1190,7 @@ static PyObject *_nuitka_loader_load_module(PyObject *self, PyObject *args, PyOb
             return callIntoInstalledExtensionModule(tstate, module_name, extension_module_filename);
         }
     }
+# endif
 #endif

     return IMPORT_EMBEDDED_MODULE(tstate, name);
@@ -1598,6 +1600,7 @@ static PyObject *_nuitka_loader_exec_module(PyObject *self, PyObject *args, PyOb
     // During spec creation, we have populated the dictionary with a filename to load from
     // for extension modules that were found installed in the system and below our package.
 #ifndef _NUITKA_STANDALONE
+# ifndef __wasi__
     if (installed_extension_modules != NULL) {
         PyObject *extension_module_filename = DICT_GET_ITEM0(tstate, installed_extension_modules, module_name);

@@ -1613,6 +1616,7 @@ static PyObject *_nuitka_loader_exec_module(PyObject *self, PyObject *args, PyOb
             return callIntoInstalledExtensionModule(tstate, module_name, extension_module_filename);
         }
     }
+# endif
 #endif

     return EXECUTE_EMBEDDED_MODULE(tstate, module);

This allows zig cc to be used as the wasm compiler.

[kayhayen]

For zig cc we did a bunch of work. For 3.11 or higher we got a workaround that uses their C++ compiler or what we call non C11 mode

[bitnom]

very interested in this