Python Bindings under Emscripten?

[gabriel-v]

I tried to build the python bindings under emscripten.

Here's how it went:

1. LONG_BIT definition appears wrong

First, a CPython-related compilation error in Python about LONG_BIT mismatch

/usr/local/include/python3.11/pyport.h:601:2: error: "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
#error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."

This is similar to what other people have been asking in pyodide and in pybind11. The pyodide issue seems to have been fixed in 0.22 but I can't figure out what they did.

I also looked into cross compiling python with wasm/wasi using upstream CPython3.11 - it doesn't raise the LONG_BIT error there.

I guess we are either missing some emscripten compiler configs in the toolchain, or some python build configurations - the system target arch there are called wasm32-unknown-emscripten instead of just emscripten.

Here is the emconfigure command for compiling python with emscripten vs. magnum emscripten toolchain, we probably have to reproduce the effects of these arguments:

emconfigure \
    --host=wasm32-unknown-emscripten \
    --with-emscripten-target=browser \
    --enable-wasm-dynamic-linking=no \

So I ignored it and patched it out to see what's next.

2. Missing Symbols: corrade plugin manager; magnum GL bindings

• Then, missing symbols from:
• corrade plugins manager
• src/python/magnum/gl.cpp

I suppose the corrade plugins manager doesn't compile because of missing functionality in emscripten, and the gl.cpp binding fails because we only have EGL not GL.

I'm willing to work on adding python support to your bindings as a hobby project of mine - do let me know if you have any tips!

Would it be possible to write bindings to have Python bindings code in the browser?

[pmp-p]

@gabriel-v would you mind give try with https://github.com/pygame-web/python-wasm-sdk ?

use . /opt/python-wasm-sdk/wasm32-mvp-emscripten-shell.sh to enter C/C++ build env
and/or /opt/python-wasm-sdk/python3-wasm to run setup.py bdist_wheel or -m build

[mosra]

Hi, wow, you're adventurous :) I never even thought the python bindings would be useful under Emscripten, so the code doesn't contain any Emscripten-/WebGL-specific handling at the moment.

For the LONG_BIT part I'm afraid I can't help. For the emconfigure options I think we're fine:

• --host=wasm32-unknown-emscripten is compiling for 32-bit WASM, which is the default in the toolchain. The other option is wasm64-unknown-emscripten and AFAIK it's still in very early stages, with no browser support yet.
• --enable-wasm-dynamic-linking=no is also the default in my toolchain
• --with-emscripten-target=browser isn't that important I think, the default is both a browser and a console. In case of Magnum, the ability to run JS in a console (via node) is useful for unit tests and other command-line tools.

For the build errors, the bindings codebase doesn't handle a case of plugin managers without dynamic library support (i.e., *.so and *.dll plugins, it's also related to the emconfigure option above, and while it could technically be possible now, I didn't bother with adding support because the overhead would be so big that it wouldn't make sense). The solution is to #ifndef the related parts accordingly:

diff --git a/src/python/corrade/pluginmanager.cpp b/src/python/corrade/pluginmanager.cpp
index 487bf06..1d5c436 100644
--- a/src/python/corrade/pluginmanager.cpp
+++ b/src/python/corrade/pluginmanager.cpp
@@ -42,16 +42,21 @@ void pluginmanager(py::module_& m) {
     py::enum_<PluginManager::LoadState> loadState{m, "LoadState", "Plugin load state"};
     loadState
         .value("NOT_FOUND", PluginManager::LoadState::NotFound)
+        #ifndef CORRADE_PLUGINMANAGER_NO_DYNAMIC_PLUGIN_SUPPORT
         .value("WRONG_PLUGIN_VERSION", PluginManager::LoadState::WrongPluginVersion)
         .value("WRONG_INTERFACE_VERSION", PluginManager::LoadState::WrongInterfaceVersion)
         .value("WRONG_METADATA_FILE", PluginManager::LoadState::WrongMetadataFile)
         .value("UNRESOLVED_DEPENDENCY", PluginManager::LoadState::UnresolvedDependency)
+        #endif
         .value("STATIC", PluginManager::LoadState::Static)
         .value("LOADED", PluginManager::LoadState::Loaded)
+        #ifndef CORRADE_PLUGINMANAGER_NO_DYNAMIC_PLUGIN_SUPPORT
         .value("NOT_LOADED", PluginManager::LoadState::NotLoaded)
         .value("UNLOAD_FAILED", PluginManager::LoadState::UnloadFailed)
         .value("REQUIRED", PluginManager::LoadState::Required)
-        .value("USED", PluginManager::LoadState::Used);
+        .value("USED", PluginManager::LoadState::Used)
+        #endif
+        ;
     corrade::enumOperators(loadState);
 
     PyNonDestructibleClass<PluginManager::AbstractManager> manager{m, "AbstractManager", "Base for plugin managers"};
@@ -61,6 +66,7 @@ void pluginmanager(py::module_& m) {
             /** @todo drop std::string in favor of our own string caster */
             return std::string{self.pluginInterface()};
         }, "Plugin interface")
+        #ifndef CORRADE_PLUGINMANAGER_NO_DYNAMIC_PLUGIN_SUPPORT
         .def_property("plugin_directory",
             /** @todo drop std::string in favor of our own string caster */
             [](PluginManager::AbstractManager& self) {
@@ -69,6 +75,7 @@ void pluginmanager(py::module_& m) {
                 self.setPluginDirectory(directory);
             }, "Plugin directory")
         .def("reload_plugin_directory", &PluginManager::AbstractManager::reloadPluginDirectory, "Reload plugin directory")
+        #endif
         /** @todo setPreferredPlugins (takes an init list) */
         .def_property_readonly("plugin_list", [](PluginManager::AbstractManager& self) {
             /** @todo make a generic caster for arbitrary arrays and strings */

For the GL errors it's similar, the codebase is currently handling differences between GL and GLES (which is a subset of GL), but it doesn't handle WebGL differences (which itself is an even more restricted subset of GLES). A patch that attempts to fix at least some of these is below:

diff --git a/src/python/magnum/gl.cpp b/src/python/magnum/gl.cpp
index 533f481..4498492 100644
--- a/src/python/magnum/gl.cpp
+++ b/src/python/magnum/gl.cpp
@@ -307,6 +307,7 @@ void gl(py::module_& m) {
     {
         py::class_<GL::Context, ContextHolder<GL::Context>> context{m, "Context", "Magnum OpenGL context"};
 
+        #ifndef MAGNUM_TARGET_WEBGL
         py::enum_<GL::Context::Flag> contextFlag{context, "Flag", "Context flag"};
         contextFlag
             .value("DEBUG", GL::Context::Flag::Debug)
@@ -319,6 +320,7 @@ void gl(py::module_& m) {
             #endif
             .value("NONE", GL::Context::Flag{});
         corrade::enumOperators(contextFlag);
+        #endif
 
         py::enum_<GL::Context::State> contextState{context, "State", "State to reset"};
         contextState
@@ -844,6 +846,7 @@ void gl(py::module_& m) {
         .value("STENCIL", GL::FramebufferClear::Stencil);
     corrade::enumOperators(framebufferClear);
 
+    #if !(defined(MAGNUM_TARGET_WEBGL) && defined(MAGNUM_TARGET_GLES2))
     py::enum_<GL::FramebufferBlit> framebufferBlit{m, "FramebufferBlit", "Mask for framebuffer blitting"};
     framebufferBlit
         .value("COLOR", GL::FramebufferBlit::Color)
@@ -854,6 +857,7 @@ void gl(py::module_& m) {
     py::enum_<GL::FramebufferBlitFilter>{m, "FramebufferBlitFilter", "Framebuffer blit filtering"}
         .value("NEAREST", GL::FramebufferBlitFilter::Nearest)
         .value("LINEAR", GL::FramebufferBlitFilter::Linear);
+    #endif
 
     py::class_<GL::AbstractFramebuffer, NonDefaultFramebufferHolder<GL::AbstractFramebuffer>> abstractFramebuffer{m,
         "AbstractFramebuffer", "Base for default and named framebuffers"};
@@ -862,12 +866,14 @@ void gl(py::module_& m) {
         /** @todo limit queries */
 
         /* Using lambdas to supply an enum and not enum set */
+        #if !(defined(MAGNUM_TARGET_WEBGL) && defined(MAGNUM_TARGET_GLES2))
         .def_static("blit", [](GL::AbstractFramebuffer& source, GL::AbstractFramebuffer& destination, const Range2Di& sourceRectangle, const Range2Di& destinationRectangle, GL::FramebufferBlit mask, GL::FramebufferBlitFilter filter) {
             GL::AbstractFramebuffer::blit(source, destination, sourceRectangle, destinationRectangle, mask, filter);
         }, "Copy a block of pixels", py::arg("source"), py::arg("destination"), py::arg("source_rectangle"), py::arg("destination_rectangle"), py::arg("mask"), py::arg("filter"))
         .def_static("blit", [](GL::AbstractFramebuffer& source, GL::AbstractFramebuffer& destination, const Range2Di& rectangle, GL::FramebufferBlit mask) {
             GL::AbstractFramebuffer::blit(source, destination, rectangle, mask);
         }, "Copy a block of pixels", py::arg("source"), py::arg("destination"), py::arg("rectangle"), py::arg("mask"))
+        #endif
         .def("bind", &GL::AbstractFramebuffer::bind, "Bind framebuffer for drawing")
         .def_property("viewport", &GL::AbstractFramebuffer::viewport, &GL::AbstractFramebuffer::setViewport, "Viewport")
         /* Using lambdas to avoid method chaining getting into signatures */

I don't have an easy way to verify that the patches are complete, so it will likely need more #ifndefing. The workflow in that case would be that for every error you look at the corresponding definition in the Magnum header and copy the appropriate #ifndef over.

Hope this helps -- and of course I'll be very happy to integrate your patches once you get this to build :)

[mosra]

Oh, and one other thing that might be a source of pain is the Sdl2Application library that the current Python examples rely on -- hopefully it will work as-is (enable it with MAGNUM_WITH_SDL2APPLICATION), maybe you'll need to add similar ifdefs there as well (#ifndef CORRADE_TARGET_EMSCRIPTEN most probably).

The go-to application to use on the web would be EmscriptenApplication though, which would need to be added as a new binding file into platform/.

The windowless / EGL applications could also work there, but are not essential for anything -- they're used just by unit tests.