New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compile Python as a shared library and dynamically load it at runtime #1

Open
wants to merge 15 commits into
base: master
from

Conversation

Projects
None yet
3 participants
@adewes
Copy link

adewes commented Jun 6, 2017

I modified the code such that it build the Python shared library and compiles it to an asm.js module. I added an example that imports this shared library at runtime.

The compilation of the shared library works as does the import, but unfortunately I still get errors when running the code in production:

Summary of changes:

  • Added --enable-shared flag to build the shared Python library
  • Add an extra build step which invokes emcc to build a JS module from the shared library (using SIDE_MODULE=1 to enable dynamic loading)
  • I added an example that loads the dynamic library at runtime

Problems encountered:

  • When trying to convert libpython3.5.a into a dynamic ASM library, I get a symbol multiply defined error. Inspecting the .a library with llvm-nm reveals that many symbols are defined more than once. I solved this by explicitly building the .so library (which uses different compiler flags it seems)
  • When importing the dynamic library and calling Py_InitializeEx I get the following error: Python error: Py_Initialize: can't import _frozen_importlib. I was not able to find the root cause for this (I checked in the output files in the debug/import_error branch for easier debugging).

Any idea where the problem could be? Would be really great to get DLL loading to work, as this would allow (in my understanding) to dynamically import additional Python modules that rely on C extensions (such as numpy).

@Beuc

This comment has been minimized.

Copy link

Beuc commented May 20, 2018

Interesting approach.

Accordingly to https://github.com/kripken/emscripten/wiki/Linking I believe that brings 5-10% slowdown, but that may be worth it.

The Py_Initialize error you get may be caused by not including any Python module (.py/.pyc) whatsoever in your lib_files.

Also one needs to make Python actually compile dlopen support (e.g. add ac_cv_func_dlopen=yes in config.site).

Last it's worth noting that with 1.38.1, which defaults to WASM=1, dynamic linking is broken.

@adewes

This comment has been minimized.

Copy link
Author

adewes commented May 22, 2018

Thanks for that feedback, I'll try your suggestions when I have time!

@Beuc

This comment has been minimized.

Copy link

Beuc commented May 22, 2018

I was a bit harsh with my comment about WASM, it has opened issues with dynamic loading but it works.
I could load Cython modules dynamically in my Python 2.7 test.
I updated the Linking wiki page accordingly.

@pmp-p

This comment has been minimized.

Copy link

pmp-p commented Nov 23, 2018

hi @Beuc , did you have succes in loading wasm modules >4K since they do not support RTLD_NOW on main thread ?

@Beuc

This comment has been minimized.

Copy link

Beuc commented Nov 23, 2018

I've switched to static modules so as to use the Emterpreter, so I can't test right now.
Also I probably didn't test extensively with Chromium at that time :)
The modules were largely over 4k though (100-1500k).
I'll try next time I have the opportunity.

@Beuc

This comment has been minimized.

Copy link

Beuc commented Nov 23, 2018

@Beuc

This comment has been minimized.

Copy link

Beuc commented Feb 1, 2019

Did some experiments today while cleaning-up my code:

Error in loading dynamic library /lib/python2.7/site-packages/pygame_sdl2/error.so: RangeError: WebAssembly.Compile is disallowed on the main thread, if the buffer size is larger than 4KB. Use WebAssembly.compile, or compile on a worker thread.

Looks like that doesn't work with Chromium indeed.
No issues with Firefox.

I asked about it at https://groups.google.com/forum/#!topic/emscripten-discuss/cE3hUV3fDSw

@pmp-p

This comment has been minimized.

Copy link

pmp-p commented Feb 1, 2019

exactly what i have hit on chromium 71, is it practical to load the .wasm with a worker while waiting to dlopen() it, seems your position is a "no" ?

i have actually an enormous libpanda3d.js of 20MiB to download/eval after already huge libpython (3.7.2), before i can import anything.

i'm quite sure wasm would be faster but i got lost in contradictory documentation and outdated samples. maybe because of difference beetween browsers ?

@Beuc

This comment has been minimized.

Copy link

Beuc commented Feb 1, 2019

I would suggest posting to the emscripten-discuss list, there are far more people there :)

@pmp-p

This comment has been minimized.

Copy link

pmp-p commented Feb 2, 2019

sure but the official response there is "I don't think we can point to a specific time, it depends on a lot of things, in particular on how many people help out and how much - help is always welcome and necessary"

by "sample" i mean things i can work on from to help, as i already have a ffi interface for micropython-wasm ( based on @brion work there https://github.com/brion/libffi/tree/emscripten-work )

@Beuc

This comment has been minimized.

Copy link

Beuc commented Feb 2, 2019

I mean more people can help and give their opinion, that is a general ML (not just dev).
And this project here seems dead.

@pmp-p

This comment has been minimized.

Copy link

pmp-p commented Feb 2, 2019

ok :) and yeah maybe https://pythonclock.org/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment