New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dynamic linking of BLAS / LAPACK #240

Open
rth opened this Issue Nov 2, 2018 · 10 comments

Comments

Projects
None yet
3 participants
@rth
Copy link
Member

rth commented Nov 2, 2018

This was discussed in #211 I would like to reformulate the issue so it's easier for someone to continue working on it.

Scipy requires BLAS and LAPACK to build. Note: my earlier comments in #211 (comment) about LAPACK being optional due to the note at the and of this paragraph in scipy docs were actually not true, as was confirmed on the scipy-dev mailing list.

Currently in #211 we link LAPACK statically, which means that it gets included in around ~8 compiled C extensions , with a resulting size of 170MB lz4 compressed / 650MB uncompressed. Different compile flags where tried in #211 but ultimately it doesn't improve size much.

The most clean way to improve the package size would be to link dynamically, which would also allow to get rid of some hacks in pyodide_build/pywasmcross.py (#211 (comment)). More information about dynamic linking in emscripten can be found in https://github.com/kripken/emscripten/wiki/Linking

Some of this was tried in #211 (comment) but didn't succeed. In retrospective, disabling validation of output in emcc / asm2wasm could be a first step. Then one would need to manually open the needed modules with dlopen, since ldd does not exist in emscripten (#211 (comment)).

@navytux

This comment has been minimized.

Copy link

navytux commented Nov 8, 2018

Some feedback about dynamic linking from wasm/emscripten newbie discovered while learning it:

It is possible in emscripten to do dynamic linking not only from main module to other libraries, but from those other libraries to next other dynamic modules. (In scipy case the main module is main pyiodide, a DSO is a *.so from scipy, and those DSO want to link to e.g. liblapack.so). The way it works is that a Module in emscripten can have .dynamicLibraries, and then the runtime automatically loads referenced libraries. Today this works for main module already (one uses e.g. -s RUNTIME_LINKED_LIBS="...") but does not automatically work from sided DSO to another DSO. However even today it could be worked around if DSO A wants to link to DSO B e.g. via

// (liba.c)
__attribute__((constructor))
static void init() {
   // XXX EM_ASM does not work in wasm mode
#  if 1
   EM_ASM({
       loadDynamicLibrary('libB_so.js');
   });
#  else
     emscripten_run_script("loadDynamicLibrary('libB_so.wasm')");
#  endif
}

this just calls loadDynamicLibrary from emscripten runtime to do the job.

It works but is not good for real usage, as if e.g. A1 and A2 both would want to load B, then B would be loaded twice. It seems to be not too hard to fix in emscripten runtime however - please see notes here:

https://lab.nexedi.com/kirr/emscripten/commit/98006640

This way conventional support for shared libraries could be implemented in emscripten - similar to what we are used to have e.g. on Linux and other OS'es.

I've put a full test/demo for such linking here:

https://lab.nexedi.com/kirr/wasm-study (rev ee35d772 as of this writing).

Kirill

navytux added a commit to navytux/emscripten that referenced this issue Nov 15, 2018

Teach dynamic linking to handle library -> library dependencies
Currently Emscripten allows to create shared libraries (DSO) and link
them to main module. However a shared library itself cannot be linked to
another shared library.

The lack of support for DSO -> DSO linking becomes problematic in cases when
there are several shared libraries that all need to use another should-be
shared functionality, while linking that should-be shared functionality to main
module is not an option for size reasons. My particular use-case is SciPy
support for Pyodide:

	iodide-project/pyodide#211,
	iodide-project/pyodide#240

where several of `*.so` scipy modules need to link to LAPACK. If we link to
LAPACK statically from all those `*.so` - it just blows up compiled size

	iodide-project/pyodide#211 (comment)

and if we link in LAPACK statically to main module, the main module size is
also increased ~2x, which is not an option, since LAPACK functionality is not
needed by every Pyodide user.

This way we are here to add support for DSO -> DSO linking:

1. similarly to how it is already working for main module -> side module
   linking, when building a side module it is now possible to specify

	-s RUNTIME_LINKED_LIBS=[...]

   with list of shared libraries that side module needs to link to.

2. to store that information, for asm.js, similarly to how it is currently
   handled for main module (which always has js part), we transform
   RUNTIME_LINKED_LIBS to

	libModule.dynamicLibraries = [...]

   (see src/preamble_sharedlib.js)

3. for wasm module, in order to store the information about to which libraries
   a module links, we could in theory use "module" attribute in wasm imports.
   However currently emscripten almost always uses just "env" for that "module"
   attribute, e.g.

	(import "env" "abortStackOverflow" (func $fimport$0 (param i32)))
	(import "env" "_ffunc1" (func $fimport$1))
	...

   and this way we have to embed the information about required libraries for
   the dynamic linker somewhere else.

   What I came up with is to extend "dylink" section with information about
   which shared libraries a shared library needs. This is similar to DT_NEEDED
   entries in ELF.

   (see tools/shared.py)

4. then, the dynamic linker (loadDynamicLibrary) is reworked to handle that information:

   - for asm.js, after loading a libModule, we check libModule.dynamicLibraries
     and post-load them recursively. (it would be better to load needed modules
     before the libModule, but for js we currently have to first eval whole
     libModule's code to be able to read .dynamicLibraries)

   - for wasm the needed libraries are loaded before the wasm module in
     question is instantiated.

     (see changes to loadWebAssemblyModule for details)

5. since we also have to teach dlopen() to handle needed libraries, and since dlopen
   was already duplicating loadDynamicLibrary() code in many ways, instead of
   adding more duplication, dlopen is now reworked to use loadDynamicLibrary
   itself.

   This moves functionality to keep track of loaded DSO, their handles,
   refcounts, etc into the dynamic linker itself, with loadDynamicLibrary now
   accepting various flags (global/nodelete) to handle e.g.
   RTLD_LOCAL/RTLD_GLOBAL and RTLD_NODELETE dlopen cases (RTLD_NODELETE
   semantic is needed for initially-linked-in libraries).

   Also, since dlopen was using FS to read libraries, and loadDynamicLibrary was
   previously using Module['read'] and friends, loadDynamicLibrary now also
   accepts fs interface, which if provided, is used as FS-like interface to load
   library data, and if not - native loading capabilities of the environment
   are still used.

   Another aspect related to deduplication is that loadDynamicLibrary now also
   uses preloaded/precompiled wasm modules, that were previously only used by
   dlopen (see a5866a5 "Add preload plugin to compile wasm side modules async
   (emscripten-core#6663)").

   (see changes to dlopen and loadDynamicLibrary)

6. The functionality to asynchronously load dynamic libraries is also
   integrated into loadDynamicLibrary.

   Libraries were asynchronously preloaded for the case when
   Module['readBinary'] is absent (browser, see 3446d2a "preload wasm dynamic
   libraries when we can't load them synchronously").

   Since this codepath was also needed to be taught of DSO -> DSO dependency,
   the most straightforward thing to do was to teach loadDynamicLibrary to do
   its work asynchronously (under flag) and to switch the preloading to use

	loadDynamicLibrary(..., {loadAsync: true})

   (see changes to src/preamble.js and loadDynamicLibrary)

7. A test is added for verifying linking/dlopening a DSO with other needed library.

   browser.test_dynamic_link is also amended to verify linking to DSO with
   dependencies.

With the patch I've made sure that all core tests (including test_dylink_* and
test_dlfcn_*) are passing for asm{0,1,2} and binaryen{0,1,2}.

However since I cannot get full browser tests to pass even on pristine incoming
(1.38.19-2-g77246e0c1 as of today; there are many failures for both Firefox
63.0.1 and Chromium 70.0.3538.67), I did not tried to verify full browser tests
with my patch. Bit I've made sure that

	browser.test_preload_module
	browser.test_dynamic_link

are passing.

"other" kind of tests also do not pass on pristine incoming for me. This
way I did not tried to verify "other" with my patch.

Thanks beforehand,
Kirill

P.S.

This is my first time I do anything with WebAssembly/Emscripten, and only a
second time with JavaScript, so please forgive me if I missed something.

P.P.S.

I can split the patch into smaller steps, if it will help review.

/cc @kripken, @juj, @sbc100, @max99x, @junjihashimoto, @mdboom, @rth
@navytux

This comment has been minimized.

Copy link

navytux commented Nov 15, 2018

So I've implemented emscripten support for linking to shared libraries which in turn are linked to other shared libraries:

emscripten-core/emscripten#7512

This way all *.so scipy modules should be able to link to lapack.so dynamically.
@rth, could you please check whether it helps for scipy/lapack case?

@mdboom, I think this have the potential to also remove freetype and libpng from main pyodide, since they are only used by matplotplib.so module (if I understood correctly).

Thanks,
Kirill

@rth

This comment has been minimized.

Copy link
Member Author

rth commented Nov 15, 2018

Very impressive work @navytux ! Thank you!

For scipy, once #211 is merged I think we can make another PR where we apply your patches to emscripten and see whether this solves the problem.

@navytux

This comment has been minimized.

Copy link

navytux commented Nov 15, 2018

@rth, thanks for feedback.

Just for the reference: the build system of scipy will have to be adjusted to leverage new library -> library linking feature. I mean it won't work automatically with just emscripten ldso patch and current unchanged way to build scipy.

navytux added a commit to navytux/emscripten that referenced this issue Nov 15, 2018

Teach dynamic linking to handle library -> library dependencies
Currently Emscripten allows to create shared libraries (DSO) and link
them to main module. However a shared library itself cannot be linked to
another shared library.

The lack of support for DSO -> DSO linking becomes problematic in cases when
there are several shared libraries that all need to use another should-be
shared functionality, while linking that should-be shared functionality to main
module is not an option for size reasons. My particular use-case is SciPy
support for Pyodide:

	iodide-project/pyodide#211,
	iodide-project/pyodide#240

where several of `*.so` scipy modules need to link to LAPACK. If we link to
LAPACK statically from all those `*.so` - it just blows up compiled size

	iodide-project/pyodide#211 (comment)

and if we link in LAPACK statically to main module, the main module size is
also increased ~2x, which is not an option, since LAPACK functionality is not
needed by every Pyodide user.

This way we are here to add support for DSO -> DSO linking:

1. similarly to how it is already working for main module -> side module
   linking, when building a side module it is now possible to specify

	-s RUNTIME_LINKED_LIBS=[...]

   with list of shared libraries that side module needs to link to.

2. to store that information, for asm.js, similarly to how it is currently
   handled for main module (which always has js part), we transform
   RUNTIME_LINKED_LIBS to

	libModule.dynamicLibraries = [...]

   (see src/preamble_sharedlib.js)

3. for wasm module, in order to store the information about to which libraries
   a module links, we could in theory use "module" attribute in wasm imports.
   However currently emscripten almost always uses just "env" for that "module"
   attribute, e.g.

	(import "env" "abortStackOverflow" (func $fimport$0 (param i32)))
	(import "env" "_ffunc1" (func $fimport$1))
	...

   and this way we have to embed the information about required libraries for
   the dynamic linker somewhere else.

   What I came up with is to extend "dylink" section with information about
   which shared libraries a shared library needs. This is similar to DT_NEEDED
   entries in ELF.

   (see tools/shared.py)

4. then, the dynamic linker (loadDynamicLibrary) is reworked to handle that information:

   - for asm.js, after loading a libModule, we check libModule.dynamicLibraries
     and post-load them recursively. (it would be better to load needed modules
     before the libModule, but for js we currently have to first eval whole
     libModule's code to be able to read .dynamicLibraries)

   - for wasm the needed libraries are loaded before the wasm module in
     question is instantiated.

     (see changes to loadWebAssemblyModule for details)

5. since we also have to teach dlopen() to handle needed libraries, and since dlopen
   was already duplicating loadDynamicLibrary() code in many ways, instead of
   adding more duplication, dlopen is now reworked to use loadDynamicLibrary
   itself.

   This moves functionality to keep track of loaded DSO, their handles,
   refcounts, etc into the dynamic linker itself, with loadDynamicLibrary now
   accepting various flags (global/nodelete) to handle e.g.
   RTLD_LOCAL/RTLD_GLOBAL and RTLD_NODELETE dlopen cases (RTLD_NODELETE
   semantic is needed for initially-linked-in libraries).

   Also, since dlopen was using FS to read libraries, and loadDynamicLibrary was
   previously using Module['read'] and friends, loadDynamicLibrary now also
   accepts fs interface, which if provided, is used as FS-like interface to load
   library data, and if not - native loading capabilities of the environment
   are still used.

   Another aspect related to deduplication is that loadDynamicLibrary now also
   uses preloaded/precompiled wasm modules, that were previously only used by
   dlopen (see a5866a5 "Add preload plugin to compile wasm side modules async
   (emscripten-core#6663)").

   (see changes to dlopen and loadDynamicLibrary)

6. The functionality to asynchronously load dynamic libraries is also
   integrated into loadDynamicLibrary.

   Libraries were asynchronously preloaded for the case when
   Module['readBinary'] is absent (browser, see 3446d2a "preload wasm dynamic
   libraries when we can't load them synchronously").

   Since this codepath was also needed to be taught of DSO -> DSO dependency,
   the most straightforward thing to do was to teach loadDynamicLibrary to do
   its work asynchronously (under flag) and to switch the preloading to use

	loadDynamicLibrary(..., {loadAsync: true})

   (see changes to src/preamble.js and loadDynamicLibrary)

7. A test is added for verifying linking/dlopening a DSO with other needed library.

   browser.test_dynamic_link is also amended to verify linking to DSO with
   dependencies.

With the patch I've made sure that all core tests (including test_dylink_* and
test_dlfcn_*) are passing for asm{0,1,2} and binaryen{0,1,2}.

However since I cannot get full browser tests to pass even on pristine incoming
(1.38.19-2-g77246e0c1 as of today; there are many failures for both Firefox
63.0.1 and Chromium 70.0.3538.67), I did not tried to verify full browser tests
with my patch. Bit I've made sure that

	browser.test_preload_module
	browser.test_dynamic_link

are passing.

"other" kind of tests also do not pass on pristine incoming for me. This
way I did not tried to verify "other" with my patch.

Thanks beforehand,
Kirill

P.S.

This is my first time I do anything with WebAssembly/Emscripten, and only a
second time with JavaScript, so please forgive me if I missed something.

P.P.S.

I can split the patch into smaller steps, if it will help review.

/cc @kripken, @juj, @sbc100, @max99x, @junjihashimoto, @mdboom, @rth
@mdboom

This comment has been minimized.

Copy link
Member

mdboom commented Nov 15, 2018

Thanks for this, @navytux, and welcome!

You are correct that freetype and png (currently statically linked into the "main" package) are only needed by matplotlib, so could separated out. Ideally, freetype and libpng would be their own "packages" specified as dependencies of matplotlib so that other packages down the road could also depend on them. But that is a detail that could be handled in a later pass.

@navytux

This comment has been minimized.

Copy link

navytux commented Nov 15, 2018

@mdboom, thanks for feedback. Yes, let's keep on going and doing things step-by-step, incrementally.

navytux added a commit to navytux/emscripten that referenced this issue Nov 16, 2018

ldso: Teach dynamic linking to handle library -> library dependencies
Currently Emscripten allows to create shared libraries (DSO) and link
them to main module. However a shared library itself cannot be linked to
another shared library.

The lack of support for DSO -> DSO linking becomes problematic in cases when
there are several shared libraries that all need to use another should-be
shared functionality, while linking that should-be shared functionality to main
module is not an option for size reasons. My particular use-case is SciPy
support for Pyodide:

        iodide-project/pyodide#211,
        iodide-project/pyodide#240

where several of `*.so` scipy modules need to link to LAPACK. If we link to
LAPACK statically from all those `*.so` - it just blows up compiled size

        iodide-project/pyodide#211 (comment)

and if we link in LAPACK statically to main module, the main module size is
also increased ~2x, which is not an option, since LAPACK functionality is not
needed by every Pyodide user.

This way we are here to add support for DSO -> DSO linking:

1. similarly to how it is already working for main module -> side module
   linking, when building a side module it is now possible to specify

        -s RUNTIME_LINKED_LIBS=[...]

   with list of shared libraries that side module needs to link to.

2. to store that information, for asm.js, similarly to how it is currently
   handled for main module (which always has js part), we transform
   RUNTIME_LINKED_LIBS to

        libModule.dynamicLibraries = [...]

   (see src/preamble_sharedlib.js)

3. for wasm module, in order to store the information about to which libraries
   a module links, we could in theory use "module" attribute in wasm imports.
   However currently emscripten almost always uses just "env" for that "module"
   attribute, e.g.

        (import "env" "abortStackOverflow" (func $fimport$0 (param i32)))
        (import "env" "_ffunc1" (func $fimport$1))
        ...

   and this way we have to embed the information about required libraries for
   the dynamic linker somewhere else.

   What I came up with is to extend "dylink" section with information about
   which shared libraries a shared library needs. This is similar to DT_NEEDED
   entries in ELF.

   (see tools/shared.py)

4. then, the dynamic linker (loadDynamicLibrary) is reworked to handle that information:

   - for asm.js, after loading a libModule, we check libModule.dynamicLibraries
     and post-load them recursively. (it would be better to load needed modules
     before the libModule, but for js we currently have to first eval whole
     libModule's code to be able to read .dynamicLibraries)

   - for wasm the needed libraries are loaded before the wasm module in
     question is instantiated.

     (see changes to loadWebAssemblyModule for details)

navytux added a commit to navytux/emscripten that referenced this issue Nov 16, 2018

ldso: Teach dynamic linking to handle library -> library dependencies
Currently Emscripten allows to create shared libraries (DSO) and link
them to main module. However a shared library itself cannot be linked to
another shared library.

The lack of support for DSO -> DSO linking becomes problematic in cases when
there are several shared libraries that all need to use another should-be
shared functionality, while linking that should-be shared functionality to main
module is not an option for size reasons. My particular use-case is SciPy
support for Pyodide:

        iodide-project/pyodide#211,
        iodide-project/pyodide#240

where several of `*.so` scipy modules need to link to LAPACK. If we link to
LAPACK statically from all those `*.so` - it just blows up compiled size

        iodide-project/pyodide#211 (comment)

and if we link in LAPACK statically to main module, the main module size is
also increased ~2x, which is not an option, since LAPACK functionality is not
needed by every Pyodide user.

This way we are here to add support for DSO -> DSO linking:

1. similarly to how it is already working for main module -> side module
   linking, when building a side module it is now possible to specify

        -s RUNTIME_LINKED_LIBS=[...]

   with list of shared libraries that side module needs to link to.

2. to store that information, for asm.js, similarly to how it is currently
   handled for main module (which always has js part), we transform
   RUNTIME_LINKED_LIBS to

        libModule.dynamicLibraries = [...]

   (see src/preamble_sharedlib.js)

3. for wasm module, in order to store the information about to which libraries
   a module links, we could in theory use "module" attribute in wasm imports.
   However currently emscripten almost always uses just "env" for that "module"
   attribute, e.g.

        (import "env" "abortStackOverflow" (func $fimport$0 (param i32)))
        (import "env" "_ffunc1" (func $fimport$1))
        ...

   and this way we have to embed the information about required libraries for
   the dynamic linker somewhere else.

   What I came up with is to extend "dylink" section with information about
   which shared libraries a shared library needs. This is similar to DT_NEEDED
   entries in ELF.

   (see tools/shared.py)

4. then, the dynamic linker (loadDynamicLibrary) is reworked to handle that information:

   - for asm.js, after loading a libModule, we check libModule.dynamicLibraries
     and post-load them recursively. (it would be better to load needed modules
     before the libModule, but for js we currently have to first eval whole
     libModule's code to be able to read .dynamicLibraries)

   - for wasm the needed libraries are loaded before the wasm module in
     question is instantiated.

     (see changes to loadWebAssemblyModule for details)

navytux added a commit to navytux/emscripten that referenced this issue Nov 18, 2018

ldso: Teach dynamic linking to handle library -> library dependencies
Currently Emscripten allows to create shared libraries (DSO) and link
them to main module. However a shared library itself cannot be linked to
another shared library.

The lack of support for DSO -> DSO linking becomes problematic in cases when
there are several shared libraries that all need to use another should-be
shared functionality, while linking that should-be shared functionality to main
module is not an option for size reasons. My particular use-case is SciPy
support for Pyodide:

        iodide-project/pyodide#211,
        iodide-project/pyodide#240

where several of `*.so` scipy modules need to link to LAPACK. If we link to
LAPACK statically from all those `*.so` - it just blows up compiled size

        iodide-project/pyodide#211 (comment)

and if we link in LAPACK statically to main module, the main module size is
also increased ~2x, which is not an option, since LAPACK functionality is not
needed by every Pyodide user.

This way we are here to add support for DSO -> DSO linking:

1. similarly to how it is already working for main module -> side module
   linking, when building a side module it is now possible to specify

        -s RUNTIME_LINKED_LIBS=[...]

   with list of shared libraries that side module needs to link to.

2. to store that information, for asm.js, similarly to how it is currently
   handled for main module (which always has js part), we transform
   RUNTIME_LINKED_LIBS to

        libModule.dynamicLibraries = [...]

   (see src/preamble_sharedlib.js)

3. for wasm module, in order to store the information about to which libraries
   a module links, we could in theory use "module" attribute in wasm imports.
   However currently emscripten almost always uses just "env" for that "module"
   attribute, e.g.

        (import "env" "abortStackOverflow" (func $fimport$0 (param i32)))
        (import "env" "_ffunc1" (func $fimport$1))
        ...

   and this way we have to embed the information about required libraries for
   the dynamic linker somewhere else.

   What I came up with is to extend "dylink" section with information about
   which shared libraries a shared library needs. This is similar to DT_NEEDED
   entries in ELF.

   (see tools/shared.py)

4. then, the dynamic linker (loadDynamicLibrary) is reworked to handle that information:

   - for asm.js, after loading a libModule, we check libModule.dynamicLibraries
     and post-load them recursively. (it would be better to load needed modules
     before the libModule, but for js we currently have to first eval whole
     libModule's code to be able to read .dynamicLibraries)

   - for wasm the needed libraries are loaded before the wasm module in
     question is instantiated.

     (see changes to loadWebAssemblyModule for details)

navytux added a commit to navytux/emscripten that referenced this issue Nov 20, 2018

ldso: Teach dynamic linking to handle library -> library dependencies
Currently Emscripten allows to create shared libraries (DSO) and link
them to main module. However a shared library itself cannot be linked to
another shared library.

The lack of support for DSO -> DSO linking becomes problematic in cases when
there are several shared libraries that all need to use another should-be
shared functionality, while linking that should-be shared functionality to main
module is not an option for size reasons. My particular use-case is SciPy
support for Pyodide:

        iodide-project/pyodide#211,
        iodide-project/pyodide#240

where several of `*.so` scipy modules need to link to LAPACK. If we link to
LAPACK statically from all those `*.so` - it just blows up compiled size

        iodide-project/pyodide#211 (comment)

and if we link in LAPACK statically to main module, the main module size is
also increased ~2x, which is not an option, since LAPACK functionality is not
needed by every Pyodide user.

This way we are here to add support for DSO -> DSO linking:

1. similarly to how it is already working for main module -> side module
   linking, when building a side module it is now possible to specify

        -s RUNTIME_LINKED_LIBS=[...]

   with list of shared libraries that side module needs to link to.

2. to store that information, for asm.js, similarly to how it is currently
   handled for main module (which always has js part), we transform
   RUNTIME_LINKED_LIBS to

        libModule.dynamicLibraries = [...]

   (see src/preamble_sharedlib.js)

3. for wasm module, in order to store the information about to which libraries
   a module links, we could in theory use "module" attribute in wasm imports.
   However currently emscripten almost always uses just "env" for that "module"
   attribute, e.g.

        (import "env" "abortStackOverflow" (func $fimport$0 (param i32)))
        (import "env" "_ffunc1" (func $fimport$1))
        ...

   and this way we have to embed the information about required libraries for
   the dynamic linker somewhere else.

   What I came up with is to extend "dylink" section with information about
   which shared libraries a shared library needs. This is similar to DT_NEEDED
   entries in ELF.

   (see tools/shared.py)

4. then, the dynamic linker (loadDynamicLibrary) is reworked to handle that information:

   - for asm.js, after loading a libModule, we check libModule.dynamicLibraries
     and post-load them recursively. (it would be better to load needed modules
     before the libModule, but for js we currently have to first eval whole
     libModule's code to be able to read .dynamicLibraries)

   - for wasm the needed libraries are loaded before the wasm module in
     question is instantiated.

     (see changes to loadWebAssemblyModule for details)

navytux added a commit to navytux/emscripten that referenced this issue Nov 21, 2018

ldso: Teach dynamic linking to handle library -> library dependencies
Currently Emscripten allows to create shared libraries (DSO) and link
them to main module. However a shared library itself cannot be linked to
another shared library.

The lack of support for DSO -> DSO linking becomes problematic in cases when
there are several shared libraries that all need to use another should-be
shared functionality, while linking that should-be shared functionality to main
module is not an option for size reasons. My particular use-case is SciPy
support for Pyodide:

        iodide-project/pyodide#211,
        iodide-project/pyodide#240

where several of `*.so` scipy modules need to link to LAPACK. If we link to
LAPACK statically from all those `*.so` - it just blows up compiled size

        iodide-project/pyodide#211 (comment)

and if we link in LAPACK statically to main module, the main module size is
also increased ~2x, which is not an option, since LAPACK functionality is not
needed by every Pyodide user.

This way we are here to add support for DSO -> DSO linking:

1. similarly to how it is already working for main module -> side module
   linking, when building a side module it is now possible to specify

        -s RUNTIME_LINKED_LIBS=[...]

   with list of shared libraries that side module needs to link to.

2. to store that information, for asm.js, similarly to how it is currently
   handled for main module (which always has js part), we transform
   RUNTIME_LINKED_LIBS to

        libModule.dynamicLibraries = [...]

   (see src/preamble_sharedlib.js)

3. for wasm module, in order to store the information about to which libraries
   a module links, we could in theory use "module" attribute in wasm imports.
   However currently emscripten almost always uses just "env" for that "module"
   attribute, e.g.

        (import "env" "abortStackOverflow" (func $fimport$0 (param i32)))
        (import "env" "_ffunc1" (func $fimport$1))
        ...

   and this way we have to embed the information about required libraries for
   the dynamic linker somewhere else.

   What I came up with is to extend "dylink" section with information about
   which shared libraries a shared library needs. This is similar to DT_NEEDED
   entries in ELF.

   (see tools/shared.py)

4. then, the dynamic linker (loadDynamicLibrary) is reworked to handle that information:

   - for asm.js, after loading a libModule, we check libModule.dynamicLibraries
     and post-load them recursively. (it would be better to load needed modules
     before the libModule, but for js we currently have to first eval whole
     libModule's code to be able to read .dynamicLibraries)

   - for wasm the needed libraries are loaded before the wasm module in
     question is instantiated.

     (see changes to loadWebAssemblyModule for details)

navytux added a commit to navytux/emscripten that referenced this issue Nov 27, 2018

ldso: Teach dynamic linking to handle library -> library dependencies
Currently Emscripten allows to create shared libraries (DSO) and link
them to main module. However a shared library itself cannot be linked to
another shared library.

The lack of support for DSO -> DSO linking becomes problematic in cases when
there are several shared libraries that all need to use another should-be
shared functionality, while linking that should-be shared functionality to main
module is not an option for size reasons. My particular use-case is SciPy
support for Pyodide:

        iodide-project/pyodide#211,
        iodide-project/pyodide#240

where several of `*.so` scipy modules need to link to LAPACK. If we link to
LAPACK statically from all those `*.so` - it just blows up compiled size

        iodide-project/pyodide#211 (comment)

and if we link in LAPACK statically to main module, the main module size is
also increased ~2x, which is not an option, since LAPACK functionality is not
needed by every Pyodide user.

This way we are here to add support for DSO -> DSO linking:

1. similarly to how it is already working for main module -> side module
   linking, when building a side module it is now possible to specify

        -s RUNTIME_LINKED_LIBS=[...]

   with list of shared libraries that side module needs to link to.

2. to store that information, for asm.js, similarly to how it is currently
   handled for main module (which always has js part), we transform
   RUNTIME_LINKED_LIBS to

        libModule.dynamicLibraries = [...]

   (see src/preamble_sharedlib.js)

3. for wasm module, in order to store the information about to which libraries
   a module links, we could in theory use "module" attribute in wasm imports.
   However currently emscripten almost always uses just "env" for that "module"
   attribute, e.g.

        (import "env" "abortStackOverflow" (func $fimport$0 (param i32)))
        (import "env" "_ffunc1" (func $fimport$1))
        ...

   and this way we have to embed the information about required libraries for
   the dynamic linker somewhere else.

   What I came up with is to extend "dylink" section with information about
   which shared libraries a shared library needs. This is similar to DT_NEEDED
   entries in ELF.

   (see tools/shared.py)

4. then, the dynamic linker (loadDynamicLibrary) is reworked to handle that information:

   - for asm.js, after loading a libModule, we check libModule.dynamicLibraries
     and post-load them recursively. (it would be better to load needed modules
     before the libModule, but for js we currently have to first eval whole
     libModule's code to be able to read .dynamicLibraries)

   - for wasm the needed libraries are loaded before the wasm module in
     question is instantiated.

     (see changes to loadWebAssemblyModule for details)

navytux added a commit to navytux/emscripten that referenced this issue Nov 29, 2018

ldso: Teach dynamic linking to handle library -> library dependencies
Currently Emscripten allows to create shared libraries (DSO) and link
them to main module. However a shared library itself cannot be linked to
another shared library.

The lack of support for DSO -> DSO linking becomes problematic in cases when
there are several shared libraries that all need to use another should-be
shared functionality, while linking that should-be shared functionality to main
module is not an option for size reasons. My particular use-case is SciPy
support for Pyodide:

        iodide-project/pyodide#211,
        iodide-project/pyodide#240

where several of `*.so` scipy modules need to link to LAPACK. If we link to
LAPACK statically from all those `*.so` - it just blows up compiled size

        iodide-project/pyodide#211 (comment)

and if we link in LAPACK statically to main module, the main module size is
also increased ~2x, which is not an option, since LAPACK functionality is not
needed by every Pyodide user.

This way we are here to add support for DSO -> DSO linking:

1. similarly to how it is already working for main module -> side module
   linking, when building a side module it is now possible to specify

        -s RUNTIME_LINKED_LIBS=[...]

   with list of shared libraries that side module needs to link to.

2. to store that information, for asm.js, similarly to how it is currently
   handled for main module (which always has js part), we transform
   RUNTIME_LINKED_LIBS to

        libModule.dynamicLibraries = [...]

   (see src/preamble_sharedlib.js)

3. for wasm module, in order to store the information about to which libraries
   a module links, we could in theory use "module" attribute in wasm imports.
   However currently emscripten almost always uses just "env" for that "module"
   attribute, e.g.

        (import "env" "abortStackOverflow" (func $fimport$0 (param i32)))
        (import "env" "_ffunc1" (func $fimport$1))
        ...

   and this way we have to embed the information about required libraries for
   the dynamic linker somewhere else.

   What I came up with is to extend "dylink" section with information about
   which shared libraries a shared library needs. This is similar to DT_NEEDED
   entries in ELF.

   (see tools/shared.py)

4. then, the dynamic linker (loadDynamicLibrary) is reworked to handle that information:

   - for asm.js, after loading a libModule, we check libModule.dynamicLibraries
     and post-load them recursively. (it would be better to load needed modules
     before the libModule, but for js we currently have to first eval whole
     libModule's code to be able to read .dynamicLibraries)

   - for wasm the needed libraries are loaded before the wasm module in
     question is instantiated.

     (see changes to loadWebAssemblyModule for details)

navytux added a commit to navytux/emscripten that referenced this issue Nov 29, 2018

ldso: Teach dynamic linking to handle library -> library dependencies
Currently Emscripten allows to create shared libraries (DSO) and link
them to main module. However a shared library itself cannot be linked to
another shared library.

The lack of support for DSO -> DSO linking becomes problematic in cases when
there are several shared libraries that all need to use another should-be
shared functionality, while linking that should-be shared functionality to main
module is not an option for size reasons. My particular use-case is SciPy
support for Pyodide:

        iodide-project/pyodide#211,
        iodide-project/pyodide#240

where several of `*.so` scipy modules need to link to LAPACK. If we link to
LAPACK statically from all those `*.so` - it just blows up compiled size

        iodide-project/pyodide#211 (comment)

and if we link in LAPACK statically to main module, the main module size is
also increased ~2x, which is not an option, since LAPACK functionality is not
needed by every Pyodide user.

This way we are here to add support for DSO -> DSO linking:

1. similarly to how it is already working for main module -> side module
   linking, when building a side module it is now possible to specify

        -s RUNTIME_LINKED_LIBS=[...]

   with list of shared libraries that side module needs to link to.

2. to store that information, for asm.js, similarly to how it is currently
   handled for main module (which always has js part), we transform
   RUNTIME_LINKED_LIBS to

        libModule.dynamicLibraries = [...]

   (see src/preamble_sharedlib.js)

3. for wasm module, in order to store the information about to which libraries
   a module links, we could in theory use "module" attribute in wasm imports.
   However currently emscripten almost always uses just "env" for that "module"
   attribute, e.g.

        (import "env" "abortStackOverflow" (func $fimport$0 (param i32)))
        (import "env" "_ffunc1" (func $fimport$1))
        ...

   and this way we have to embed the information about required libraries for
   the dynamic linker somewhere else.

   What I came up with is to extend "dylink" section with information about
   which shared libraries a shared library needs. This is similar to DT_NEEDED
   entries in ELF.

   (see tools/shared.py)

4. then, the dynamic linker (loadDynamicLibrary) is reworked to handle that information:

   - for asm.js, after loading a libModule, we check libModule.dynamicLibraries
     and post-load them recursively. (it would be better to load needed modules
     before the libModule, but for js we currently have to first eval whole
     libModule's code to be able to read .dynamicLibraries)

   - for wasm the needed libraries are loaded before the wasm module in
     question is instantiated.

     (see changes to loadWebAssemblyModule for details)

navytux added a commit to navytux/emscripten that referenced this issue Dec 3, 2018

ldso: Teach dynamic linking to handle library -> library dependencies
Currently Emscripten allows to create shared libraries (DSO) and link
them to main module. However a shared library itself cannot be linked to
another shared library.

The lack of support for DSO -> DSO linking becomes problematic in cases when
there are several shared libraries that all need to use another should-be
shared functionality, while linking that should-be shared functionality to main
module is not an option for size reasons. My particular use-case is SciPy
support for Pyodide:

        iodide-project/pyodide#211,
        iodide-project/pyodide#240

where several of `*.so` scipy modules need to link to LAPACK. If we link to
LAPACK statically from all those `*.so` - it just blows up compiled size

        iodide-project/pyodide#211 (comment)

and if we link in LAPACK statically to main module, the main module size is
also increased ~2x, which is not an option, since LAPACK functionality is not
needed by every Pyodide user.

This way we are here to add support for DSO -> DSO linking:

1. similarly to how it is already working for main module -> side module
   linking, when building a side module it is now possible to specify

        -s RUNTIME_LINKED_LIBS=[...]

   with list of shared libraries that side module needs to link to.

2. to store that information, for asm.js, similarly to how it is currently
   handled for main module (which always has js part), we transform
   RUNTIME_LINKED_LIBS to

        libModule.dynamicLibraries = [...]

   (see src/preamble_sharedlib.js)

3. for wasm module, in order to store the information about to which libraries
   a module links, we could in theory use "module" attribute in wasm imports.
   However currently emscripten almost always uses just "env" for that "module"
   attribute, e.g.

        (import "env" "abortStackOverflow" (func $fimport$0 (param i32)))
        (import "env" "_ffunc1" (func $fimport$1))
        ...

   and this way we have to embed the information about required libraries for
   the dynamic linker somewhere else.

   What I came up with is to extend "dylink" section with information about
   which shared libraries a shared library needs. This is similar to DT_NEEDED
   entries in ELF.

   (see tools/shared.py)

4. then, the dynamic linker (loadDynamicLibrary) is reworked to handle that information:

   - for asm.js, after loading a libModule, we check libModule.dynamicLibraries
     and post-load them recursively. (it would be better to load needed modules
     before the libModule, but for js we currently have to first eval whole
     libModule's code to be able to read .dynamicLibraries)

   - for wasm the needed libraries are loaded before the wasm module in
     question is instantiated.

     (see changes to loadWebAssemblyModule for details)

@rth rth referenced this issue Dec 3, 2018

Merged

Package scikit-learn #226

navytux added a commit to navytux/emscripten that referenced this issue Dec 5, 2018

ldso: Teach dynamic linking to handle library -> library dependencies
Currently Emscripten allows to create shared libraries (DSO) and link
them to main module. However a shared library itself cannot be linked to
another shared library.

The lack of support for DSO -> DSO linking becomes problematic in cases when
there are several shared libraries that all need to use another should-be
shared functionality, while linking that should-be shared functionality to main
module is not an option for size reasons. My particular use-case is SciPy
support for Pyodide:

        iodide-project/pyodide#211,
        iodide-project/pyodide#240

where several of `*.so` scipy modules need to link to LAPACK. If we link to
LAPACK statically from all those `*.so` - it just blows up compiled size

        iodide-project/pyodide#211 (comment)

and if we link in LAPACK statically to main module, the main module size is
also increased ~2x, which is not an option, since LAPACK functionality is not
needed by every Pyodide user.

This way we are here to add support for DSO -> DSO linking:

1. similarly to how it is already working for main module -> side module
   linking, when building a side module it is now possible to specify

        -s RUNTIME_LINKED_LIBS=[...]

   with list of shared libraries that side module needs to link to.

2. to store that information, for asm.js, similarly to how it is currently
   handled for main module (which always has js part), we transform
   RUNTIME_LINKED_LIBS to

        libModule.dynamicLibraries = [...]

   (see src/preamble_sharedlib.js)

3. for wasm module, in order to store the information about to which libraries
   a module links, we could in theory use "module" attribute in wasm imports.
   However currently emscripten almost always uses just "env" for that "module"
   attribute, e.g.

        (import "env" "abortStackOverflow" (func $fimport$0 (param i32)))
        (import "env" "_ffunc1" (func $fimport$1))
        ...

   and this way we have to embed the information about required libraries for
   the dynamic linker somewhere else.

   What I came up with is to extend "dylink" section with information about
   which shared libraries a shared library needs. This is similar to DT_NEEDED
   entries in ELF.

   (see tools/shared.py)

4. then, the dynamic linker (loadDynamicLibrary) is reworked to handle that information:

   - for asm.js, after loading a libModule, we check libModule.dynamicLibraries
     and post-load them recursively. (it would be better to load needed modules
     before the libModule, but for js we currently have to first eval whole
     libModule's code to be able to read .dynamicLibraries)

   - for wasm the needed libraries are loaded before the wasm module in
     question is instantiated.

     (see changes to loadWebAssemblyModule for details)

navytux added a commit to navytux/emscripten that referenced this issue Dec 5, 2018

ldso: Teach dynamic linking to handle library -> library dependencies
Currently Emscripten allows to create shared libraries (DSO) and link
them to main module. However a shared library itself cannot be linked to
another shared library.

The lack of support for DSO -> DSO linking becomes problematic in cases when
there are several shared libraries that all need to use another should-be
shared functionality, while linking that should-be shared functionality to main
module is not an option for size reasons. My particular use-case is SciPy
support for Pyodide:

        iodide-project/pyodide#211,
        iodide-project/pyodide#240

where several of `*.so` scipy modules need to link to LAPACK. If we link to
LAPACK statically from all those `*.so` - it just blows up compiled size

        iodide-project/pyodide#211 (comment)

and if we link in LAPACK statically to main module, the main module size is
also increased ~2x, which is not an option, since LAPACK functionality is not
needed by every Pyodide user.

This way we are here to add support for DSO -> DSO linking:

1. similarly to how it is already working for main module -> side module
   linking, when building a side module it is now possible to specify

        -s RUNTIME_LINKED_LIBS=[...]

   with list of shared libraries that side module needs to link to.

2. to store that information, for asm.js, similarly to how it is currently
   handled for main module (which always has js part), we transform
   RUNTIME_LINKED_LIBS to

        libModule.dynamicLibraries = [...]

   (see src/preamble_sharedlib.js)

3. for wasm module, in order to store the information about to which libraries
   a module links, we could in theory use "module" attribute in wasm imports.
   However currently emscripten almost always uses just "env" for that "module"
   attribute, e.g.

        (import "env" "abortStackOverflow" (func $fimport$0 (param i32)))
        (import "env" "_ffunc1" (func $fimport$1))
        ...

   and this way we have to embed the information about required libraries for
   the dynamic linker somewhere else.

   What I came up with is to extend "dylink" section with information about
   which shared libraries a shared library needs. This is similar to DT_NEEDED
   entries in ELF.

   (see tools/shared.py)

4. then, the dynamic linker (loadDynamicLibrary) is reworked to handle that information:

   - for asm.js, after loading a libModule, we check libModule.dynamicLibraries
     and post-load them recursively. (it would be better to load needed modules
     before the libModule, but for js we currently have to first eval whole
     libModule's code to be able to read .dynamicLibraries)

   - for wasm the needed libraries are loaded before the wasm module in
     question is instantiated.

     (see changes to loadWebAssemblyModule for details)

navytux added a commit to navytux/emscripten that referenced this issue Dec 5, 2018

ldso: Teach dynamic linking to handle library -> library dependencies
Currently Emscripten allows to create shared libraries (DSO) and link
them to main module. However a shared library itself cannot be linked to
another shared library.

The lack of support for DSO -> DSO linking becomes problematic in cases when
there are several shared libraries that all need to use another should-be
shared functionality, while linking that should-be shared functionality to main
module is not an option for size reasons. My particular use-case is SciPy
support for Pyodide:

        iodide-project/pyodide#211,
        iodide-project/pyodide#240

where several of `*.so` scipy modules need to link to LAPACK. If we link to
LAPACK statically from all those `*.so` - it just blows up compiled size

        iodide-project/pyodide#211 (comment)

and if we link in LAPACK statically to main module, the main module size is
also increased ~2x, which is not an option, since LAPACK functionality is not
needed by every Pyodide user.

This way we are here to add support for DSO -> DSO linking:

1. similarly to how it is already working for main module -> side module
   linking, when building a side module it is now possible to specify

        -s RUNTIME_LINKED_LIBS=[...]

   with list of shared libraries that side module needs to link to.

2. to store that information, for asm.js, similarly to how it is currently
   handled for main module (which always has js part), we transform
   RUNTIME_LINKED_LIBS to

        libModule.dynamicLibraries = [...]

   (see src/preamble_sharedlib.js)

3. for wasm module, in order to store the information about to which libraries
   a module links, we could in theory use "module" attribute in wasm imports.
   However currently emscripten almost always uses just "env" for that "module"
   attribute, e.g.

        (import "env" "abortStackOverflow" (func $fimport$0 (param i32)))
        (import "env" "_ffunc1" (func $fimport$1))
        ...

   and this way we have to embed the information about required libraries for
   the dynamic linker somewhere else.

   What I came up with is to extend "dylink" section with information about
   which shared libraries a shared library needs. This is similar to DT_NEEDED
   entries in ELF.

   (see tools/shared.py)

4. then, the dynamic linker (loadDynamicLibrary) is reworked to handle that information:

   - for asm.js, after loading a libModule, we check libModule.dynamicLibraries
     and post-load them recursively. (it would be better to load needed modules
     before the libModule, but for js we currently have to first eval whole
     libModule's code to be able to read .dynamicLibraries)

   - for wasm the needed libraries are loaded before the wasm module in
     question is instantiated.

     (see changes to loadWebAssemblyModule for details)

navytux added a commit to navytux/emscripten that referenced this issue Dec 6, 2018

ldso: Teach dynamic linking to handle library -> library dependencies
Currently Emscripten allows to create shared libraries (DSO) and link
them to main module. However a shared library itself cannot be linked to
another shared library.

The lack of support for DSO -> DSO linking becomes problematic in cases when
there are several shared libraries that all need to use another should-be
shared functionality, while linking that should-be shared functionality to main
module is not an option for size reasons. My particular use-case is SciPy
support for Pyodide:

        iodide-project/pyodide#211,
        iodide-project/pyodide#240

where several of `*.so` scipy modules need to link to LAPACK. If we link to
LAPACK statically from all those `*.so` - it just blows up compiled size

        iodide-project/pyodide#211 (comment)

and if we link in LAPACK statically to main module, the main module size is
also increased ~2x, which is not an option, since LAPACK functionality is not
needed by every Pyodide user.

This way we are here to add support for DSO -> DSO linking:

1. similarly to how it is already working for main module -> side module
   linking, when building a side module it is now possible to specify

        -s RUNTIME_LINKED_LIBS=[...]

   with list of shared libraries that side module needs to link to.

2. to store that information, for asm.js, similarly to how it is currently
   handled for main module (which always has js part), we transform
   RUNTIME_LINKED_LIBS to

        libModule.dynamicLibraries = [...]

   (see src/preamble_sharedlib.js)

3. for wasm module, in order to store the information about to which libraries
   a module links, we could in theory use "module" attribute in wasm imports.
   However currently emscripten almost always uses just "env" for that "module"
   attribute, e.g.

        (import "env" "abortStackOverflow" (func $fimport$0 (param i32)))
        (import "env" "_ffunc1" (func $fimport$1))
        ...

   and this way we have to embed the information about required libraries for
   the dynamic linker somewhere else.

   What I came up with is to extend "dylink" section with information about
   which shared libraries a shared library needs. This is similar to DT_NEEDED
   entries in ELF.

   (see tools/shared.py)

4. then, the dynamic linker (loadDynamicLibrary) is reworked to handle that information:

   - for asm.js, after loading a libModule, we check libModule.dynamicLibraries
     and post-load them recursively. (it would be better to load needed modules
     before the libModule, but for js we currently have to first eval whole
     libModule's code to be able to read .dynamicLibraries)

   - for wasm the needed libraries are loaded before the wasm module in
     question is instantiated.

     (see changes to loadWebAssemblyModule for details)

navytux added a commit to navytux/emscripten that referenced this issue Dec 8, 2018

ldso: Teach dynamic linking to handle library -> library dependencies
Currently Emscripten allows to create shared libraries (DSO) and link
them to main module. However a shared library itself cannot be linked to
another shared library.

The lack of support for DSO -> DSO linking becomes problematic in cases when
there are several shared libraries that all need to use another should-be
shared functionality, while linking that should-be shared functionality to main
module is not an option for size reasons. My particular use-case is SciPy
support for Pyodide:

        iodide-project/pyodide#211,
        iodide-project/pyodide#240

where several of `*.so` scipy modules need to link to LAPACK. If we link to
LAPACK statically from all those `*.so` - it just blows up compiled size

        iodide-project/pyodide#211 (comment)

and if we link in LAPACK statically to main module, the main module size is
also increased ~2x, which is not an option, since LAPACK functionality is not
needed by every Pyodide user.

This way we are here to add support for DSO -> DSO linking:

1. similarly to how it is already working for main module -> side module
   linking, when building a side module it is now possible to specify

        -s RUNTIME_LINKED_LIBS=[...]

   with list of shared libraries that side module needs to link to.

2. to store that information, for asm.js, similarly to how it is currently
   handled for main module (which always has js part), we transform
   RUNTIME_LINKED_LIBS to

        libModule.dynamicLibraries = [...]

   (see src/preamble_sharedlib.js)

3. for wasm module, in order to store the information about to which libraries
   a module links, we could in theory use "module" attribute in wasm imports.
   However currently emscripten almost always uses just "env" for that "module"
   attribute, e.g.

        (import "env" "abortStackOverflow" (func $fimport$0 (param i32)))
        (import "env" "_ffunc1" (func $fimport$1))
        ...

   and this way we have to embed the information about required libraries for
   the dynamic linker somewhere else.

   What I came up with is to extend "dylink" section with information about
   which shared libraries a shared library needs. This is similar to DT_NEEDED
   entries in ELF.

   (see tools/shared.py)

4. then, the dynamic linker (loadDynamicLibrary) is reworked to handle that information:

   - for asm.js, after loading a libModule, we check libModule.dynamicLibraries
     and post-load them recursively. (it would be better to load needed modules
     before the libModule, but for js we currently have to first eval whole
     libModule's code to be able to read .dynamicLibraries)

   - for wasm the needed libraries are loaded before the wasm module in
     question is instantiated.

     (see changes to loadWebAssemblyModule for details)

kripken added a commit to emscripten-core/emscripten that referenced this issue Dec 9, 2018

ldso: Teach dynamic linking to handle library -> library dependencies
Currently Emscripten allows to create shared libraries (DSO) and link
them to main module. However a shared library itself cannot be linked to
another shared library.

The lack of support for DSO -> DSO linking becomes problematic in cases when
there are several shared libraries that all need to use another should-be
shared functionality, while linking that should-be shared functionality to main
module is not an option for size reasons. My particular use-case is SciPy
support for Pyodide:

        iodide-project/pyodide#211,
        iodide-project/pyodide#240

where several of `*.so` scipy modules need to link to LAPACK. If we link to
LAPACK statically from all those `*.so` - it just blows up compiled size

        iodide-project/pyodide#211 (comment)

and if we link in LAPACK statically to main module, the main module size is
also increased ~2x, which is not an option, since LAPACK functionality is not
needed by every Pyodide user.

This way we are here to add support for DSO -> DSO linking:

1. similarly to how it is already working for main module -> side module
   linking, when building a side module it is now possible to specify

        -s RUNTIME_LINKED_LIBS=[...]

   with list of shared libraries that side module needs to link to.

2. to store that information, for asm.js, similarly to how it is currently
   handled for main module (which always has js part), we transform
   RUNTIME_LINKED_LIBS to

        libModule.dynamicLibraries = [...]

   (see src/preamble_sharedlib.js)

3. for wasm module, in order to store the information about to which libraries
   a module links, we could in theory use "module" attribute in wasm imports.
   However currently emscripten almost always uses just "env" for that "module"
   attribute, e.g.

        (import "env" "abortStackOverflow" (func $fimport$0 (param i32)))
        (import "env" "_ffunc1" (func $fimport$1))
        ...

   and this way we have to embed the information about required libraries for
   the dynamic linker somewhere else.

   What I came up with is to extend "dylink" section with information about
   which shared libraries a shared library needs. This is similar to DT_NEEDED
   entries in ELF.

   (see tools/shared.py)

4. then, the dynamic linker (loadDynamicLibrary) is reworked to handle that information:

   - for asm.js, after loading a libModule, we check libModule.dynamicLibraries
     and post-load them recursively. (it would be better to load needed modules
     before the libModule, but for js we currently have to first eval whole
     libModule's code to be able to read .dynamicLibraries)

   - for wasm the needed libraries are loaded before the wasm module in
     question is instantiated.

     (see changes to loadWebAssemblyModule for details)
@navytux

This comment has been minimized.

Copy link

navytux commented Dec 12, 2018

My patches for DSO -> DSO linking landed into Emscripten incoming and should be hopefully available as part of upcoming Emscripten 1.38.22 release.

@rth, it would be interesting to see how it helps scipy packaging. You can see how to use the linking e.g. here: https://lab.nexedi.com/kirr/wasm-study/blob/de232252f8/Makefile#L34.

@mdboom, I also have the feeling that after the fix for concurrent loading of wasm modules, preloading them serially (1, 2) is no longer required. I've actually had to do the fix while testing concurrent DSO loading under browser environment: https://github.com/kripken/emscripten/blob/1.38.21-27-g9ecca00df/tests/runner.py#L799, and this way it should for sure help preloading too. But I had not tried myself.

Kirill

@mdboom

This comment has been minimized.

Copy link
Member

mdboom commented Dec 12, 2018

Thanks for all this great work!

I created #278 to remind us to do the emscripten upgrade when it's ready. Then we can come back to this issue and rework the Scipy build to use dynamic linking against BLAS and watch our file sizes shrink.

I've also created #279 to remind us to come back and look into the serial wasm module loading later.

@rth

This comment has been minimized.

Copy link
Member Author

rth commented Dec 12, 2018

Thanks a lot for your hard work on this @navytux ! I'm really glad DSO -> DSO linking works in Emscritpen now.

Yes, as soon as Emscripten 1.38.22 is out and we migrate to it, we'll have a look at scipy, and will keep you informed of how it goes.

@navytux

This comment has been minimized.

Copy link

navytux commented Dec 13, 2018

@mdboom, @rth, thanks for feedback and kind words. Good luck with the tries.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment