Dynamic linking of BLAS / LAPACK

[rth]

This was discussed in #211 I would like to reformulate the issue so it's easier for someone to continue working on it.

Scipy requires BLAS and LAPACK to build. Note: my earlier comments in #211 (comment) about LAPACK being optional due to the note at the and of this paragraph in scipy docs were actually not true, as was confirmed on the scipy-dev mailing list.

Currently in #211 we link LAPACK statically, which means that it gets included in around ~8 compiled C extensions , with a resulting size of 170MB lz4 compressed / 650MB uncompressed. Different compile flags where tried in #211 but ultimately it doesn't improve size much.

The most clean way to improve the package size would be to link dynamically, which would also allow to get rid of some hacks in pyodide_build/pywasmcross.py (#211 (comment)). More information about dynamic linking in emscripten can be found in https://github.com/kripken/emscripten/wiki/Linking

Some of this was tried in #211 (comment) but didn't succeed. In retrospective, disabling validation of output in emcc / asm2wasm could be a first step. Then one would need to manually open the needed modules with dlopen, since ldd does not exist in emscripten (#211 (comment)).

[navytux]

Some feedback about dynamic linking from wasm/emscripten newbie discovered while learning it:

It is possible in emscripten to do dynamic linking not only from main module to other libraries, but from those other libraries to next other dynamic modules. (In scipy case the main module is main pyiodide, a DSO is a *.so from scipy, and those DSO want to link to e.g. liblapack.so). The way it works is that a Module in emscripten can have .dynamicLibraries, and then the runtime automatically loads referenced libraries. Today this works for main module already (one uses e.g. -s RUNTIME_LINKED_LIBS="...") but does not automatically work from sided DSO to another DSO. However even today it could be worked around if DSO A wants to link to DSO B e.g. via

// (liba.c)
__attribute__((constructor))
static void init() {
   // XXX EM_ASM does not work in wasm mode
#  if 1
   EM_ASM({
       loadDynamicLibrary('libB_so.js');
   });
#  else
     emscripten_run_script("loadDynamicLibrary('libB_so.wasm')");
#  endif
}

this just calls loadDynamicLibrary from emscripten runtime to do the job.

It works but is not good for real usage, as if e.g. A1 and A2 both would want to load B, then B would be loaded twice. It seems to be not too hard to fix in emscripten runtime however - please see notes here:

https://lab.nexedi.com/kirr/emscripten/commit/98006640

This way conventional support for shared libraries could be implemented in emscripten - similar to what we are used to have e.g. on Linux and other OS'es.

I've put a full test/demo for such linking here:

https://lab.nexedi.com/kirr/wasm-study (rev ee35d772 as of this writing).

Kirill

[navytux]

So I've implemented emscripten support for linking to shared libraries which in turn are linked to other shared libraries:

emscripten-core/emscripten#7512

This way all *.so scipy modules should be able to link to lapack.so dynamically.
@rth, could you please check whether it helps for scipy/lapack case?

@mdboom, I think this have the potential to also remove freetype and libpng from main pyodide, since they are only used by matplotplib.so module (if I understood correctly).

Thanks,
Kirill

[rth]

Very impressive work @navytux ! Thank you!

For scipy, once #211 is merged I think we can make another PR where we apply your patches to emscripten and see whether this solves the problem.

[navytux]

@rth, thanks for feedback.

Just for the reference: the build system of scipy will have to be adjusted to leverage new library -> library linking feature. I mean it won't work automatically with just emscripten ldso patch and current unchanged way to build scipy.

[mdboom]

Thanks for this, @navytux, and welcome!

You are correct that freetype and png (currently statically linked into the "main" package) are only needed by matplotlib, so could separated out. Ideally, freetype and libpng would be their own "packages" specified as dependencies of matplotlib so that other packages down the road could also depend on them. But that is a detail that could be handled in a later pass.

[navytux]

@mdboom, thanks for feedback. Yes, let's keep on going and doing things step-by-step, incrementally.

[navytux]

My patches for DSO -> DSO linking landed into Emscripten incoming and should be hopefully available as part of upcoming Emscripten 1.38.22 release.

@rth, it would be interesting to see how it helps scipy packaging. You can see how to use the linking e.g. here: https://lab.nexedi.com/kirr/wasm-study/blob/de232252f8/Makefile#L34.

@mdboom, I also have the feeling that after the fix for concurrent loading of wasm modules, preloading them serially (1, 2) is no longer required. I've actually had to do the fix while testing concurrent DSO loading under browser environment: https://github.com/kripken/emscripten/blob/1.38.21-27-g9ecca00df/tests/runner.py#L799, and this way it should for sure help preloading too. But I had not tried myself.

Kirill

[mdboom]

Thanks for all this great work!

I created #278 to remind us to do the emscripten upgrade when it's ready. Then we can come back to this issue and rework the Scipy build to use dynamic linking against BLAS and watch our file sizes shrink.

I've also created #279 to remind us to come back and look into the serial wasm module loading later.

[rth]

Thanks a lot for your hard work on this @navytux ! I'm really glad DSO -> DSO linking works in Emscritpen now.

Yes, as soon as Emscripten 1.38.22 is out and we migrate to it, we'll have a look at scipy, and will keep you informed of how it goes.

[navytux]

@mdboom, @rth, thanks for feedback and kind words. Good luck with the tries.