Upgrade to emscripten 1.38.34 with fastcomp

[rth]

This aims to upgrade the build system to use emscripten 1.36.36 (still with fastcomp), which should help with moving to the upstream llvm backend in #476 (using the same emscrtipten version).

One challenge is that previously we used emscripten, binaryen, patched clang of the same emsdk version, built from source to which additional patches were applied.

Currently, the latest available tags in emsdk are,

• 1.38.36 [default (fastcomp) backend]
• 1.38.36-upstream [upstream LLVM wasm backend]
• sdk-tag-1.38.31-64bit
• clang-e1.38.31-64bit
• binaryen-tag-1.38.31-64bit
• emscripten-tag-1.38.31-64bit

when we use the 1.38.36 tag, we do get that version of emscripten but then it's unclear whether previous versions of binaryen should be used (or if there is another way of setting NUM_PARAMS for FuncCastEmulation).

Here, I tried the 1.38.36 tag (emscripten) + binaryen-tag-1.38.31-64bit and I tried removing the clang-e1.38.31-64bit tag (since anyway we wouldn't be able to patch it with upstream llvm). This builds (locally) but fails to load for tests.

Will try re-adding clang tag, next.

cc @mdboom

[rth]

@mdboom Any thoughts on this, or on how to proceed with the migration in #476 ?

[mdboom]

No updates -- I had meant to look at this last week, but got derailed on other things...

[mdboom]

I'm getting a little farther by moving this to emscripten 1.38.38... Will report more later.

[mdboom]

Getting an error about needing -fPIC, so recompiling with that...

[rth]

Closing in favor of #531

[rth]

This is another attempt to update emscripten version by small increments.

Current situation is that building CPython works (at least locally for me), but loading pyodide fails in tests which needs more investigation. Once CPython is working, it might be necessary to manually install binaryen as we did before to re-apply the emsdk/patches/num_params.patch. I really hope it can work without emsdk/patches/callHandlers.patch as otherwise it means that we need to build fastcomp-clang from sources as we did before which takes a while.

[SimonBiggs]

I believe the callHandlers.patch was merged into 1.38.31. See emscripten-core/emscripten-fastcomp#256

[rth]

This should be a bit easier now that update to 1.38.31 was made in #674

[dalcde]

The error doesn't seem to be a numpy error. It happens with all C libraries. For example, the following also fail:

import regex
from lxml import etree

[rth]

The error doesn't seem to be a numpy error. It happens with all C libraries.

Ahh, thank you that's very helpful! In particular, regex should be much easier to debug as it has only one C extension (lxml less so).

[dalcde]

I am very confused about why this happens. Running wasm-objdump on the .so file shows that the same symbols are exported. On the other hand, as far as I can tell, python is looking at the right file, because if I delete the file, python complains in a different way. There doesn't seem to be much room in between for things to go wrong.

[rth]

FYI, lxml import occasionally fails on master as well, so that would not be a reliable package to use for debugging here. I'll need to re-build this PR to investigate.

[dalcde]

I inserted the following snippet into src/main.c:

  void *handle = dlopen("/lib/python3.8/site-packages/regex/_regex.so", RTLD_LAZY);
  if (handle == NULL) {
    printf("Cannot dlopen\n");
  } else {
    printf("Can dlopen\n");
  }

  void *sym = dlsym(handle, "PyInit__regex");
  if (sym == NULL) {
    char* err = dlerror();
    if (err != NULL) {
      printf("%s\n", err);
      printf("Cannot dlsym\n");
    } else {
      printf("Loaded symbol is NULL\n");
    }
  } else {
    printf("Can dlsym\n");
  }

and forced the js bits to load regex.js early on. What I get is the following:

Can dlopen
Loaded symbol is NULL
Python initialization complete

In other words, both dlopen and dlsym are successful, but the symbol returned is the null pointer. I'm not very well-versed in wasm itself, but it seems to me that in _regex.so, PyInit__regex is not the null pointer, so the mystery continues. (EDIT: changing RTLD_LAZY to RTLD_NOW does not help)

EDIT 2: On the javascript side, the symbol _PyInit__regex is in the list of exports.
EDIT 3: The problem is on the dlsym side. Copying regex.{js,data} compiled with 1.38.34 to pyodide compiled with 1.38.31 works, but copying regex.{js,data} compiled with 1.38.31 to pyodide compiled with 1.38.34 does not. Attempting to load other symbols in the snippet above also return NULL.
EDIT4: dlsym works in a simple test case using our patched emscripten.

[dalcde]

The problem is identified and fixed in rth#1 . Feel free to merge, cherry pick, and/or squash it into your PR.

[rth]

Great work @dalcde, thank you!

[rth]

Thanks so much @dalcde ! The rest of the CI was fine. I'm truly happy about this being merged. That typo in emscripten was really a blocker. Now we can try to incrementally update to the latest emscripten version #476 (comment)