Compiled WASM not supported in node 14 or earlier

[danez]

We just upgraded from version 1.39.18 to 3.1.18 of emscripten and now the wasm cannot be loaded anymore node 14 as well as node 12. It works in node 16 and up. I'm totally clueless why emscripten suddenly produces wasm that cannot be used in node 14 or earlier.

Is there a way to disable atomics in emscripten?

I see that the atomics in my error are coming from the library re2, but this was exactly the same with emscripten@1.39.18 and this older version did not produce a wasm file with atomics in it. And obviously I cannot change external libraries.

Version of emscripten/emsdk:

emcc (Emscripten gcc/clang-like replacement + linker emulating GNU ld) 3.1.18 (d5ca9bba6513763d5bdddbd0efff759332bd85d7)
clang version 16.0.0 (https://github.com/llvm/llvm-project 48129cf0ed5053ed3fdf4f928180635e84892614)
Target: wasm32-unknown-emscripten
Thread model: posix
InstalledDir: /emsdk/upstream/bin

The error I'm getting is this when compiling with -g -sASSERTIONS:

RuntimeError: Aborted(CompileError: WebAssembly.instantiate(): Compiling function #1216:"re2::DFA::RunStateOnByte(re2::DFA::State*, int)" failed: Invalid opcode (enable with --experimental-wasm-threads) @+386207)

The cli calls:

export EMCC_CFLAGS="-O2 -s DEMANGLE_SUPPORT=1 -s ALLOW_MEMORY_GROWTH=1 -s WASM=1"
echo "Compiling wasm wrapper"
emcc -c -std=c++14 -g -sASSERTIONS\
	-I include \
	src/redirects.cc \
	-o build/lib/redirects.bc

echo "Generating Javascript bindings"
emcc --bind -g -sASSERTIONS\
	build/lib/redirects.bc \
	lib/libnetlify-redirects.a \
	lib/libcrypto.a \
	lib/libre2.a \
	lib/libssl.a \
	-o build/lib/redirects.js

This is how we build re2 (which uses atomics):

export EMCC_CFLAGS="-O2"
export CMAKE_CXX_FLAGS="-O2"
export CMAKE_C_FLAGS="-O2"

echo "Configuring RE2"
emcmake cmake -S /src/re2/ -B /tmp/build -DRE2_BUILD_TESTING=OFF

echo "Compiling RE2"
cmake --build /tmp/build -j $(nproc)

Complete output with -v flag:

+ export 'EMCC_CFLAGS=-O2 -s DEMANGLE_SUPPORT=1 -s ALLOW_MEMORY_GROWTH=1 -s WASM=1'
+ EMCC_CFLAGS='-O2 -s DEMANGLE_SUPPORT=1 -s ALLOW_MEMORY_GROWTH=1 -s WASM=1'
+ echo 'Compiling wasm wrapper'
Compiling wasm wrapper
+ emcc -c -std=c++14 -g -v -sASSERTIONS -I include src/redirects.cc -o build/lib/redirects.bc
emcc: warning: linker setting ignored during compilation: 'ASSERTIONS' [-Wunused-command-line-argument]
emcc: warning: linker setting ignored during compilation: 'DEMANGLE_SUPPORT' [-Wunused-command-line-argument]
emcc: warning: linker setting ignored during compilation: 'ALLOW_MEMORY_GROWTH' [-Wunused-command-line-argument]
emcc: warning: linker setting ignored during compilation: 'WASM' [-Wunused-command-line-argument]
 "/emsdk/upstream/bin/clang++" -target wasm32-unknown-emscripten -fignore-exceptions -fvisibility=default -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -DEMSCRIPTEN -D__EMSCRIPTEN_major__=3 -D__EMSCRIPTEN_minor__=1 -D__EMSCRIPTEN_tiny__=18 -Werror=implicit-function-declaration -I/emsdk/upstream/emscripten/cache/sysroot/include/SDL --sysroot=/emsdk/upstream/emscripten/cache/sysroot -Xclang -iwithsysroot/include/compat -c -std=c++14 -g3 -v -Iinclude -O2 src/redirects.cc -o build/lib/redirects.bc
clang version 16.0.0 (https://github.com/llvm/llvm-project 48129cf0ed5053ed3fdf4f928180635e84892614)
Target: wasm32-unknown-emscripten
Thread model: posix
InstalledDir: /emsdk/upstream/bin
 (in-process)
 "/emsdk/upstream/bin/clang-16" -cc1 -triple wasm32-unknown-emscripten -emit-obj --mrelax-relocations -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name redirects.cc -mrelocation-model static -mframe-pointer=none -ffp-contract=on -fno-rounding-math -mconstructor-aliases -target-cpu generic -mllvm -treat-scalable-fixed-error-as-warning -debug-info-kind=constructor -dwarf-version=4 -debugger-tuning=gdb -v -fcoverage-compilation-dir=/src/node -resource-dir /emsdk/upstream/lib/clang/16.0.0 -D EMSCRIPTEN -D __EMSCRIPTEN_major__=3 -D __EMSCRIPTEN_minor__=1 -D __EMSCRIPTEN_tiny__=18 -I /emsdk/upstream/emscripten/cache/sysroot/include/SDL -I include -isysroot /emsdk/upstream/emscripten/cache/sysroot -internal-isystem /emsdk/upstream/emscripten/cache/sysroot/include/wasm32-emscripten/c++/v1 -internal-isystem /emsdk/upstream/emscripten/cache/sysroot/include/c++/v1 -internal-isystem /emsdk/upstream/lib/clang/16.0.0/include -internal-isystem /emsdk/upstream/emscripten/cache/sysroot/include/wasm32-emscripten -internal-isystem /emsdk/upstream/emscripten/cache/sysroot/include -O2 -Werror=implicit-function-declaration -std=c++14 -fdeprecated-macro -fdebug-compilation-dir=/src/node -ferror-limit 19 -fvisibility default -fgnuc-version=4.2.1 -fcxx-exceptions -fignore-exceptions -fexceptions -fcolor-diagnostics -vectorize-loops -vectorize-slp -iwithsysroot/include/compat -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -o build/lib/redirects.bc -x c++ src/redirects.cc
clang -cc1 version 16.0.0 based upon LLVM 16.0.0git default target x86_64-unknown-linux-gnu
ignoring nonexistent directory "/emsdk/upstream/emscripten/cache/sysroot/include/wasm32-emscripten/c++/v1"
ignoring nonexistent directory "/emsdk/upstream/emscripten/cache/sysroot/include/wasm32-emscripten"
#include "..." search starts here:
#include <...> search starts here:
 /emsdk/upstream/emscripten/cache/sysroot/include/SDL
 include
 /emsdk/upstream/emscripten/cache/sysroot/include/compat
 /emsdk/upstream/emscripten/cache/sysroot/include/c++/v1
 /emsdk/upstream/lib/clang/16.0.0/include
 /emsdk/upstream/emscripten/cache/sysroot/include
End of search list.
+ echo 'Generating Javascript bindings'
Generating Javascript bindings
+ emcc --bind -g -v -sASSERTIONS build/lib/redirects.bc lib/libnetlify-redirects.a lib/libcrypto.a lib/libre2.a lib/libssl.a -o build/lib/redirects.js
 "/emsdk/upstream/bin/wasm-ld" -o build/lib/redirects.wasm build/lib/redirects.bc lib/libnetlify-redirects.a lib/libcrypto.a lib/libre2.a lib/libssl.a -L/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten --whole-archive -lembind-rtti --no-whole-archive -lGL -lal -lhtml5 -lstubs-debug -lnoexit -lc-debug -ldlmalloc -lcompiler_rt -lc++-noexcept -lc++abi-noexcept -lsockets -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr --import-undefined --export-if-defined=main --export-if-defined=__start_em_asm --export-if-defined=__stop_em_asm --export-if-defined=__main_argc_argv --export-if-defined=fflush --export=emscripten_stack_get_end --export=emscripten_stack_get_free --export=emscripten_stack_get_base --export=emscripten_stack_init --export=__cxa_demangle --export=stackSave --export=stackRestore --export=stackAlloc --export=__wasm_call_ctors --export=__errno_location --export=malloc --export=free --export=__cxa_is_pointer_type --export=ntohs --export=htons --export=emscripten_builtin_memalign --export-table -z stack-size=5242880 --initial-memory=16777216 --no-entry --max-memory=2147483648 --global-base=1024
 "/emsdk/upstream/bin/wasm-emscripten-finalize" -g --dyncalls-i64 --dwarf build/lib/redirects.wasm -o build/lib/redirects.wasm --detect-features
 "/emsdk/node/14.18.2_64bit/bin/node" /emsdk/upstream/emscripten/src/compiler.js /tmp/tmp2igkkjf6.json
emcc: warning: running limited binaryen optimizations because DWARF info requested (or indirectly required) [-Wlimited-postlink-optimizations]
 "/emsdk/upstream/bin/wasm-opt" --post-emscripten -O2 --low-memory-unused --zero-filled-memory --strip-producers build/lib/redirects.wasm -o build/lib/redirects.wasm -g --mvp-features --enable-threads --enable-bulk-memory --enable-mutable-globals --enable-sign-ext

[kripken]

Hmm, that is very strange... "invalid opcode" suggests it's emitting atomics. But you linked without atomics support, so it shouldn't.

Can you attach that failing wasm file?

Another option, since this is a regression, is to bisect:

https://emscripten.org/docs/contributing/developers_guide.html#bisecting

Even though the range here is quite large, bisection takes a logarithmic number of steps so it should be pretty fast for you.

[danez]

Here is the wasm file: https://unpkg.com/browse/netlify-redirector@0.3.0/lib/

So I tested with version 3.0.0 and in this version emscripten (correctly) errored with something like re2 being compiled with atomics but shared-memory not active.

But in the latest version there is no such error. Trying to use the compiled re2 (libre2.a) from 3.1.18, but then doing the last linking/binding with 3.0.0 also does not print the error. So to me it seems that the compilation of re2 yields a different output with different emscripten versions? But yeah I have no idea.

I now fixed the problem by patching RE2 to disable pthread support, and now it seems to work correctly. The version of RE2 we use is old though.

50,51c50,51
<   add_compile_options(-pthread)
<   list(APPEND EXTRA_TARGET_LINK_LIBRARIES -pthread)
---
>   # add_compile_options(-pthread)
>   # list(APPEND EXTRA_TARGET_LINK_LIBRARIES -pthread)

I attached the compiled output from RE2 (with emscripten 3.1.18). Maybe that helps.
re2_compiled_with_pthread.zip

Anyway, thanks for the great tool! Besides this, it just worked ❤️

[kripken]

Glad it's working now @danez !

This is still quite odd though. The final wasm file in your first link in the last comment does have atomic-using operations. But the link command from the top comment does not enable pthreads. In that case wasm-ld should error on a feature mismatch (when it tries to link threads-using code with non-threaded standard libraries). Or am I forgetting something there? cc @tlively

[tlively]

My guess is that compiling to .bc files instead of .o files is causing the problem. When linking .bc files, any of them having pthreads enabled will cause all of them to have pthreads enabled rather than causing the expected error. I'm not sure why that would have changed between Emscripten versions, though. Maybe older Emscriptens didn't actually compile to bitcode when the output ended in .bc?

[sbc100]

Yes, emscripten used to be much more magical in terms of how it handled the output filename. These days it wont build with bitcode until you build with -c and -flto (or -emit-llvm)

[kripken]

It would be great to add that to the docs, if it isn't there already? Perhaps in https://emscripten.org/docs/compiling/Building-Projects.html

[sbc100]

There is already a note specifically about the behaviour change (that happened a couple of years ago now): https://emscripten.org/docs/compiling/Building-Projects.html#emscripten-linker-output-files

Regardless of the name of the output file emcc will always perform linking and produce a final exectuable, unless a specific flags (e.g. -c) direct it do something else. This differs to previous behaviour where emcc would default to combining object files (essentially assuming -r) unless given a specific executable extension (e.g. .js or .html).

[sbc100]

I've added a new warning in #17678 so folks using -c -o out.bc without -flto will now get a warning.