em++: error: undefined exported symbol: "_LLVMAddSymbol" [-Wundefined] [-Werror]

[makslevental]

Bug

I'm trying to build LLVM (plus some subprojects) and I'm getting this error no matter what I do:

em++: error: undefined exported symbol: "_LLVMAddSymbol" [-Wundefined] [-Werror]

I don't think it's super important what this function is/does but it is in fact unused in the LLVM codebase itself and it is part of LLVM's C API. So it's in a header somewhere and wrapped in extern "C" and etc.

The full link line is

/eudsl/scripts/llvm_wasm/.pyodide_build/pywasmcross_symlinks/c++ -fPIC -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wno-pass-failed -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -Wundef -Werror=mismatched-tags -Werror=global-constructors -O3 -DNDEBUG  -sALLOW_TABLE_GROWTH -sASSERTIONS -sWASM_BIGINT -Wl,--color-diagnostics   -Wl,--gc-sections -shared -Wl,-soname,libmlir_c_runner_utils.so -o lib/libmlir_c_runner_utils.so tools/mlir/lib/ExecutionEngine/CMakeFiles/mlir_c_runner_utils.dir/CRunnerUtils.cpp.o tools/mlir/lib/ExecutionEngine/CMakeFiles/mlir_c_runner_utils.dir/SparseTensorRuntime.cpp.o  lib/libMLIRSparseTensorRuntime.a  lib/libLLVMSupport.a  lib/libmlir_float16_utils.so  -lrt  -lm  lib/libLLVMDemangle.a

Here's the utterly baffling part:

(venv) root@b0a8b42982ce:/eudsl# /emsdk/upstream/bin/llvm-nm scripts/llvm_wasm/build/lib/libLLVMSupport.a | grep -C3 LLVMAddSymbol
scripts/llvm_wasm/build/lib/libLLVMSupport.a:blake3_neon.c.o: no symbols
scripts/llvm_wasm/build/lib/libLLVMSupport.a:AutoConvert.cpp.o: no symbols

DynamicLibrary.cpp.o:
0000000c d .L.str
00000e68 T LLVMAddSymbol
00000da2 T LLVMLoadLibraryPermanently
00000e5d T LLVMSearchForAddressOfSymbol

i.e., the symbol is in lib/libLLVMSupport.a I believe (the leading _ prefix notwithstanding).

Things I've tried

Mostly toggling LLVM_ENABLE_PIC (i.e., -fPIC) and LLVM_BUILD_STATIC. I also tried various permutations of -sERROR_ON_UNDEFINED_SYMOLS=0 and finally landed on -Wno-undefined which does silence the error but just kicks the can down the road (the shlib fails to load).

I also tried commenting out the decl of that function in its header and that just uncovers more such unused decls in headers - naturally it's not practical to patch all of these out...

Version info

This is happening on 4.0.9:

(venv) root@b0a8b42982ce:/eudsl# emcc -v
emcc (Emscripten gcc/clang-like replacement + linker emulating GNU ld) 4.0.9 (bbf1caa6e24f64fca9eb6a13a9e02d3f42123e77)
clang version 21.0.0git (https:/github.com/llvm/llvm-project 2f05451198e2f222ec66cec4892ada0509519290)
Target: wasm32-unknown-emscripten
Thread model: posix
InstalledDir: /emsdk/upstream/bin

The other baffling part is I can actually successfully do all what I'm trying to do on 3.1.58. I haven't tried bisecting but something broke somewhere along the line 🤷.

Related

Possibly related to #21516 but the issue identifies 3.1.52 as the source while like I said I have a working build on 3.1.58 🤷.

[makslevental]

Possibly LINKABLE or RELOCATABLE or SIDE_MODULE is what I need?

[makslevental]

Eureka!

CMAKE_C_FLAGS = "-sLINKABLE"
CMAKE_CXX_FLAGS = "-sLINKABLE"

worked! But -sSIDE_MODULE=1 did not. So is there any chance y'all will keep -sLINKABLE?

[sbc100]

I wonder if the compiler wrapper (/eudsl/scripts/llvm_wasm/.pyodide_build/pywasmcross_symlinks/c++) is doing anything clever?

The em++: error: undefined exported symbol message should only happen for symbols that are listed in -sEXPORTED_FUNCTIONS but I don't see that flag in you command line flags.

Can see see the full em++ command?

[makslevental]

I wonder if the compiler wrapper (/eudsl/scripts/llvm_wasm/.pyodide_build/pywasmcross_symlinks/c++) is doing anything clever?

The em++: error: undefined exported symbol message should only happen for symbols that are listed in -sEXPORTED_FUNCTIONS but I don't see that flag in you command line flags.

Can see see the full em++ command?

I did dump the entire em++ command and it is indeed filled with an enormous number of --exports but from reading

emscripten/tools/building.py

Line 204 in d1ba27c

c_exports = [e for e in settings.EXPORTED_FUNCTIONS if is_c_symbol(e)]

I believe this is being done by emscripten not pyodide. So it seems to me that what's happening is

1. pyodide passes -sSIDE_MODULE
2. emscripten tries to export all of the extern "C" symbols
3. wasm-ld fails to link/find/extract that symbol

One other thing, I actually am still stuck even with -sLINKABLE (I had successfully built but now I can't recover the permutation of CMake configured vars...) - the missing symbols is now one of these LLVM_ABI fields which is initialized constexpr.

[sbc100]

Can you share the entire em++ command line?

The --export flags are part of the wasm-ld command line BTW, not the em++ command line.

Do you know why -sSIDE_MODULE is being added, are you trying to build llvm itself as a side module, or shared library? maybe you can explain too why you are using or why you are using pyodide` and not just using emcc directly?

[makslevental]

Can you share the entire em++ command line?

new_args ['em++', '-fPIC', '-fPIC', '-fvisibility-inlines-hidden', '-Werror=date-time', '-Werror=unguarded-availability-new', '-Wall', '-Wextra', '-Wno-unused-parameter', '-Wwrite-strings', '-Wcast-qual', '-Wmissing-field-initializers', '-pedantic', '-Wno-long-long', '-Wc++98-compat-extra-semi', '-Wimplicit-fallthrough', '-Wcovered-switch-default', '-Wno-noexcept-type', '-Wnon-virtual-dtor', '-Wdelete-non-virtual-dtor', '-Wsuggest-override', '-Wstring-conversion', '-Wno-pass-failed', '-Wmisleading-indentation', '-Wctad-maybe-unsupported', '-fdiagnostics-color', '-ffunction-sections', '-fdata-sections', '-Wundef', '-Werror=mismatched-tags', '-Werror=global-constructors', '-O3', '-DNDEBUG', '-sALLOW_TABLE_GROWTH', '-sASSERTIONS', '-sWASM_BIGINT', '-sLINKABLE', '-sRELOCATABLE', '-sERROR_ON_UNDEFINED_SYMBOLS=0', '-Wl,--color-diagnostics', '-Wl,--gc-sections', '-shared', '-Wl,-soname,libmlir_runner_utils.so', '-o', 'lib/libmlir_runner_utils.so', 'tools/mlir/lib/ExecutionEngine/CMakeFiles/mlir_runner_utils.dir/RunnerUtils.cpp.o', 'lib/libmlir_float16_utils.so', 'lib/libLLVMSupport.a', '-lrt', '-lm', 'lib/libLLVMDemangle.a', '-Werror=implicit-function-declaration', '-Werror=mismatched-parameter-types', '-Werror=return-type', '-Wl,--fatal-warnings', '-O2', '-g0', '-L/root/.cache/.pyodide-xbuildenv-0.30.9/0.29.0/xbuildenv/pyodide-root/cpython/installs/python-3.13.2/lib/', '-s', 'WASM_BIGINT', '-fwasm-exceptions', '-sSUPPORT_LONGJMP', '-s', 'SIDE_MODULE=1', '-sSIDE_MODULE=2', '-sEXPORTED_FUNCTIONS=@/tmp/tmpte30l76k']

Do you know why -sSIDE_MODULE is being added,

I believe pyodide does this for me.

are you trying to build llvm itself as a side module, or shared library? maybe you can explain too why you are using or why you are using pyodide` and not just using emcc directly?

I'm trying (and was previously succeeding) to build a Python package that runs on pyodide which contains yes various LLVM components as Python C extensions libraries - you can see it live here (and the impl here).

[sbc100]

I guess the problem is with -sEXPORTED_FUNCTIONS=@/tmp/tmpte30l76k .. i'm not sure where that comes from

[makslevental]

Brilliant (I could kiss you!) - it's being inserted here and the remedy is export PYODIDE_BUILD_EXPORTS=whole_archive (which passes -Wl,--whole-archive to wasm-ld). With this I believe I can actually build successfully (will report back...)

[makslevental]

I'm happy to say I made it all the way through (building LLVM as a distro and then using that distro to build the Python package) but ultimately it failed to work:

I get lots of "Failed to load lib" errors where the symbols are from 2 categories:

Failed to load lib  /Users/mlevental/dev_projects/eudsl/pyodide-venv/lib/python3.13/site-packages/mlir/_mlir_libs/_mlirAsyncPasses.cpython-313-wasm32-emscripten.so
Error: Failed to load dynamic library /Users/mlevental/dev_projects/eudsl/pyodide-venv/lib/python3.13/site-packages/mlir/_mlir_libs/_mlirAsyncPasses.cpython-313-wasm32-emscripten.so: 'Could not load dynamic lib: /Users/mlevental/dev_projects/eudsl/pyodide-venv/lib/python3.13/site-packages/mlir/_mlir_libs/_mlirAsyncPasses.cpython-313-wasm32-emscripten.so
Error: bad export type for '_ZN4llvm11APFloatBase18semPPCDoubleDoubleE': undefined

which is the same LLVM_ABI field which is initialized constexpr.

and

Failed to load lib  /Users/mlevental/dev_projects/eudsl/pyodide-venv/lib/python3.13/site-packages/mlir/_mlir_libs/lib32b_mlir_runner_utils.so
Error: Failed to load dynamic library /Users/mlevental/dev_projects/eudsl/pyodide-venv/lib/python3.13/site-packages/mlir/_mlir_libs/lib32b_mlir_runner_utils.so: 'Could not load dynamic lib: /Users/mlevental/dev_projects/eudsl/pyodide-venv/lib/python3.13/site-packages/mlir/_mlir_libs/lib32b_mlir_runner_utils.so
Error: bad export type for 'mlirTypeIsAGPUAsyncTokenType': undefined

which is also a C API (in my chosen LLVM component).

Both of these are of course actually present (somewhere):

# llvm-nm libMLIRPythonWasmCAPI.so | grep mlirTypeIsAGPUAsyncTokenType
004b5a74 T mlirTypeIsAGPUAsyncTokenType

and

# llvm-nm libMLIRPythonWasmCAPI.so | grep mlirTypeIsAGPUAsyncTokenType
00213e48 D _ZN4llvm11APFloatBase18semPPCDoubleDoubleE

I say somewhere because they're not in the lib they're being searched for (lib32b_mlir_runner_utils.so -> libMLIRPythonWasmCAPI.so) so maybe the issue is that the runtime loader is failing to load libMLIRPythonWasmCAPI.so first. I can't figure out a way to pull up the equivalent of ldd for one of these shlibs - do you know something that would work?

I also get

Failed to load lib  /Users/mlevental/dev_projects/eudsl/pyodide-venv/lib/python3.13/site-packages/mlir/_mlir_libs/libMLIRPythonWasmCAPI.so
Error: Failed to load dynamic library /Users/mlevental/dev_projects/eudsl/pyodide-venv/lib/python3.13/site-packages/mlir/_mlir_libs/libMLIRPythonWasmCAPI.so: 'Could not load dynamic lib: /Users/mlevental/dev_projects/eudsl/pyodide-venv/lib/python3.13/site-packages/mlir/_mlir_libs/libMLIRPythonWasmCAPI.so
RuntimeError: null function or function signature mismatch

which I can't make heads or tails of 🤷.

[sbc100]

There is no ldd that understands wasm dynamic libraries. You can using wasm-objdump -x (from the wabt project) to see the dylink.0 section which contains the dynamic linking metadata.

If possible I would recommend trying to reduce the size of the dylib graph. Can you build all of the parts of llvm that you need into once single loadable module rather than shipping a whole graph of interdependent shared libraries?

[makslevental]

Thanks! But as I expected the runtime linkage is present:

_mlirAsyncPasses.cpython-313-wasm32-emscripten.so:      file format wasm 0x1

Section Details:

Custom:
 - name: "dylink.0"
 - mem_size     : 394620
 - mem_p2align  : 4
 - table_size   : 1204
 - table_p2align: 0
 - needed_dynlibs[1]:
  - libMLIRPythonWasmCAPI.so

If possible I would recommend trying to reduce the size of the dylib graph. Can you build all of the parts of llvm that you need into once single loadable module rather than shipping a whole graph of interdependent shared libraries?

There is and indeed that was/is going to be my next step but it is highly non-standard (effectively bundles multiple Python C extension in a single shlib) so I was hoping to avoid it.

Looking more closely at the errors I guess what's happening is pyodide is trying to load libMLIRPythonWasmCAPI.so and failing and therefore all the other libraries are failing to load:

# FIRST ERROR
Failed to load lib  /Users/mlevental/dev_projects/eudsl/pyodide-venv/lib/python3.13/site-packages/mlir/_mlir_libs/_mlir.cpython-313-wasm32-emscripten.so
Error: Failed to load dynamic library /Users/mlevental/dev_projects/eudsl/pyodide-venv/lib/python3.13/site-packages/mlir/_mlir_libs/_mlir.cpython-313-wasm32-emscripten.so: 'Could not load dynamic lib: /Users/mlevental/dev_projects/eudsl/pyodide-venv/lib/python3.13/site-packages/mlir/_mlir_libs/_mlir.cpython-313-wasm32-emscripten.so
Error: Failed to find dynamic library "libMLIRPythonWasmCAPI.so"
    at X.loadDynlib (/Users/mlevental/Library/Caches/.pyodide-xbuildenv-0.30.9/0.29.0/xbuildenv/pyodide-root/dist/pyodide.asm.js:8:659788)
    at async main (file:///Users/mlevental/Library/Caches/.pyodide-xbuildenv-0.30.9/0.29.0/xbuildenv/pyodide-root/dist/python_cli_entry.mjs:91:7)
Failed to load lib  /Users/mlevental/dev_projects/eudsl/pyodide-venv/lib/python3.13/site-packages/mlir/_mlir_libs/_mlirAsyncPasses.cpython-313-wasm32-emscripten.so
...
Failed to load lib  /Users/mlevental/dev_projects/eudsl/pyodide-venv/lib/python3.13/site-packages/mlir/_mlir_libs/libMLIRPythonWasmCAPI.so
Error: Failed to load dynamic library /Users/mlevental/dev_projects/eudsl/pyodide-venv/lib/python3.13/site-packages/mlir/_mlir_libs/libMLIRPythonWasmCAPI.so: 'Could not load dynamic lib: /Users/mlevental/dev_projects/eudsl/pyodide-venv/lib/python3.13/site-packages/mlir/_mlir_libs/libMLIRPythonWasmCAPI.so
RuntimeError: null function or function signature mismatch
    at X.loadDynlib (/Users/mlevental/Library/Caches/.pyodide-xbuildenv-0.30.9/0.29.0/xbuildenv/pyodide-root/dist/pyodide.asm.js:8:659788)
    at async main (file:///Users/mlevental/Library/Caches/.pyodide-xbuildenv-0.30.9/0.29.0/xbuildenv/pyodide-root/dist/python_cli_entry.mjs:91:7)

Any idea about how I can debug that given the last message RuntimeError: null function or function signature mismatch?

But actually this makes me wonder if my problem is

CMAKE_C_FLAGS = "-Wl,--allow-multiple-definition"
CMAKE_CXX_FLAGS = "-Wl,--allow-multiple-definition"

which I had to flip on because -Wl,--whole-archive of course causes multiple duplicate symbols to try to be linked 🤔.

[sbc100]

You should be able to get more information about the error from a debug build with assertions enabled.

See RuntimeError: null function or function signature mismatch at X.loadDynlib (/Users/mlevental/Library/Caches/.pyodide-xbuildenv-0.30.9/0.29.0/xbuildenv/pyodide-root/dist/pyodide.asm.js:8:659788)

Notice pyodide.asm.js:8:659788 this means char 659788 of line 8 which I imagine means you have a minified and less useful build of pyodide.asm.js.

Are you actually using asm.js or is the file just called that? Build with assertions and/or debug info should give you a useful backtrace.

[makslevental]

You should be able to get more information about the error from a debug build with assertions enabled.

I have assertions enabled but yea I should build in debug.

Are you actually using asm.js or is the file just called that? Build with assertions and/or debug info should give you a useful backtrace.

Ah but this is pyodide itself (which I could of course also build in debug mode...). Either way what I'm currently trying is not doing --allow-multiple-definition and instead letting CMP0156 solve my duplication problems. Maybe the loading issues will also be resolved! But with how this has been going, probably not lol.