Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Assertion `gotIndex == INVALID_INDEX' failed #9317

Open
Beuc opened this issue Aug 25, 2019 · 24 comments

Comments

@Beuc
Copy link
Contributor

commented Aug 25, 2019

Hi,

I ran into an assertion error when linking zlib.

Steps to reproduce:

wget http://prdownloads.sourceforge.net/libpng/zlib-1.2.11.tar.gz
tar xf zlib-1.2.11.tar.gz 
cd zlib-1.2.11/
emconfigure env CFLAGS="-O3" ./configure --prefix $(pwd)/destdir
make

Output:

...
emcc -O3 -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -o libz.so.1.2.11 adler32.lo crc32.lo deflate.lo infback.lo inffast.lo inflate.lo inftrees.lo trees.lo zutil.lo compress.lo uncompr.lo gzclose.lo gzlib.lo gzread.lo gzwrite.lo  -lc 
wasm-ld: /b/s/w/ir/cache/builder/emscripten-releases/llvm-project/lld/wasm/Symbols.cpp:115: void lld::wasm::Symbol::setGOTIndex(uint32_t): Assertion `gotIndex == INVALID_INDEX' failed.
Stack dump:
0.	Program arguments: /home/me/workdir/emtests/renpyweb/toolchain/emsdk/upstream/bin/wasm-ld -o libz.so.1.2.11 --allow-undefined --import-memory --import-table --lto-O0 adler32.lo crc32.lo deflate.lo infback.lo inffast.lo inflate.lo inftrees.lo trees.lo zutil.lo compress.lo uncompr.lo gzclose.lo gzlib.lo gzread.lo gzwrite.lo -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr --export __wasm_call_ctors --export __data_end --export main --export malloc --export free --export setThrew --export __errno_location -z stack-size=5242880 --initial-memory=16777216 --no-entry --max-memory=16777216 --global-base=1024 --relocatable 
 #0 0x00007f3d675d69f4 PrintStackTraceSignalHandler(void*) (/home/me/workdir/emtests/renpyweb/toolchain/emsdk/upstream/bin/../lib/libLLVM-10svn.so+0x6d79f4)
 #1 0x00007f3d675d473e llvm::sys::RunSignalHandlers() (/home/me/workdir/emtests/renpyweb/toolchain/emsdk/upstream/bin/../lib/libLLVM-10svn.so+0x6d573e)
 #2 0x00007f3d675d6ca8 SignalHandler(int) (/home/me/workdir/emtests/renpyweb/toolchain/emsdk/upstream/bin/../lib/libLLVM-10svn.so+0x6d7ca8)
 #3 0x00007f3d6a4a9730 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x12730)
 #4 0x00007f3d66a547bb raise (/lib/x86_64-linux-gnu/libc.so.6+0x377bb)
 #5 0x00007f3d66a3f535 abort (/lib/x86_64-linux-gnu/libc.so.6+0x22535)
 #6 0x00007f3d66a3f40f (/lib/x86_64-linux-gnu/libc.so.6+0x2240f)
 #7 0x00007f3d66a4d102 (/lib/x86_64-linux-gnu/libc.so.6+0x30102)
 #8 0x00000000006ac4fb (/home/me/workdir/emtests/renpyweb/toolchain/emsdk/upstream/bin/wasm-ld+0x6ac4fb)
 #9 0x00000000006c691b lld::wasm::GlobalSection::assignIndexes() (/home/me/workdir/emtests/renpyweb/toolchain/emsdk/upstream/bin/wasm-ld+0x6c691b)
#10 0x00000000006b1466 (anonymous namespace)::Writer::run() (/home/me/workdir/emtests/renpyweb/toolchain/emsdk/upstream/bin/wasm-ld+0x6b1466)
#11 0x00000000006adaaf lld::wasm::writeResult() (/home/me/workdir/emtests/renpyweb/toolchain/emsdk/upstream/bin/wasm-ld+0x6adaaf)
#12 0x0000000000690ae7 (anonymous namespace)::LinkerDriver::link(llvm::ArrayRef<char const*>) (/home/me/workdir/emtests/renpyweb/toolchain/emsdk/upstream/bin/wasm-ld+0x690ae7)
#13 0x000000000068b6d8 lld::wasm::link(llvm::ArrayRef<char const*>, bool, llvm::raw_ostream&) (/home/me/workdir/emtests/renpyweb/toolchain/emsdk/upstream/bin/wasm-ld+0x68b6d8)
#14 0x000000000041eadb main (/home/me/workdir/emtests/renpyweb/toolchain/emsdk/upstream/bin/wasm-ld+0x41eadb)
#15 0x00007f3d66a4109b __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2409b)
#16 0x000000000041e669 _start (/home/me/workdir/emtests/renpyweb/toolchain/emsdk/upstream/bin/wasm-ld+0x41e669)
shared:ERROR: '/home/me/workdir/emtests/renpyweb/toolchain/emsdk/upstream/bin/wasm-ld -o libz.so.1.2.11 --allow-undefined --import-memory --import-table --lto-O0 adler32.lo crc32.lo deflate.lo infback.lo inffast.lo inflate.lo inftrees.lo trees.lo zutil.lo compress.lo uncompr.lo gzclose.lo gzlib.lo gzread.lo gzwrite.lo -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr --export __wasm_call_ctors --export __data_end --export main --export malloc --export free --export setThrew --export __errno_location -z stack-size=5242880 --initial-memory=16777216 --no-entry --max-memory=16777216 --global-base=1024 --relocatable' failed (-6)
make: *** [Makefile:282: libz.so.1.2.11] Error 1

$ emcc --version
emcc (Emscripten gcc/clang-like replacement) 1.38.42 (commit da4900b83f6bce7511189b9307569d6c9c344e36)
@supernow

This comment has been minimized.

Copy link

commented Aug 26, 2019

I got this error ,too

@sbc100 sbc100 self-assigned this Aug 26, 2019

@sbc100

This comment has been minimized.

Copy link
Collaborator

commented Aug 26, 2019

Certainly looks like an lld bug. Will try to reproduce locally.

@Beuc

This comment has been minimized.

Copy link
Contributor Author

commented Aug 27, 2019

I got it again when linking SDL2_mixer 2.0.1 and freebidi 0.19.2.
Let me know if you're interested in more traces.

@sbc100

This comment has been minimized.

Copy link
Collaborator

commented Aug 27, 2019

Fix is in progress in https://reviews.llvm.org/D66784. As a workaround (assuming are you are not actually trying to use MAIN_MODULE/SIDE_MODULE) you can make sure you are building everything without -fPIC.

@Beuc

This comment has been minimized.

Copy link
Contributor Author

commented Aug 31, 2019

Are there instructions on how to manually test this patch? (it is still waiting for review at LLVM)
(I tried getting rid of all -fPIC but apparently I missed one and the error doesn't tell me where)
Building Emscripten from Source is quite vague for "upstream", and I don't know how the official binaries are built.

@pmp-p

This comment has been minimized.

Copy link

commented Aug 31, 2019

same question there :)

@kripken

This comment has been minimized.

Copy link
Member

commented Aug 31, 2019

Oh, what's missing from the docs in that link? It links to the upstream docs and it mentions the extra flag -DLLVM_ENABLE_PROJECTS=lld;clang.

@Beuc

This comment has been minimized.

Copy link
Contributor Author

commented Aug 31, 2019

The upstream doc has like 50 project-specific options that one may or may not need (the previous fastcomp doc set 5 precise options). Each build requires at least one hour, each mistake requires a complete rebuild. Consequently I don't want to spend the week-end running builds and finding a working configuration by trial-and-error, I'd rather use a tested set of options -- preferably the one you use for official builds.
With the added insurance that errors I'll report will come from the patch and not from my build :)

@kripken

This comment has been minimized.

Copy link
Member

commented Aug 31, 2019

Oh, yeah - LLVM does have a lot of options. But you don't need any, in most cases just a standard cmake build should work, so

mkdir mybuild
cd mybuild
cmake .. [extra flags from emscripten docs]
make -j8

If you do need any special flags to work around a local issue (like if the cmake step fails on something), then the LLVM docs can be very helpful. I don't think there's anything we can add in the emscripten docs, except to add the code snippet I just wrote, would that help?

@Beuc

This comment has been minimized.

Copy link
Contributor Author

commented Aug 31, 2019

Thanks for the info!

This is typically the kind of steps where, if one is already familiar with the target system, things appear natural, but if one is new, one has to assemble information from numerous pages. Also the LLVM documentation is adamant about reading the full documentation before trying anything, which is daunting. So providing explicit instructions, or pointing to the official build scripts (which I didn't find in e.g. .circleci) would help.

In either case, IMHO, the needed information include:

  • where to checkout llvm from (repository "llvm-project" - https://github.com/llvm/llvm-project.git)
  • how to identify to version to checkout (no tags for v10.x - always build "master"?)
  • clarify whether clang needs to be checked-out separately or not (since we had to that with fastcomp, and that's optional now AFAIU; there's a warning about that in the LLVM doc)
  • clarify whether "make install" is required or if the build is intended to be used in-place (installing would require carefully specifying a working install prefix, most probably non-default)

The cmake instructions seems to target ../llvm, rather than ...
There's a missing starting quote in the current documentation (-DLLVM_ENABLE_PROJECTS=lld;clang').
The phrase "using something like -DLLVM_..." was implying there was more configuration work/fiddling to perform, but since this fortunately doesn't seem to be the case, let's leave it out :)

I'm compiling as we speak, I'll add more remarks if I find some.

@Beuc

This comment has been minimized.

Copy link
Contributor Author

commented Aug 31, 2019

Possibly, mention that this requires at least 70GB build space and 2h compilation. If there's a simple way to lower these requirements, it would be good to note them.

It seems my LLVM is 10x slower than the Emscriten build when compiling SDL2, I guess I missed an option, possibly CMAKE_BUILD_TYPE=Release. I'd have to spend 2 more hours to test this hypothesis.

-j8 will require a huge amount of memory at the linking phase and the build will probably fail (even with -j1 linking clang filled my 16GB RAM), so I would recommend a more conservative value.

--

binaryen/wasm-opt gives me Unknown option '--asyncify' - the binaryen port doesn't fetch the right version (86 vs. 89).
This may be a bug in the current port file.
If not, we also need:

  • how to identify the required binaryen version
  • the need to specify the location of binaryen in emconfig with BINARYEN_ROOT

--

At last I can build my project for the first time, but the browser greets me with:
uncaught exception: could not load memory initializer http://localhost:8000/libz.raw.js.mem
This file is nowhere to be found. Though given all the above, I don't have a high confidence in my build.
I hope this emphasizes that testing a simple patch could be made smoother! ;)

@Beuc

This comment has been minimized.

Copy link
Contributor Author

commented Aug 31, 2019

2 long builds and testings later I eventually refined to what I wanted to read from emscripten.org:

cmake ../llvm -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_PROJECTS='lld;clang' -DLLVM_TARGETS_TO_BUILD="host;WebAssembly" -DLLVM_INCLUDE_EXAMPLES=OFF -DLLVM_INCLUDE_TESTS=OFF

--

This particular issue sounds fixed, but I get several other errors in my project.
No more time to investigate this week :/

@kripken

This comment has been minimized.

Copy link
Member

commented Sep 3, 2019

Thanks @Beuc!

I think you're right, we should have an example of reasonable build flags for LLVM - I didn't realize the defaults are to debug, and that debug is so painful :( Thanks for the findings, I opened #9371 with them.

@sbc100

This comment has been minimized.

Copy link
Collaborator

commented Sep 3, 2019

Sorry, the upstream change is not finished so if you are trying to build llvm so that you can integrate https://reviews.llvm.org/D66784 then I would not advise that for now.

A much quicker solution would be to filter out -fPIC in the emcc.py driver. You can do this in parse_args.. you can either assert that -fPIC is not in the newargs in order to find out which of your libraries is being compiled wrong.. or you can just filter it out to have it ignored by the underlying clang.

kripken added a commit that referenced this issue Sep 3, 2019
@Beuc

This comment has been minimized.

Copy link
Contributor Author

commented Sep 4, 2019

I tried preventing -fPIC in emcc.py, good idea. The project now compiles (with tot-upstream), and I didn't get the weird cython error that I got when trying out D66784.

Thanks for adding the LLVM compilation line. There's a mistake, and this doesn't treat the other doc points I raised, but I'll open a separate bug/pr.

@pmp-p

This comment has been minimized.

Copy link

commented Sep 4, 2019

A much quicker solution would be to filter out -fPIC in the emcc.py driver. You can do this in parse_args.. you can either assert that -fPIC is not in the newargs in order to find out which of your libraries is being compiled wrong.. or you can just filter it out to have it ignored by the underlying clang.

out of topic but can that method be used to easy fix removing "-s USE_*" and other things when emcc is required to act as a preprocessor ( clang -E ) ?

@sbc100

This comment has been minimized.

Copy link
Collaborator

commented Sep 4, 2019

A much quicker solution would be to filter out -fPIC in the emcc.py driver. You can do this in parse_args.. you can either assert that -fPIC is not in the newargs in order to find out which of your libraries is being compiled wrong.. or you can just filter it out to have it ignored by the underlying clang.

out of topic but can that method be used to easy fix removing "-s USE_*" and other things when emcc is required to act as a preprocessor ( clang -E ) ?

I'm not quite sure what you are asking. Are you saying that the '-s USE_*' flags are being passed down the the underlying clang when -E is used? If this is the case it is certainly a bug, please open a seperate issue. If not, perhaps I misunderstand you.

@pmp-p

This comment has been minimized.

Copy link

commented Sep 4, 2019

it is certainly a bug, please open a seperate issue

i wasn't sure emcc is meant to be used also as preprocessor with multiple files input ( but we need to in micropython javascript port) so i will open an issue thanks for clarifying.

edit/ my bad issue was not with input args but with number of files input : "I tried defining CPP = emcc -E but this doesn't like multiple input files"
sorry for the noise

@Beuc

This comment has been minimized.

Copy link
Contributor Author

commented Sep 18, 2019

Hi!
Are there news on static linking with -fPIC? :)

@sbc100

This comment has been minimized.

Copy link
Collaborator

commented Sep 18, 2019

The upstream change is still not ready: https://reviews.llvm.org/D66784. Once it lands this issue can closed. I should be able to take another look before the end of the week.

@Beuc

This comment has been minimized.

Copy link
Contributor Author

commented Sep 18, 2019

I did check the reviews.llvm.org link but the information there is apparently stalled since Aug 26th.
Hence why I asked.
Let me know if there is was a better way :)

@sbc100

This comment has been minimized.

Copy link
Collaborator

commented Sep 18, 2019

The code is simply not ready to review yet and no work has been done on it since that date. There is no better way. Asking about status fine.

@Shachlan

This comment has been minimized.

Copy link

commented Sep 19, 2019

@sbc100
As a novice to the complexities of the c++ build system, I'm not sure - how can a library be built without fPIC? Wouldn't that make it unusable in other code?

@sbc100

This comment has been minimized.

Copy link
Collaborator

commented Sep 19, 2019

No, compiling code without fPIC simply means that the resulting objects files cannot be linked into a shared library. Its perfectly possible for the shared library and that static library to contains different object files, one set built with fPIC and the other set built without fPIC. In fact this is technically correct and depending the platform will result in a faster static build since fPIC can have performance costs.

Wohlstand added a commit to WohlSoft/PGE-Project that referenced this issue Sep 22, 2019
Editor, Engine and Build: Minor fixes
- Engine: unsuccessfull attempt to build with `ASYNCIFY` on Emscripten
(this issue emscripten-core/emscripten#9317 prevents a build of project)
- Editor: Disable code of styling because of crashes on Linux in some cases
- Build: Fixed CLang warnings because of incompatible --gc-section arguments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
6 participants
You can’t perform that action at this time.