LLVM LexicalScopes build exception when building with --release flag

[fridgerator]

I was able to build fine in OSX, however Ubuntu 16.04 is failing with this error:

remote: crystal: /var/cache/omnibus/src/llvm/llvm-3.8.1.src/lib/CodeGen/LexicalScopes.cpp:160: llvm::LexicalScope* llvm::LexicalScopes::getOrCreateRegularScope(const llvm::DILocalScope*): Assertion cast(Scope)->describes(MF->getFunction())' failed.`

Originally reported by @crisward here

Replicated by @hak8or and @matiasgarciaisaia. @matiasgarciaisaia mentioned that building with --no-debug works as a temporary workaround.

[crisward]

Has anyone tried build with --release on llvm 3.8.1 on mac? Mac ships with llvm 4.0, so this could be an llvm version issue.

[straight-shoota]

quote from @matiasgarciaisaia

It's really weird to see that /var/cache/omnibus/... there, given that the Crystal package doesn't install that.

It's really weird. That /var/cache/omnibus directory doesn't even exist on my machine and should never be created.

[konovod]

I have similar (but maybe not identical) problem:

crystal: /var/cache/omnibus/src/llvm/llvm-3.8.1.src/include/llvm/IR/DataLayout.h:513: uint64_t llvm::DataLayout::getTypeSizeInBits(llvm::Type*) const: Assertion `Ty->isSized() && "Cannot getTypeInfo() on a type that is unsized!"' failed.
/usr/bin/crystal: line 102:  6027 Aborted                 (core dumped) "$INSTALL_DIR/embedded/bin/crystal" "$@"

https://travis-ci.org/konovod/linalg/builds/253769479
Maybe offtopic (and i still haven't found minimal example for reporting), but it also has /var/cache/omnibus and also appeared after 0.23.1 so i think worth mentioning.

[crisward]

I also don't have /var/cache/omnibus/ and get that error. Also this is very easy to reproduce.

# on Ubuntu 14.04
sudo apt-get install crystal
echo 'p "hello world"' > test.cr
crystal build test.cr --release

crystal: /var/cache/omnibus/src/llvm/llvm-3.8.1.src/lib/CodeGen/LexicalScopes.cpp:160: llvm::LexicalScope* llvm::LexicalScopes::getOrCreateRegularScope(const llvm::DILocalScope*): Assertion `cast<DISubprogram>(Scope)->describes(MF->getFunction())' failed.
/usr/bin/crystal: line 102: 16249 Aborted                 "$INSTALL_DIR/embedded/bin/crystal" "$@"

[RX14]

I don't find it confusing at all. It's telling you the location in the source file where the runtime assertion failed, doesn't mean the source file has to be there on the PC where it runs, because it's a compiled library. It's exactly the same as __FILE__ in crystal.

[straight-shoota]

@konovod This runs without --release, so it seems to be a different thing, still might as well be related. Perhaps it's an issue with liblapacke?
Does this error also appear on local builds? And is it present in 0.23.0 or the inofficial release of 0.23.1 (#4647)?

[straight-shoota]

@RX14 Ok, makes sense. I didn't expect LLVM to have such detailed internal debug info in a released binary^^

@crisward Even smaller example would be crystal eval --release 'p "hello world'

[RX14]

It appears to me that LLVM was manually compiled by the omnibus, instead of using a prebuilt version. By the appearance of assertion errors, it's probably not a release build either. It's likely a release build would simply not have these runtime checks.

[konovod]

Does this error also appear on local builds?

Still haven't managed to get it on local build. If assertions in llvm are present, perhaps this problem is unrelated.

[crisward]

Also

• crystal eval --release 'p "hello world"' --no-debug works
• crystal eval --release 'p "hello world"' fails

[crisward]

Not sure if this is relevant but the old llvm setup used to configure disable-assertions

crystal-lang/omnibus-crystal@57e0ebc#diff-004ad5a5c6f368464daff6c7cbde83a5L26

[crisward]

Looks like setting CMAKE_BUILD_TYPE to debug makes it enable assertions otherwise it's in disabled. So probably not that then.

[RX14]

@crisward confusingly, I think that's the wrong branch: https://github.com/crystal-lang/omnibus-crystal/blob/current/config/software/llvm.rb

Still has assertions disabled, even though it's not a release build.

[crisward]

That version doesn't appear to have CMAKE_BUILD_TYPE set at all, though the docs don't indicate the default setting. Perhaps setting it to MinSizeRel would do the trick?

[ysbaddaden]

@RX14 you can compile LLVM in release mode yet keep assertions. It's a little slower, but still a good idea, as it will catch understandable errors, that would otherwise lead to segfaults that are hard to understand.

[ysbaddaden]

Which happened to me before. I couldn't understand an LLVM segfault, which was catched quickly by asterite which had assertions turned on.

[straight-shoota]

So, any idea what the failing assertion is telling us? :)

[konovod]

googling text of error shows some links:
rust-lang/rust#17677 -> rust-lang/rust#17201 ->

The bad stuff happens when a function with debuginfo is inlined at a call-site that has no associated debuginfo.

http://clang-developers.42468.n3.nabble.com/clang-compilation-error-on-adding-debug-flag-td4050831.html

LLVM will not crash immediately if you bypass this particular assertion, it will just generate bogus debug info somewhere.

[RX14]

@ysbaddaden I'm sure you can, but i'm not sure how that're relevant to this error: the omnibus repo shows that LLVM appears to be compiled in debug mode with assertions turned off. Which makes getting assertion errors more weird. Perhaps the latest commit of the omnibus doesn't exactly represent the state of the distributed package.

[RX14]

Interestingly, i've been noticing lots of bogus debug info in release builds, perhaps this is the cause and somehow we've never had assertions turned on before.

[bararchy]

So, right now I can't install latest Crystal in Arch Linux because of this bug crashing the compiler build in release mode, any fixes or tips to "baypass" this ?

[RX14]

@bararchy This has been limited to the omnibus compiler, and the crystal package in arch community is up to date. Are you using crenv perhaps?

[bararchy]

Nope, building crystal-git from AUR, which show me the same issue, I can post output also if needed

[bararchy]

Using /usr/bin/llvm-config [version=4.0.1]
g++ -c -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong -fno-plt  -o src/llvm/ext/llvm_ext.o src/llvm/ext/llvm_ext.cc `/usr/bin/llvm-config --cxxflags`
cc -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong -fno-plt -fPIC  -D_FORTIFY_SOURCE=2  -c -o src/ext/sigfault.o src/ext/sigfault.c
ar -rcs src/ext/libcrystal.a src/ext/sigfault.o
./bin/crystal build --release  -o .build/crystal src/compiler/crystal.cr -D without_openssl -D without_zlib
crystal: /var/cache/omnibus/src/llvm/llvm-3.8.1.src/lib/CodeGen/LexicalScopes.cpp:160: llvm::LexicalScope* llvm::LexicalScopes::getOrCreateRegularScope(const llvm::DILocalScope*): Assertion `cast<DISubprogram>(Scope)->describes(MF->getFunction())' failed.
/tmp/yaourt-tmp-unshadow/aur-crystal-git/src/crystal-0.23.1-3/bin/crystal: line 102:  7853 Aborted                 (core dumped) "$INSTALL_DIR/embedded/bin/crystal" "$@"
make: *** [Makefile:117: .build/crystal] Error 134

[RX14]

@bararchy yes that package also uses the omnibus as it's bootstrap compiler

[asterite]

What I'm saying is, most of the previous comments suggest this is an LLVM bug that was fixed in 3.9. We should just make sure to use that LLVM version, or a later one, when doing the next release. There's nothing we can do in this repository to mitigate this problem.

[lbguilherme]

It works with LLVM 4.0.1.

[straight-shoota]

I've installed the pre-release debian package 0.24.0-2_amd64 on Ubunutu 14.05.5 LTS in Windows Subsystem for Linux.

$ crystal --version
vCrystal 0.23.0+313 [ea4187c] (2017-10-27)

LLVM: 3.9.1
Default target: x86_64-unknown-linux-gnu

Unfortunately, it seems the issue still persists with LLVM 3.9.1:

$ crystal eval 'puts "Hello World"' --release
crystal: /var/cache/omnibus/src/llvm/llvm-3.9.1.src/lib/CodeGen/LexicalScopes.cpp:159: llvm::LexicalScope* llvm::LexicalScopes::getOrCreateRegularScope(const llvm::DILocalScope*): Assertion `cast<DISubprogram>(Scope)->describes(MF->getFunction())' failed.
/usr/bin/crystal: line 102: 28893 Aborted                 (core dumped) "$INSTALL_DIR/embedded/bin/crystal" "$@"

Adding --no-debug, it compiles without error.

[RX14]

If this omnibus issue still isn't solved, i'm going to have another crack at my statically-compiled omnibus.

[RX14]

I have a new omnibus that I would like people to try and see if it fixes the --release issue for them: https://rx14.co.uk/crystal-0.24.0.tar.gz

You can build it yourself by running ./build.sh inside this repo with docker installed. A tar file will be placed in the build directory.

This package (correctly) only contains a statically compiled libgc. This means that you may have to install some new packages (libevent-dev, libpcre3-dev) to get it to compile. This is essentially the same as when compiling the compiler yourself, but without the requirement for llvm.

[crisward]

If it includes libgc, it could be good to compile it with

./configure --enable-munmap=3 

as per this thread.
#3997 (comment)

It allows it to return memory back to the OS after 3 GC cycles and works really well at allowing processes to return memory after high usage. The GC seems to slow down the frequency of it's cycles during high memory consumption, so 3 cycles isn't as often as you may think.

[ysbaddaden]

We shouldn't enable memory shrink by default: we'll (probably) have another GC in the future, and while growing memory is easy, shrinking is more complex and may not be available or even a goal to achieve.

[crisward]

Long running processes holding onto the memory they need at peak traffic seems wasteful. I'm not aware of any other languages doing this. Node, Go, even PHP return memory after a few minutes. I've been running Crystal with libgc compiled with this flag since that comment in march without any obvious ill effects.

Not using something because it may be difficult to replicate later on seems a little overly cautious and something that could be worked out if the need arises.

[ysbaddaden]

Sure. Implementing a GC with moving pointers to defragment memory isn't hard to implement. On free time. Without money support from Google.

If you need to give back memory then compile boehm-gc manually and be aware that this isn't to be expected by default. A custom GC likely won't have this feature to begin with. Don't give wrong expectations, otherwise implementing a GC for Crystal will never happen, because the mountain to climb will seem absurdly too high.

[RX14]

Per @ivmai

• unmapping+MPROTECT_VDB is not supported yet (so the client app won't be able to use incremental collection (except for Solaris and Windows target) if GC is compiled with unmapping)

This looks like a very good reason not to make this the default.

Still looking for feedback on how well my omnibus build works.

[straight-shoota]

As discussed in chat yesterday, Ubunutu 16.04.3 LTS works.

On Ubuntu 14.04.5 LTS on Windows Subsystem for Linux it throws a segmentation fault.

warning: Error disabling address space randomization: Success

Program received signal SIGSEGV, Segmentation fault.
_start_c (sp=0x7fffe3b66370, dynv=<optimized out>)
    at crt/../ldso/dlstart.c:139
139     in crt/../ldso/dlstart.c

[eregon]

This still happens in the latest available Crystal on Fedora, 0.23.1.

[stronny]

Please take a look at the debian package crystal_0.24.2-1_i386.deb from http://dist.crystal-lang.org/apt, I have the similar/same issue with it:

0:skuld:~/src/zorg$ cat test.cr puts "zorg"
0:skuld:~/src/zorg$ crystal test.cr --release --static crystal: /var/cache/omnibus/src/llvm/llvm-3.9.1.src/lib/CodeGen/LexicalScopes.cpp:159: llvm::LexicalScope* llvm::LexicalScopes::getOrCreateRegularScope(const llvm::DILocalScope*): Assertion `cast<DISubprogram>(Scope)->describes(MF->getFunction())' failed.
/usr/bin/crystal: line 102: 12249 Aborted                 "$INSTALL_DIR/embedded/bin/crystal" "$@"
134:skuld:~/src/zorg$ dpkg -l | grep cryst
ii  crystal                            0.24.2-1                       i386         The full stack of crystal
0:skuld:~/src/zorg$ cat /etc/debian_version
9.4
0:skuld:~/src/zorg$ crystal test.cr
zorg

[RX14]

@stronny crystal -v please, it seems you have an 0.24.2 package built with omnibus and llvm 3.9, which doesn't make sense.

[stronny]

0:skuld:~$ crystal -v
Crystal 0.24.2 [4f9ed8d] (2018-03-08)

LLVM: 3.9.1
Default target: i686-pc-linux-gnu

It does seem like that indeed, however I can't really judge how much sense does it make.

[ysbaddaden]

Doesn't the 32 bit version was still built with omnibus? The next release shall fix that, I believe.

[RX14]

ohh yeah, 32 bit x86, I forgot that exists

[bcardiff]

I've tested building crystal in a vagrant xenial 32 bits machine with llvm-4.0 as part of test-ecosystem and seems to work fine so far.