Crash in llvm-lipo (with minimal repro)

[sztomi]

Stacktrace

0.      Program arguments: /Users/tamas/.conan_plex/v2/data/clang-bootstrap/15.0.4-5c68a1cb/_/_/package/0dadf5e389aec9c830aab3a96420c9a50871a193/bin/lipo -create -output PrettyStackTrace.o /va
r/folders/qf/hwn8fdhx6k329g6vh0fv9c940000gn/T/PrettyStackTrace-642083/PrettyStackTrace-x86_64.o /var/folders/qf/hwn8fdhx6k329g6vh0fv9c940000gn/T/PrettyStackTrace-b71132/PrettyStackTrace-arm64.
o
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
0  lipo                     0x0000000108b37feb llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) + 43
1  lipo                     0x0000000108b36f74 llvm::sys::RunSignalHandlers() + 260
2  lipo                     0x0000000108b38640 SignalHandler(int) + 272
3  libsystem_platform.dylib 0x00007fff20472d7d _sigtramp + 29
4  libsystem_platform.dylib 0x0002ffff00001f80 _sigtramp + 703686900249120
5  lipo                     0x0000000108ac3c79 llvm::ModuleSymbolTable::addModule(llvm::Module*) + 329
6  lipo                     0x0000000108ab42cb llvm::object::IRObjectFile::create(llvm::MemoryBufferRef, llvm::LLVMContext&) + 1291
7  lipo                     0x0000000108ac731f llvm::object::SymbolicFile::createSymbolicFile(llvm::MemoryBufferRef, llvm::file_magic, llvm::LLVMContext*, bool) + 495
8  lipo                     0x0000000108a8c6d7 llvm::object::createBinary(llvm::MemoryBufferRef, llvm::LLVMContext*, bool) + 119
9  lipo                     0x0000000108a8c9b6 llvm::object::createBinary(llvm::StringRef, llvm::LLVMContext*, bool) + 198
10 lipo                     0x0000000108a7e8ed readInputBinaries(llvm::LLVMContext&, llvm::ArrayRef<(anonymous namespace)::InputFile>) + 157
11 lipo                     0x0000000108a7de39 main + 4201
12 libdyld.dylib            0x00007fff20448f3d start + 1
13 libdyld.dylib            0x0000000000000006 start + 18446603339974799562
clang-15: error: unable to execute command: Segmentation fault: 11
clang-15: error: lipo command failed due to signal (use -v to see invocation)
PlexBootstrap clang version 15.0.4
Target: arm64-apple-darwin20.6.0
Thread model: posix
InstalledDir: /Users/tamas/.conan_plex/v2/data/clang-bootstrap/15.0.4-5c68a1cb/_/_/package/0dadf5e389aec9c830aab3a96420c9a50871a193/bin
clang-15: note: diagnostic msg: Error generating preprocessed source(s) - cannot generate preprocessed source with multiple -arch options.

Steps to repro

1. Symlink llvm-lipo to lipo in the compiler install dir
2. Use the following source and clang invocation

asm(".desc ___crashreporter_info__, 0x10");

clang++ -flto=thin -arch x86_64 -arch arm64 -c PrettyStackTrace.cpp 

The flags are also minimal.

My steps to minimize the case (in a not too fine granularity) can be found in the git history here, in case that's interesting: https://github.com/sztomi/llvm-lipo-crash-repro

Context

I was trying to build a PGO-optimized clang on macOS Big Sur using the host Xcode (Apple clang version 13.0.0 (clang-1300.0.29.30)). I ran into all sorts of issues with libtool and lipo:

warning: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: Invalid value (Producer: 'LLVM15.0.4' Reader: 'LLVM APPLE_1_1300.0.29.30_0')
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: file: lib/Demangle/CMakeFiles/LLVMDemangle.dir/Demangle.cpp.o is not an object file (not
allowed in a library)

and

warning: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/lipo: Invalid value (Producer: 'LLVM15.0.4' Reader: 'LLVM APPLE_1_1300.0.29.30_0')
fatal error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/lipo: can't figure out the architecture type of: /var/folders/qf/hwn8fdhx6k329g6vh0fv9c94000
0gn/T/getErrc-9db0ee/getErrc-x86_64.o

While I was able to use llvm-libtool-darwin in place of the Xcode libtool via setting CMAKE_LIBTOOL, lipo can't be replaced the same way because it's hardcoded in the driver. So I symlinked llvm-lipo to lipo in the install dir (right next to my stage1 clang I was using).

[tru]

It seems like this might have been fixed in 61e5438 by @thevinster

[tru]

I think there is another thing that could be fixed here btw. I think the driver should look for llvm-lipo if lipo is not found. What do you think @keith ?

[keith]

I think that would be nice. Although in the macOS case lipo will always exist so I don't think that would have changed anything here.

[sztomi]

I think the driver should look for llvm-lipo if lipo is not found.

But lipo is found in a typical macOS build environment and making it "un-findable" as a means of controlling which lipo to use is quite clunky. I'd rather have a more explicit way to control it (like -fuse-lipo=llvm-lipo flag as we discussed on Discord).

[sztomi]

I cherry-picked dde23bf and 61e5438. It seems that the crash is fixed (though I did get another, unrelated error: error: archive member blake3_sse2_x86-64_unix.S.o is a MachO, while previous archive member blake3.c.o was an IR LLVM object). It seems there is an implicit requirement for a certain Apple-clang version to build a LTO LLVM, so I'm going to bite the bullet and upgrade. As for this other error and setting the desired lipo command, I think I will open other issues. If there is consensus about the lipo situation, I'll try and put together a patch for that.

[keith]

Note that in the case of apple's tools you can sometimes override the LTO library path if needed with env vars. LIBLTO_PATH is respected by at least apple's libtool

[keith]

Is there a link to the -fuse-lipo discussion on discord? I don't see anything from a search of the llvm discord

[keith]

Ah I think another option for overriding is to create a directory containing only llvm-lipo named lipo and pass -Bpath/to/dir to clang. As executables are looked up in those paths first

llvm-project/clang/lib/Driver/Driver.cpp

Lines 5786 to 5798 in 70ac598

	// Respect a limited subset of the '-Bprefix' functionality in GCC by
	// attempting to use this prefix when looking for program paths.
	for (const auto &PrefixDir : PrefixDirs) {
	if (llvm::sys::fs::is_directory(PrefixDir)) {
	SmallString<128> P(PrefixDir);
	if (ScanDirForExecutable(P, Name))
	return std::string(P.str());
	} else {
	SmallString<128> P((PrefixDir + Name).str());
	if (llvm::sys::fs::can_execute(Twine(P)))
	return std::string(P.str());
	}
	}

[sztomi]

Is there a link to the -fuse-lipo discussion on discord? I don't see anything from a search of the llvm discord

Sorry, it was in DMs. But we pretty much only said what was also repeated here.

another option for overriding is to create a directory containing only llvm-lipo named lipo and pass -Bpath/to/dir to clang

Sounds like it could work, but then again, I can also make a symlink in the install dir which seems like a cleaner option to me (and I have to call it lipo anyway).

[tru]

IMHO I just wish there was a way for us to use the full llvm toolchain if you want to avoid problems with LTO. This is probably better discussed in a new issue since this one is full of scope creep now.

But I think a flag like -fuse-llvm-darwin-tools or something that would opt-in to use llvm-libtool, llvm-lipo etc instead of the system one would be a good thing.

[sztomi]

@tru @keith opened the above for further discussion ^