Failed rlib archive processing with rustc 1.17 compiled against LLVM 4.0

[copumpkin]

I apologize that this isn't the best bug description, but after several hours on this, I still haven't been able to reduce it to get a small repro that would fit the template, so I'll describe what I know so far and hope that someone has hints for how to debug further so I can improve this issue report (or close it if it turns out to be unrelated to rustc!)

I'm running on macOS with rustc 1.17.0 compiled against a mostly vanilla LLVM 4.0, under the Nix package manager (which doesn't have 1.18.0 yet or I'd try that; it doesn't look like the relevant code has changed much so I haven't put much effort on that front yet).

When I try to build https://github.com/pantsbuild/pants/tree/master/src/rust/engine in that environment (it's a Python module written in rust), I get most of the way through compilation but then rustc barfs with the following message:

error: failed to build archive: bad archive: truncated or malformed archive (offset to next archive member past the end of the archive after member collections-ef2067477795e5ee.9.bytecode.deflate)

(the failure is consistent but not always on the collections library; I've seen it happen on libstd too)

The strings in that error message tell me that it's likely coming from https://github.com/rust-lang/rust/blob/1.17.0/src/librustc_trans/back/archive.rs#L228 and one of the two iterator loops over archive children below there (due to the "bad archive" error snippet that's added to each child).

Looking at the relevant rlibs, I can get pure LLVM to spit out exactly the same error message just by running llvm-nm (which is conveniently bundled with the macOS developer kit) against it:

$ nm /nix/store/kwn3vvcbl7rwxlz8j3vppdwwqjkkl3mz-rustc-1.17.0/lib/rustlib/x86_64-apple-darwin/lib/libcollections-ef2067477795e5ee.rlib

[lots of reasonable symbols]

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/nm: /nix/store/kwn3vvcbl7rwxlz8j3vppdwwqjkkl3mz-rustc-1.17.0/lib/rustlib/x86_64-apple-darwin/lib/libcollections-ef2067477795e5ee.rlib(rust.metadata.bin) The end of the file was unexpectedly encountered

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/nm: /nix/store/kwn3vvcbl7rwxlz8j3vppdwwqjkkl3mz-rustc-1.17.0/lib/rustlib/x86_64-apple-darwin/lib/libcollections-ef2067477795e5ee.rlib truncated or malformed archive (offset to next archive member past the end of the archive after member collections-ef2067477795e5ee.9.bytecode.deflate)

I think the warning on rust.metadata.bin is expected since it's not something that nm understands, but the final error message about the truncated or malformed archive is identical to what rustc spits out.

What I've been able to make of this is that something between rustc, LLVM 4, or my weird Nix build of the two of them together is either producing invalid rlib/.a archives or producing valid archives that the LLVM archive iterator then doesn't know how to handle when it consumes them again later.

As an additional data point, Nix allows us to run almost exactly the same build on multiple platforms (almost same combination of flags, dependencies, etc.) and I can run the exact same build just fine on Linux. So this seems to be some unhappiness with Darwin in particular...

Anyway, I'm not expecting anyone to be able to figure out the issue based on the weird situation I describe above, but would appreciate any debugging hints anyone has to help me track this down; if this turns out to be a real issue in rustc or LLVM, I'll file new issues or update this one as I learn more, but right now I'm getting kind of stumped. My best debugging approach so far is creating (sloow) custom rustc builds with prints sprinked across relevant code to see what's going on, so I'd welcome anything less painful than that.

cc @alexcrichton who is seemingly deeply involved in archive.rs and friends

Verbose version

/nix/store/kwn3vvcbl7rwxlz8j3vppdwwqjkkl3mz-rustc-1.17.0/bin/rustc --version --verbose
rustc 1.17.0
binary: rustc
commit-hash: unknown
commit-date: unknown
host: x86_64-apple-darwin
release: 1.17.0
LLVM version: 4.0

Edit: I've tried various -Z options that sound potentially helpful but none of them told me anything around the point of the crash. Here are some of the ones I tried to no avail: print-llvm-passes, debug-llvm, verbose, print-link-args

[alexcrichton]

Hm it sounds like the rlib is likely corrupt here, although it's certainly odd how that happened! This may be a bug with the way nix builds rustc? Or maybe a bug in our own build system? Hard to say...

[copumpkin]

Okay cool. Wasn't sure if rlib files were "expected" to be unreadable by llvm-nm for some reason. I know they're mostly archive files with custom rust metadata and ar -t doesn't seem to barf (or if it does, it doesn't barf visibly on that file). If llvm-nm is expected to not barf on that file I'll try harder to figure out what happened when it was created.

[alexcrichton]

Yeah currently rlibs should be valid ar archives, if they're not then something has gone wrong!

[copumpkin]

Getting very confused at this point: my libstd rlib is still "invalid" but I'm able to build (and run) a ton of stuff with this compiler. Tried a bunch of assorted tools like ripgrep and they all built just fine. So it doesn't seem like the "invalid rlib" thing is invalid enough to make rustc unhappy most of the time. Perhaps this is only triggered when building something other than a standalone executable with rust? I'm going to poke at it some more but figured I'd update with whatever I found.

[copumpkin]

Aha, I got it to work again by switching my crate-type = dylib to crate-type = cdylib, which IRC says is almost always what I want. So I guess either it's a legitimate bug with dylib and that code path doesn't get exercised much, or the project I was trying to build was just doing something wrong.

[copumpkin]

Turns out cdylib isn't quite working, but at least the rust build completes. It doesn't seem to integrate some external C code integrated via the gcc crate but I'll try to figure that out separately.

Will continue to poke at this issue to figure out why my compiler is so unhappy with the dylib crate-type.

[copumpkin]

I have a minimal reproduction, but it's a Nix expression; I haven't (yet?) been able to get it to fail outside of Nix. If you're curious, it's here: NixOS/nixpkgs#27370

[copumpkin]

A few more data points: I've tried compiling rustc (1.15 through 1.17) against its bundled LLVM and that doesn't help. The only thing I've been able to get to work is explicitly compiling rustc against LLVM 3.7, which only seems to happen because LLVM didn't have the assertions that are failing today back then.

Getting more and more stumped at why this doesn't seem to affect official rust builds and only affects the Nix ones; I've tried copying most of the build/configure flags from https://github.com/rust-lang/rust/blob/master/src/ci/run.sh and none of them helped.

I'm starting to wonder if it could be the ld we're using somehow (which is not /usr/bin/ld but should be very similar). As I understand it, rustc has LLVM generate .o files, then calls the system linker to make them into .a files and then uses LLVM machinery to pack those into .rlibs (not the system ar). So if somehow our ld does unexpected things when generating the .a file before it goes into the .rlib, that might cause this issue.

[jackh726]

I almost went to close this as not actionable, but it seems someone has hit this as lately as February. It's tough though, because I don't even know how to begin to reproduce with a modern version of rustc.

[Enselic]

Triage: Yes, let's close. LLVM 4 is really old now. If this is still a problem it is better to file a new issue with up to date info about toolchains used, etc, etc.