Build is not reproducible wrt build directory path length

[t184256]

Shared runtime libs like libc++.so have their rpath set to somewhere in the build directory. Then, as one of the very last steps of the installation, these internal references to build directory are erased with RPATH_SET:

Set runtime path of "/...build-dir.../lib/x86_64-unknown-linux-musl/libc++.so.1.0" to ""

This good intention becomes one toe-stubbing brick forming the road to utter confusion.
The problem is, this erasure is not re-linking, and whatever build-dir-path-len-dependent offsets the shared library had because of the varying builddir length are kept intact.
The result looks as if libc++.so/libc++abi.so compiled from same .o files just differ in internal offsets for no good reason.

One (hacky) workaround to that is using a constant-len build-dir. Clearly, there must be a better way.

[ldionne]

Could you check whether compiling with BUILD_WITH_INSTALL_RPATH solves your problem?

[t184256]

No dice. -DBUILD_WITH_INSTALL_RPATH=YES gave me Manually-specified variables were not used by the project: BUILD_WITH_INSTALL_RPATH. With -DCMAKE_BUILD_WITH_INSTALL_RPATH / -DBOOTSTRAP_CMAKE_BUILD_WITH_INSTALL_RPATH I've got no warning, but still got output identical to one w/o them. With them + a builddir of different length I got different hashes.

I've actually thought it's enabled by default when I was filing this bug, because I've seen this line during my investigations:

llvm-project/llvm/cmake/modules/AddLLVM.cmake

Line 2378 in 9243b1b

# Enable BUILD_WITH_INSTALL_RPATH unless CMAKE_BUILD_RPATH is set and not

,
but now when I'm looking at the actual code below it, I'm not sure.

[ldionne]

I am not sure I understand why the current behavior is undesirable, though. Also, note that I don't think we are doing anything specific in libc++ here -- is it possible that this should be filed as an issue against CMake instead?

[ldionne]

I'm not trying to dismiss your report, but I don't see something actionable I could do to resolve this issue at the moment -- I guess I don't really understand how to reproduce it and what the impact is.

[t184256]

OK, I'll try to provide more context, maybe it'll help. Pardon my verbosity, but, since I'm not super-knowledgeable about all this stuff and don't know what specifically would be of help, that verbosity is kind of intentional.

The impact of the issue is that otherwise identical build invocations issued in build directories with different path lengths will end up slightly different, specifically in libcxx.so/libcxxabi.so. I've discovered that when building clang using Nix with and without sandboxing and getting different offsets in libcxx.so/libcxxabi.so linked from identical .o files. Took me a while to arrive at a conclusion that it's because of rpath pointing to a build directory, as rpath was actually stripped from them, so it wasn't immediately obvious why is that happening when I was inspecting the end result.

Once I have realized that rpath stripping is performed and that this stripping preserves build-path-dependent-offsets, I have instated fairly silly workarounds: 1, 2. Despite these hacks solving my original problem, I still decided to file a ticket to fix it upstream because:

1. the fact this was just one single issue preventing me from achieving reproducibility-in-the-face-of-changing-build-directory suggested that this property is cared for, and being just a single isolated issue away from it is just an oversight.
2. the fact that rpath stripping is performed further suggests this is a conscious decision to prevent build outputs from depending on build paths, it's just that the used stripping method does that in a somewhat incomplete fashion

The ways out of this situation I've already come up with are:

0. deciding that build-path-resistant-reproducibility is a non-goal, telling people to just build at fixed paths if they want reproducibility and closing the issue. That'd be a shame, IMO, as it's a useful property and it's almost there
1. telling the linker to provide fixed ample size for rpath, so that it can be re-written within a set size limit freely without disturbing the offsets of the following sections. that'd be a kludgy solution though
2. stripping rpath in a way that relocates stuff around and changes the offsets, and then the builds should arrive at the same .so files after relocations. I think patchelf does that, so maybe it is reasonably doable
3. instead of stripping rpath, perform re-linking libcxx/libcxxabi from the same .o files, but without setting rpath to a build-dir-dependent path. Might be the easiest way out of the situation, at the expense of wasting some resources on linking these two twice.

Hope that helped.

[ldionne]

Ok, thanks a lot for this additional context, it is indeed really helpful. I agree this is a problem worth solving -- I don't know whether we ever had that on our radar, but it's a good property to have IMO.

It looks like CMake has a bunch of variables to control this very behavior:
BUILD_WITH_INSTALL_RPATH: https://cmake.org/cmake/help/v3.27/prop_tgt/BUILD_WITH_INSTALL_RPATH.html
SKIP_BUILD_RPATH: https://cmake.org/cmake/help/v3.27/prop_tgt/SKIP_BUILD_RPATH.html

In particular, the documentation for CMAKE_SKIP_BUILD_RPATH says:

Normally CMake uses the build tree for the RPATH when building executables etc on systems that use RPATH. When the software is installed the executables etc are relinked by CMake to have the install RPATH. If this variable is set to TRUE then the software is always built with no RPATH.

So normally, CMake re-links the libraries at the point of installation to get the right RPATH (possibly to solve this very issue). It must be the case that we somehow inhibit the re-linking step from happening (and indeed when I run install-cxx I don't see any re-linking happening, or if it happens then it's not logged anywhere).

[t184256]

From reading the documentation of this options, I'm left with an impression that none of the three cases described there:

1. default with re-linking using install RPATH during installation
2. not specifying rpath at all using SKIP_BUILD_RPATH
3. specifying BUILD_WITH_INSTALL_RPATH

would manifest the effect I'm observing. So the theory that something custom is being done instead sounds plausible. This, or CMake is buggy =)

The message from the first comment seems to come from
https://github.com/Kitware/CMake/blob/dd949c77cb94c59616bd288c10da6f315c1e1a52/Source/cmFileCommand.cxx#L1015

which doesn't look like it bothers with relocations:

https://github.com/Kitware/CMake/blob/dd949c77cb94c59616bd288c10da6f315c1e1a52/Source/cmSystemTools.cxx#L2968

[ldionne]

Can you check whether #67835 would solve your issue? You could apply it locally and rebuild your stuff.

[t184256]

[100%] Built target clang-ast-dump
[100%] ASTNodeAPI.json
Error loading shared library libLLVM-18git.so: No such file or directory (needed
 by ../../../../bin/clang-ast-dump)
Error relocating ../../../../bin/clang-ast-dump: _ZN4llvm17deallocate_bufferEPvm
m: symbol not found
Error relocating ../../../../bin/clang-ast-dump: _ZN4llvm4json6isUTF8ENS_9String
RefEPm: symbol not found