Link errors for tbb and xxhash libraries

[rhauser]

Check duplicate issues.

• Checked for duplicates

Description

This is for the specific ROOT build configuration used in LCG's dev3 nightly where recently more and more of the builtins are turned off. Last week we encountered link errors because xxhash is now coming from a place external to ROOT (but still from the LCG cvmfs area, not the system).

ATLAS sees this issue in both online and offline software and has worked around similar issues with vdt and tbb in the past. However, it seems this should be fixed on the ROOT side instead.

In the reproducer a minimal environment is setup that I would expect to work. The creation of a shared library linking against ROOT::Core and ROOT::Hist succeeds, while linking an executable against the same two libraries fails, complaining about not finding the tbb and xxhash libraries.

[ 75%] Linking CXX executable example
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.40-acaab/x86_64-el9/bin/ld: warning: libtbb.so.12, needed by /cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Mon/ROOT/HEAD/x86_64-el9-gcc14-opt/lib/libThread.so, not found (try using -rpath or -rpath-link)
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.40-acaab/x86_64-el9/bin/ld: warning: libxxhash.so.0, needed by /cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Mon/ROOT/HEAD/x86_64-el9-gcc14-opt/lib/libCore.so, not found (try using -rpath or -rpath-link)
[...]

I assume tbb and xxhash are considered internal implementation details of ROOT, that's why they do not appear anywhere in the ROOT-targets*.cmake files as dependencies, and no find_dependency() call is made. That sounds logical as a client never references any symbols from these libraries directly. And indeed this is fine for a shared library as the reproducer shows.

However, the standard Linux linker (but not mold/gold/... it seems) has a unique behaviour where it tries to resolve symbols even in indirectly referenced libraries when the final executable is linked. The exact search is described under --rpath-link option in the ld man page.

A simple test is to add the tbb and xxhash library paths to LD_LIBRARY_PATH : the linking now succeeds. However, this is not something we do: LD_LIBRARY_PATH is a variable for the run-time dynamic linker and should not be needed at link time.

Shouldn't ROOT have both TBB::tbb and xxHash::xxHash in the INTERFACE_LINK_LIBRARIES of the exported targets, where relevant, together with the find_depency() calls in the case where these libraries are external to ROOT ?

The alternative is that every client who links an executable has to know what libraries are ROOT builtins or not and has to modify their own CMakeLists.txt file accordingly.

Having to set LD_LIBRARY_PATH is similar cumbersome - for a bootstrap build we generate our run-time setup script at the end of the build where we know what is needed, and it is not straightforward to inject environment variables into the build environment from cmake itself (i.e. not having the user do it explicitly by sourcing some file etc.). Never mind that this is a variable meant to be used at run-time.

Reproducer

On an Alma 9 host with cvmfs:

. ./setup.sh
cmake -B build
cmake --build build

setup.sh

. /cvmfs/sft-nightlies.cern.ch/lcg/contrib/gcc/14.2.0/x86_64-el9/setup.sh
export CMAKE_PREFIX_PATH=/cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Mon/xxHash/0.8.3/x86_64-el9-gcc14-opt:/cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Mon/vdt/0.4.4/x86_64-el9-gcc14-
opt:/cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Mon/tbb/2022.2.0/x86_64-el9-gcc14-opt:/cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Mon/ROOT/HEAD/x86_64-el9-gcc14-opt

CMakeLists.txt

cmake_minimum_required(VERSION 3.26)

project(rootTest LANGUAGES CXX)

find_package(ROOT REQUIRED COMPONENTS Core Hist)

add_library(libexample SHARED libexample.cxx)
target_link_libraries(libexample PRIVATE ROOT::Core ROOT::Hist)

add_executable(example example.cxx)
target_link_libraries(example PRIVATE ROOT::Core ROOT::Hist)

libexample.cxx

#include "TH1F.h"

void f()
{
   auto h = new TH1F("t", "t", 100, 0.0, 100.0);
   delete h;
}

example.cxx

#include "TH1F.h"

int main()
{
    auto h = new TH1F("Test", "Test", 100, 0.0, 100.0);
}

ROOT version

  ------------------------------------------------------------------
  | Welcome to ROOT 6.39.01                        https://root.cern |
  | (c) 1995-2025, The ROOT Team; conception: R. Brun, F. Rademakers |
  | Built for linuxx8664gcc on Apr 19 2026, 22:52:36                 |
  | From heads/master@0481b1d8                                       |
  | With g++ (GCC) 14.3.0 std202002                                  |
  | Try '.help'/'.?', '.demo', '.license', '.credits', '.quit'/'.q'  |
   ------------------------------------------------------------------

Installation method

Pre-built by SFT/LCG team on /cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Mon/ROOT

Operating system

Linux Alma 9

Additional context

Was discussed in yesterday's LIM meeting and we were asked to open a ticket directly here.

[rhauser]

Mentioning @andresailer @krasznaa here FYI.

[ferdymercury]

Side note:
#21806 and https://github.com/root-project/root/pull/21806/changes#r3093303976
and
#21904

use more proper CMake targets for tbb and xxhash
(but not sure if that would change anything for this issue.)

[andresailer]

When builtin_xxhash=ON the static library of xxhash is used. Maybe the FindXXHash could be adapted to preferably pick up the static library?

root/builtins/xxhash/CMakeLists.txt

Line 26 in 9689118

add_library(xxhash STATIC xxhash.h xxhash.c)

We have the static library in our xxhash folder (also the dynamic one)

ls -l /cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Mon/xxHash/0.8.3/x86_64-el9-gcc14-opt/lib/
total 106
-rw-r--r--. 1 cvmfs cvmfs 50408 Apr 14 00:08 libxxhash.a
[snip]

[krasznaa]

Let me second Andre's suggestion. Since we lived a happy life so far without being exposed to libCore.so's private dependence on xxHash, I think there would be value in keeping it so.

Or at least there would be value in having the ability to build ROOT such that it would link libCore.so against the static version of libxxhash.a. As that could be useful in the LCG build for a number of clients. Even if the default build would pick up the shared library. As that's definitely a better setup for a "system installation" of ROOT. 🤔

[hageboeck]

I would like to ask what the expectation is for a package to work out of the LCG stack. Let me go through a few cases, from the most barebones to the "I include all packages by using a view":

1. When trying to invoke an executable directly from cvmfs, we expect this to break, don't we? There has never been an attempt to set enough runpaths for everything to work just "out of the box" on cvmfs. The same goes for linking to a library without doing anything in addition.
2. When setting up a compiler and trying to use a package, I still expect this to break because of not attempting to set all the required runpaths.
3. When setting up a compiler and setting a sufficient number of LD_LIBRARY_PATHs, I would expect packages to start working. I haven't tested it for ROOT, but this seems to be what some users were doing. (I don't know if that's how the LCG stack is supposed to be used, though. It is certainly a bit cumbersome to inch your way to a working LD_LIBRARY_PATH.)
4. Now, the case described in this issue: When setting up a compiler and trying to link to a few selected packages by setting CMAKE_PREFIX_PATH but not setting any LD_LIBRARY_PATH, using the argument in 1., I would expect this to break. To my knowledge, there has never been an attempt to set enough runpaths for this to work, so one would have to fall back to setting library paths.
5. And finally, we probably all agree that when sourcing an LCG view, all packages are supposed to work. ROOT is built to support this case, as its runpaths are set up as "when you need libxxx, check in the same folder as the library that needs it". This, however, only works when all libs are found in the same folder, which is only supported by the view.

I'm testing currently whether we can leverage CMake to resolve link paths to their actual install location, and potentially include those as runpaths in the ROOT libraries. This might enable case 4., but first of all I'm not sure if this should ever have been supported, and secondly it seems to be targeted to the cvmfs case. We might get in trouble in conda or similar environments, so I can't promise yet that this is going to work.

Update:
CMake should have recorded the runpaths to the externals. It's set by default in ROOT's config, and I could verify that it works with a manually installed libxxhash in a non-standard location. So the solution to the problem might be to investigate why this didn't work on the LCG build.

[rhauser]

I do not expect anything to run. In practice we start with sourcing the setup.sh for a LCG compiler and binutils, our own CMake version (typically) and that's it. As long as all dependencies are properly described for CMake we should be able to compile and link. We read the information about the LCG release from LCG_externals_*.txt and set XXXX_ROOT etc. for each package, or CMAKE_PREFIX_PATH if package name and LCG name differ.

When we run rootcling e.g. we have to indeed inject stuff into the environment, but that is done via add_custom_command(). and rootcling is essentially the only external code generator for us.

Regarding runpath: I tried several years ago and at the end decided it's not worth it if the (non-view) LCG installation is not prepared for it either. And I couldn't convince my ATLAS colleagues to use LCG view ;)

[rhauser]

3. When setting up a compiler and setting a sufficient number of LD_LIBRARY_PATHs, I would expect packages to start working. I haven't tested it for ROOT, but this seems to be what some users were doing. (I don't know if that's how the LCG stack is supposed to be used, though. It is certainly a bit cumbersome to inch your way to a working LD_LIBRARY_PATH.)
4. Now, the case described in this issue: When setting up a compiler and trying to link to a few selected packages by setting CMAKE_PREFIX_PATH but not setting any LD_LIBRARY_PATH, using the argument in 1., I would expect this to break. To my knowledge, there has never been an attempt to set enough runpaths for this to work, so one would have to fall back to setting library paths.

Let me clarify why I mentioned LD_LIBRARY_PATH: I first tried to create a small reproducer by just sourcing the ROOT-env.sh file in the LCG ROOT package. This sets up the correct runtime environment for all that is needed. The error did not show up. So I thought they do something with CMAKE_PREFIX_PATH etc. that I missed. In the end it turned out to be LD_LIBRARY_PATH. But that is definitely a configuration for the run-time linker, not ld, which confused me a lot. Also that the problem does not show up when creating a shared library, but only for the final executable

LD_LIBRARY_PATH is only mentioned once in the gigantic ld man page and it takes careful reading to see why it applies here (plus an explanation on stack overflow I found).

[jmcarcell]

@hageboeck runpaths are removed for LCG packages. This gives us two options, either libxxhash.so is in LD_LIBRARY_PATH (how it works for LCG views), or the static version is used (this was the case with builtin_xxhash enabled, rigth? I don't see libxxhash.so inside the ROOT installation for older versions). We found this when running some tests without setting up the view, trying to use dlopen on libCore.so will not work because it won't be able to find libxxhash.so. I think even if CMake was changed to be able to link using ROOT wouldn't work (with the LCG settings):

# source /cvmfs/sft.cern.ch/lcg/contrib/gcc/15.2.0/x86_64-el9/setup.sh
# source /cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Tue/ROOT/HEAD/x86_64-el9-gcc15-opt/bin/thisroot.sh
# root
/cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Tue/ROOT/HEAD/x86_64-el9-gcc15-opt/bin/root.exe: error while loading shared libraries: libxxhash.so.0: cannot open shared object file: No such file or directory

Note that this is already happening to other libraries, ldd libThread.so won't find libtbb.so but these are being correctly set by CMake downstream and maybe not opened as often.

[hageboeck]

Hello @jmcarcell and @rhauser,

If you say that runpaths are not an option, ROOT indeed needs either to be used as part of view or we have to go the route of using multiple setup scripts. For example, setting up tbb solves the problem you outlined:

# source /cvmfs/sft.cern.ch/lcg/contrib/gcc/15.2.0/x86_64-el9/setup.sh
# source /cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Tue/ROOT/HEAD/x86_64-el9-gcc15-opt/bin/thisroot.sh
# source /cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Tue/tbb/2022.2.0/x86_64-el9-gcc15-opt/tbb-env.sh 
$ root.exe
   ------------------------------------------------------------------
  | Welcome to ROOT 6.39.01                        https://root.cern |
  | (c) 1995-2025, The ROOT Team; conception: R. Brun, F. Rademakers |
  | Built for linuxx8664gcc on Apr 20 2026, 23:05:20                 |
$ ldd -r /cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Tue/ROOT/HEAD/x86_64-el9-gcc15-opt/lib/libThread.so 
	linux-vdso.so.1 (0x00007fffcc9d6000)
	libCore.so => /cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Tue/ROOT/HEAD/x86_64-el9-gcc15-opt/lib/libCore.so (0x00007ff4f5800000)
	libtbb.so.12 => /cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Tue/tbb/2022.2.0/x86_64-el9-gcc15-opt/lib64/libtbb.so.12 (0x00007ff4f5d73000)
	libstdc++.so.6 => /cvmfs/sft.cern.ch/lcg/releases/gcc/15.2.0-35657/x86_64-el9/lib64/libstdc++.so.6 (0x00007ff4f5400000)
	libm.so.6 => /lib64/libm.so.6 (0x00007ff4f5725000)
	libgcc_s.so.1 => /cvmfs/sft.cern.ch/lcg/releases/gcc/15.2.0-35657/x86_64-el9/lib64/libgcc_s.so.1 (0x00007ff4f5d24000)
	libc.so.6 => /lib64/libc.so.6 (0x00007ff4f5000000)
	/lib64/ld-linux-x86-64.so.2 (0x00007ff4f5e3b000)
	libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007ff4f5689000)
	libz.so.1 => /lib64/libz.so.1 (0x00007ff4f5d0a000)
	liblzma.so.5 => /lib64/liblzma.so.5 (0x00007ff4f5cde000)
	libxxhash.so.0 => /lib64/libxxhash.so.0 (0x00007ff4f53ed000)
	liblz4.so.1 => /lib64/liblz4.so.1 (0x00007ff4f53c9000)
	libzstd.so.1 => /lib64/libzstd.so.1 (0x00007ff4f5312000)

That being said, given that the static xxhash library is on cvmfs, we can remove xxhash from the list of packages that need to be set up if somebody wants to use ROOT outside of a view. @dpiparo is already looking into it.

[dpiparo]

Hello,

in general it may not be useful to instruct a library, including ROOT, to keep track of the location of the libraries it depended on at build time.
If packages from the LCG releases are required, the full lcg-view environment can be set, or, alternatively, if only a few are needed, the dependencies need to be set by hand, e.g. by crafting the environment accordingly.

For the specific case of xxhash, #21996 is proposed, in order to look with priority to the static version of libxxhash on the system, therewith restoring the symmetry with the builtin mechanism as mentioned above.