libc++.so.1 and libc++abi.so.1 not found in latest apt package for LLVM 18

[Masrepus]

This morning I updated my llvm-18 packages to version 1:18.1.8~++20240714071213+3b5b5c1ec4a3-1~exp1~20240714191314.138 via apt.llvm.org and suddenly my binaries built against libc++ and libc++abi can't be executed anymore, stating that libc++.so.1 and libc++abi.so.1 cannot be found. When I tried to find out what is going on, I saw that e.g. /usr/lib/llvm-18/lib/libc++.so is now a text file containing this line: INPUT(libc++.so.1 -lc++abi -lunwind). Not sure if this is related, but it certainly looks wrong.

[ldionne]

It looks like the packages are now using the linker script.

@sylvestre What kind of testing do the apt packages go through? It seems like they get broken quite a bit. Would it make sense to control how they are built from the LLVM monorepo instead? We keep getting bug reports that we can't do anything about because the packages are not controlled by us.

[sylvestre]

@ldionne

yeah, i run two set of integration/packaging tests
https://github.com/opencollab/llvm-toolchain-integration-test-suite/
and an ugly script
https://salsa.debian.org/pkg-llvm-team/llvm-toolchain/-/blob/18/debian/qualify-clang.sh
covering the packaging bugs I fixed in the past.
Both are executed by Debian CI:
https://ci.debian.net/packages/l/llvm-toolchain-18/unstable/amd64/

I think this regression is that previously built binaries were looking for libraries in LLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON (#96210)
the path is now different ?!
@Masrepus could you please try to rebuild your binaries? sorry!

[ldionne]

Thanks for the pointers. Do you actually run the libc++ test suite itself against the generated package? That could replace a good amount of your ad-hoc testing, since we have over 10 000 tests that would almost certainly find the kind of issues folks ran into above.

[Masrepus]

Hm, I did actually try to run my whole build pipeline already with all caches cleaned etc, didn't help. For context, the pipeline compiles the protobuf compiler from source and then as one of the build steps it tries to run it on some other input files. This was the step where it crashed with the message I posted.

[mpusz]

I have the same issue here: https://github.com/mpusz/mp-units/actions/runs/9930500562/job/27508441029.

[sylvestre]

Do you actually run the libc++ test suite itself against the generated package?

From the source tree, yes

Total Discovered Tests: 9637
  Unsupported        :  621 (6.44%)
  Passed             : 8985 (93.23%)
  Expectedly Failed  :   25 (0.26%)
  Unexpectedly Passed:    6 (0.06%)

https://buildd.debian.org/status/fetch.php?pkg=llvm-toolchain-18&arch=amd64&ver=1%3A18.1.8-3&stamp=1721060178&raw=0
(i don't break the build when it is failing - it would be a full time job to monitor it)

But not against the generated packages (too hard, too complex)

I guess the error is:
/home/runner/.conan2/p/b/mp-unf38fae89a4d71/b/build/test/runtime/Debug/unit_tests_runtime: error while loading shared libraries: libc++.so.1: cannot open shared object file: No such file or directory

Do you have some step to reproduce ?

[Masrepus]

My feeling is that reproducing it should be as easy as spinning up some e.g. Ubuntu docker, installing the latest package and then compiling and running something with -stdlib=libc++ -lc++abi

[sylvestre]

Indeed, I can reproduce with:

FROM gcc:13.1
ARG TARGETARCH
ARG VERSION=18

RUN set -ex && \
    apt-get update && \
    apt-get install -y gpg lsb-release wget software-properties-common

RUN set -ex && \
    wget https://apt.llvm.org/llvm.sh && \
    chmod +x llvm.sh && \
    ./llvm.sh ${VERSION} all

RUN echo '#include <vector>\n#include <string>\n#include <iostream>\nusing namespace std;\nint main(void) {\n    vector<string> tab;\n    tab.push_back("the");\n    tab.push_back("world");\n    tab.insert(tab.begin(), "Hello");\n    for(vector<string>::iterator it=tab.begin(); it!=tab.end(); ++it) {\n        cout << *it << " ";\n    }\n    return 0;\n}' > foo.cpp
# Compile and link the C++ code with libc++
RUN clang++-${VERSION} -stdlib=libc++ foo.cpp -o o && \
    ./o > /dev/null

[sylvestre]

I think it is now fixed on apt.llvm.org

[Masrepus]

Thanks for fixing this so quickly! Can't test it yet because there was only an update for llvm 19 available at this time, not for 18, but I assume that's just some apt stuff that makes it take a little while to be available.

[michael-jabbour-sonarsource]

@sylvestre I am still hitting the same issue on Ubuntu 22.04. I could reproduce by changing the Dockerfile example you shared to use FROM ubuntu:22.04 instead of FROM gcc:13.1.

...
#9 44.05 Setting up clang-18 (1:18.1.8~++20240714071213+3b5b5c1ec4a3-1~exp1~20240714191314.138) ...
#9 44.06 Setting up clang-tools-18 (1:18.1.8~++20240714071213+3b5b5c1ec4a3-1~exp1~20240714191314.138) ...
#9 44.06 Setting up clang-tidy-18 (1:18.1.8~++20240714071213+3b5b5c1ec4a3-1~exp1~20240714191314.138) ...
#9 44.07 Setting up libclang-18-dev (1:18.1.8~++20240714071213+3b5b5c1ec4a3-1~exp1~20240714191314.138) ...
#9 44.08 Setting up llvm-18 (1:18.1.8~++20240714071213+3b5b5c1ec4a3-1~exp1~20240714191314.138) ...
#9 44.08 Setting up python3-lldb-18 (1:18.1.8~++20240714071213+3b5b5c1ec4a3-1~exp1~20240714191314.138) ...
#9 44.09 Setting up llvm-18-dev (1:18.1.8~++20240714071213+3b5b5c1ec4a3-1~exp1~20240714191314.138) ...
#9 44.09 Setting up lldb-18 (1:18.1.8~++20240714071213+3b5b5c1ec4a3-1~exp1~20240714191314.138) ...
#9 44.10 Processing triggers for libc-bin (2.35-0ubuntu3.3) ...
#9 DONE 44.4s

#10 [6/7] RUN echo '#include <vector>\n#include <string>\n#include <iostream>\nusing namespace std;\nint main(void) {\n    vector<string> tab;\n    tab.push_back("the");\n    tab.push_back("world");\n    tab.insert(tab.begin(), "Hello");\n    for(vector<string>::iterator it=tab.begin(); it!=tab.end(); ++it) {\n        cout << *it << " ";\n    }\n    return 0;\n}' > foo.cpp
#10 DONE 0.2s

#11 [7/7] RUN clang++-18 -stdlib=libc++ foo.cpp -o o &&     ./o > /dev/null
#11 0.807 ./o: error while loading shared libraries: libc++.so.1: cannot open shared object file: No such file or directory
#11 ERROR: process "/bin/sh -c clang++-${VERSION} -stdlib=libc++ foo.cpp -o o &&     ./o > /dev/null" did not complete successfully: exit code: 127
------
 > [7/7] RUN clang++-18 -stdlib=libc++ foo.cpp -o o &&     ./o > /dev/null:
0.807 ./o: error while loading shared libraries: libc++.so.1: cannot open shared object file: No such file or directory
------
Dockerfile:23
--------------------
  22 |     # Compile and link the C++ code with libc++
  23 | >>> RUN clang++-${VERSION} -stdlib=libc++ foo.cpp -o o && \
  24 | >>>     ./o > /dev/null
  25 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c clang++-${VERSION} -stdlib=libc++ foo.cpp -o o &&     ./o > /dev/null" did not complete successfully: exit code: 127

[sylvestre]

@michael-jabbour-sonarsource yeah, i only triggered the debian builds, just did for ubuntu

[michael-jabbour-sonarsource]

Thank you very much for the quick response and the fix. I can confirm it is working now on Ubuntu.