After 'brew upgrade' llvm@14 got installed and is now broken

[root42]

The last "brew upgrade" installed the new llvm 14.0.6 (before it was llvm 13 or something like that) and now clang and clang++ are broken. I get this when I run clang:

$ clang++                                                                                     127 ↵
clang++: /home/linuxbrew/.linuxbrew/lib/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by clang++)
clang++: /home/linuxbrew/.linuxbrew/lib/libstdc++.so.6: version `GLIBCXX_3.4.26' not found (required by /home/linuxbrew/.linuxbrew/Cellar/llvm/14.0.6/lib/libclang-cpp.so.14)
clang++: /home/linuxbrew/.linuxbrew/lib/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /home/linuxbrew/.linuxbrew/Cellar/llvm/14.0.6/lib/libclang-cpp.so.14)
clang++: /home/linuxbrew/.linuxbrew/lib/libstdc++.so.6: version `CXXABI_1.3.11' not found (required by /home/linuxbrew/.linuxbrew/Cellar/llvm/14.0.6/lib/libclang-cpp.so.14)
clang++: /home/linuxbrew/.linuxbrew/lib/libstdc++.so.6: version `CXXABI_1.3.13' not found (required by /home/linuxbrew/.linuxbrew/Cellar/llvm/14.0.6/lib/libclang-cpp.so.14)
clang++: /home/linuxbrew/.linuxbrew/lib/libstdc++.so.6: version `GLIBCXX_3.4.22' not found (required by /home/linuxbrew/.linuxbrew/Cellar/llvm/14.0.6/lib/libclang-cpp.so.14)
clang++: /home/linuxbrew/.linuxbrew/lib/libstdc++.so.6: version `CXXABI_1.3.11' not found (required by /home/linuxbrew/.linuxbrew/Cellar/llvm/14.0.6/lib/libLLVM-14.so)
clang++: /home/linuxbrew/.linuxbrew/lib/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /home/linuxbrew/.linuxbrew/Cellar/llvm/14.0.6/lib/libLLVM-14.so)
clang++: /home/linuxbrew/.linuxbrew/lib/libstdc++.so.6: version `CXXABI_1.3.13' not found (required by /home/linuxbrew/.linuxbrew/Cellar/llvm/14.0.6/lib/libLLVM-14.so)
clang++: /home/linuxbrew/.linuxbrew/lib/libstdc++.so.6: version `GLIBCXX_3.4.22' not found (required by /home/linuxbrew/.linuxbrew/Cellar/llvm/14.0.6/lib/libLLVM-14.so)
clang++: /home/linuxbrew/.linuxbrew/lib/libstdc++.so.6: version `GLIBCXX_3.4.26' not found (required by /home/linuxbrew/.linuxbrew/Cellar/llvm/14.0.6/lib/libLLVM-14.so)

I could reinstall llvm from source, but that always takes hours, so I want to avoid that. Any ideas what's wrong here?
I am using Ubuntu focal 20.04.4 LTS, if that makes any difference.

[carlocab]

What is the output of brew config, brew doctor, brew info llvm gcc?

[root42]

Here you go:

brew-config.txt
brew-doctor.txt
brew-info.txt

Note that I removed the error output regarding clang -- which is the same as above -- as not to clutter the files.

[carlocab]

Hmmm, no idea, unfortunately. @Homebrew/linux?

[xsteadfastx]

same for me.

[digitalsparky]

Same goes for me too.

I've attached my outputs (as one file) to hopefully give you some more info to go on, hopefully it helps.

output.txt

[danielnachun]

I've reproduced this and I know exactly what's happening here. The root cause of the problem is that we have to bypass our compiler shims when doing the profile guided optimization build so we manually add back the right flags:

      # Add linker flags on Linux so that stage1 clang can find libstdc++ from brewed GCC and all of the other
      # brewed libraries that need to be linked.
      unless OS.mac?
        ldflags = %W[
          -L#{libstdcxx_dir}
          -Wl,-rpath,#{libstdcxx_dir}
          -L#{HOMEBREW_PREFIX}/lib
          -Wl,-rpath,#{HOMEBREW_PREFIX}/lib
        ]
        args << "-DCMAKE_EXE_LINKER_FLAGS=#{ldflags.join(" ")}"
        args << "-DCMAKE_SHARED_LINKER_FLAGS=#{ldflags.join(" ")}"
      end

What is supposed to happen here is that HOMEBREW_PREFIX/opt/gcc/lib/gcc/11 is should precede HOMEBREW_PREFIX/lib in the RPATH, so that libstdc++ from GCC 11 is found before libstdc++ from GCC 5. However, when I look at the RPATH for libLLVM-14.so, for example, I see this:

/home/dnachun/.linuxbrew/Cellar/llvm/14.0.6/lib
/home/dnachun/.linuxbrew/opt/xz/lib
/home/dnachun/.linuxbrew/opt/gdbm/lib
/home/dnachun/.linuxbrew/opt/mpdecimal/lib
/home/dnachun/.linuxbrew/opt/openssl@1.1/lib
/home/dnachun/.linuxbrew/opt/ncurses/lib
/home/dnachun/.linuxbrew/opt/readline/lib
/home/dnachun/.linuxbrew/opt/zlib/lib
/home/dnachun/.linuxbrew/opt/sqlite/lib
/home/dnachun/.linuxbrew/opt/bzip2/lib
/home/dnachun/.linuxbrew/opt/expat/lib
/home/dnachun/.linuxbrew/opt/libffi/lib
/home/dnachun/.linuxbrew/opt/libxcrypt/lib
/home/dnachun/.linuxbrew/opt/python@3.10/lib
/home/dnachun/.linuxbrew/opt/libedit/lib
/home/dnachun/.linuxbrew/opt/icu4c/lib
/home/dnachun/.linuxbrew/opt/libxml2/lib
/home/dnachun/.linuxbrew/opt/binutils/lib
/home/dnachun/.linuxbrew/opt/elfutils/lib
/home/dnachun/.linuxbrew/opt/gmp/lib
/home/dnachun/.linuxbrew/opt/isl/lib
/home/dnachun/.linuxbrew/opt/mpfr/lib
/home/dnachun/.linuxbrew/opt/libmpc/lib
/home/dnachun/.linuxbrew/opt/zstd/lib
/home/dnachun/.linuxbrew/opt/gcc/lib
/home/dnachun/.linuxbrew/lib
/home/dnachun/.linuxbrew/opt/gcc/lib/gcc/11
$ORIGIN/../lib

As you can see, /home/dnachun/.linuxbrew/lib precedes /home/dnachun/.linuxbrew/opt/gcc/lib/gcc/11, which is wrong and means that the libstdc++ from GCC 5 is found before the one from GCC 11.

The emergency fix for this is just replace the symlinks in HOMEBREW_PREFIX/lib that point to GCC 5 with the ones pointing to GCC 11:

rm $(brew --prefix)/lib/libstdc++.so.6; ln -s $(brew --prefix gcc)/lib/gcc/11/libstdc++.so.6 $(brew --prefix)/lib
rm $(brew --prefix)/lib/libgcc_s.so.1; ln -s $(brew --prefix gcc)/lib/gcc/11/libgcc_s.so.1 $(brew --prefix)/lib

This work because libstdc++.so.6 and libgcc_s.so.1 are backwards compatible and will still work with bottles built with GCC 5 or any version of GCC older than 11.

[danielnachun]

I believe the solution is just to replace HOMEBREW_PREFIX/lib with HOMEBREW_PREFIX/opt/glibc/lib in

      # Add linker flags on Linux so that stage1 clang can find libstdc++ from brewed GCC and all of the other
      # brewed libraries that need to be linked.
      unless OS.mac?
        ldflags = %W[
          -L#{libstdcxx_dir}
          -Wl,-rpath,#{libstdcxx_dir}
          -L#{HOMEBREW_PREFIX}/lib
          -Wl,-rpath,#{HOMEBREW_PREFIX}/lib
        ]
        args << "-DCMAKE_EXE_LINKER_FLAGS=#{ldflags.join(" ")}"
        args << "-DCMAKE_SHARED_LINKER_FLAGS=#{ldflags.join(" ")}"
      end

I had used HOMEBREW_PREFIX/lib because I didn't know if the RPATHs were being added correctly for all the linked dependencies, but it seems they are so the only thing we need to add is the path to brewed glibc. This won't have any effect for users that are using brewed glibc as HOMEBREW_PREFIX/opt/glibc/lib won't exist and will be ignored, while also guaranteeing that brewed glibc can be found if it is installed.

I will try to open this PR tomorrow though given how long LLVM takes to build on Intel macOS it could take a bit longer to get the updated bottles published.

[root42]

Very nice. Will you post a note here once this has successfully built?

[danielnachun]

Yes, I'll post an update once the bottle artifact is built and I've been able to test the bottle locally in my CentOS 7 install.

[danielnachun]

This should now be fixed by Homebrew/homebrew-core#105092.