Clang-15 fails to select proper candidate GCC installation on CentOS7 with multiple devtoolset versions

[sfc-gh-mheimel]

This is a (somewhat obscure) issue that was introduced by a recent driver change to the GCC candidate selection code for RHEL devtoolset installations:

• Before this change, the driver would enumerate a list of hardcoded /opt/rh/devtoolset-XXX path prefixes for the various devtoolset versions, from which the driver selects the one that provides the GCC installation with the highest version number.
• After this change, the driver iterates over the /opt/rh folder and selects the devtoolset-XXX path with the highest number. Only this path is checked for a GCC candidate installation.

This change causes problems on systems that have multiple devtoolset versions installed where the latest version does not provide a GCC installation. For instance, consider a system that has a full devtoolset-10 installation as well as a partial (say, only binutils) devtoolset-11 installation. In this case, the clang-15 driver will fail to detect the devtoolset-10 GCC candidate and fall back to the ancient system GCC, while clang-14 will correctly pick up devtoolset-10.

This issue particularly affects ARM installations of CentOS7: For some reason, CentOS does not provide a devtoolset-11-gcc package for ARM. In our case, we therefore have to rely on using both devtoolset-11-binutils and devtoolset-10-gcc, which means the above bug currently prevents us from migrating to clang 15.

[llvmbot]

@llvm/issue-subscribers-clang-driver

[tbaederr]

This is pretty ugly; we can't just Prefixes.push_back() all the directories because the list we get might not be sorted, and the order of entries in Prefixes is important since the code later in GCCInstallationDetector::init() stops searching on the first valid directory. Any opinions on what to do there, @MaskRay?

[MaskRay]

(I don't have a CentOS/Rocky system at hand.) I think Clang Driver should stop detecting gcc-toolset-* devtoolset-* file hiearchy. Also, I am not sure whether changing clang to follow devtoolset-* is a good idea. Instead, CentOS/Rocky should do something like Gentoo #57570

There is ongoing effort improving clang --config. devtoolset-* can provide a config file for clang to use.

(I landed --gcc-install-dir today https://reviews.llvm.org/D133329 which may be of little help.)

[tbaederr]

Alright, I documented the config file task internally. What's your opinion for fixing this in the next clang 15 release? Should we just revert my patch and add the devtoolset-12/gcc-toolset-12 paths after all?

[tbaederr]

@MaskRay Ping about the question above

[tstellar]

What if we just backported the --gcc-install-dir change?

[MaskRay]

If clang16 stops doing the gcc/devtoolset detection and relies on a config file instead, that's fine but for the time being (and for a backport to clang15), we need to fix this problem.

Yeah. That will be a good direction forward.

On Debian, to use GCC 10, just run clang --gcc-install-dir=/usr/lib/gcc/x86_64-linux-gnu/10.
If CentOS wants to couple Clang with a specific devtoolset- , configure llvm-project with CLANG_CONFIG_FILE_SYSTEM_DIR and install a configuration file specifying an appropriate --gcc-install-dir=. See https://blogs.gentoo.org/mgorny/2022/10/07/clang-in-gentoo-now-sets-default-runtimes-via-config-file/

I do not know whether mismatched devtoolset-11-binutils and devtoolset-10-gcc should be supported.