ubuntu-latest runners have an incompatible combination of clang and libstdc++

[wheremyfoodat]

Description

Some ubuntu-latest runners come with a newer version of libstdc++ and still have clang 14 which only has partial consteval support. As a result, trying to compile a file that does #include <chrono> fails like this:

In file included from /home/runner/work/Panda3DS/Panda3DS/src/config.cpp:8:
In file included from /home/runner/work/Panda3DS/Panda3DS/third_party/toml11/toml.hpp:32:
In file included from /home/runner/work/Panda3DS/Panda3DS/third_party/toml11/toml/parser.hpp:9:
In file included from /home/runner/work/Panda3DS/Panda3DS/third_party/toml11/toml/combinator.hpp:16:
In file included from /home/runner/work/Panda3DS/Panda3DS/third_party/toml11/toml/result.hpp:5:
In file included from /home/runner/work/Panda3DS/Panda3DS/third_party/toml11/toml/traits.hpp:10:
/usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/chrono:2320:48: error: call to consteval function 'std::chrono::hh_mm_ss::_S_fractional_width' is not a constant expression
        static constexpr unsigned fractional_width = {_S_fractional_width()};
                                                      ^
/usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/chrono:2320:48: note: undefined function '_S_fractional_width' cannot be used in a constant expression
/usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/chrono:2275:2: note: declared here
        _S_fractional_width()
        ^
1 error generated.
gmake[2]: *** [CMakeFiles/Alber.dir/build.make:104: CMakeFiles/Alber.dir/src/config.cpp.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:588: CMakeFiles/Alber.dir/all] Error 2
gmake: *** [Makefile:136: all] Error 2

Should be fixable by using clang 17.

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• macOS 11
• macOS 12
• macOS 13
• Windows Server 2019
• Windows Server 2022

Image version and build link

2.310.2

Link to a failing build: https://github.com/wheremyfoodat/Panda3DS/actions/runs/6640518979/job/18041094730
Ignore the "Setting up newer Clang" step in this run, the compilation step was still using the stock ubuntu-latest clang

Is it regression?

Yes (worked fine until yesterday: https://github.com/wheremyfoodat/Panda3DS/actions/runs/6614729350/job/17965319784)

Expected behavior

Runner clang should probably be able to compile this

Actual behavior

Some (not all) ubuntu-latest runners fail to compile C++ files that do #include <chrono>

Repro steps

1. Have any C++ code that does #include <chrono>, such as the Panda3DS repo above (without updating the compiler like it does now)
2. Set up some ubuntu-latest actions for it.
3. Specific runners should fail with the error message listed in the "Description" field

[saran-t]

Also being hit by this: https://github.com/google-deepmind/mujoco_mpc/actions/runs/6644237828/job/18052982016

I think this is only affecting C++20 builds.

[wheremyfoodat]

Oh yeah, it does only affect C++20 builds, the STL headers shouldn't touch consteval and friends if building for an earlier standard version.

[Ozaq]

Same issue for us :/ https://github.com/PedestrianDynamics/jupedsim/actions/runs/6645042045/job/18055479382

I noticed that after the image upgrade clang-15 is available but not the default clang, is this related?

Also gradual rollout fine and dandy but how am I supposed to select a newly added clang version if it is random if I get an old or new image? Should I just apt install to ensure that there is a clang-15? And why are you doing the same for the ubuntu-22.04 image? Now I have no save version to fall back on...

[wheremyfoodat]

Same issue for us :/ https://github.com/PedestrianDynamics/jupedsim/actions/runs/6645042045/job/18055479382

I noticed that after the image upgrade clang-15 is available but not the default clang, is this related?

Also gradual rollout fine and dandy but how am I supposed to select a newly added clang version if it is random if I get an old or new image? Should I just apt install to ensure that there is a clang-15? And why are you doing the same for the ubuntu-22.04 image? Now I have no save version to fall back on...

You can use the LLVM Ubuntu repository as a workaround until this is fixed. You can make it install clang-17 like here: https://github.com/wheremyfoodat/Panda3DS/blob/master/.github/workflows/Linux_Build.yml

Note that the clang command will still invoke the default clang 14, you need to use clang-17 everywhere as well.

[ilia-shipitsin]

thank you for the report. we'll have a look

[vkottler]

Maybe an anecdote, but one solution I tried (that didn't work, but seems like it should) was:

Having clang-15 / clang++-15 use --gcc-toolchain=$(dirname $(gcc-12 -print-libgcc-file-name)), but even after that + adding a number of extra hard-coded -I / -L arguments, I couldn't get figure out /usr/bin/ld: cannot find crtbeginS.o: No such file or directory (during final linking, when all other includes and library arguments were located fine and crtbeginS.o and other startup objects are in the "toolchain"
directory).

Setting LD_LIBRARY_PATH to also point at the toolchain directory didn't seem to do anything (nor LIBRARY_PATH).

It seems that --gcc-toolchain= doesn't work out of the box with these Debian toolchain builds or something.

Anyway, since GCC 12 + 13 and clang 15 can currently be installed, pairing GCC 12 with clang 15 seems like an escape hatch but I wasn't personally able to get it working.

edit: this comment's suggestion to use clang 17 seems promising as an alternative

[jviotti]

Same issue here on C++20: https://github.com/sourcemeta/jsontoolkit/actions/runs/6646548576/job/18060229938. Interestingly enough, the problem is not always reproducible. Retrying the same job a couple times makes it go away.

[Ozaq]

Microsoft / Github seems to not care. Bad optics.

[wheremyfoodat]

The real treasure is the friends we made along the way while waiting 3 months for a fix

[jviotti]

The real treasure is the friends we made along the way while waiting 3 months for a fix

Haha happy C++ coding everybody! We'll stand together even in the face of broken CI 😅 GitHub is us after all.