ubuntu-latest runners have an incompatible combination of clang and libstdc++

[wheremyfoodat]

Description

Some ubuntu-latest runners come with a newer version of libstdc++ and still have clang 14 which only has partial consteval support. As a result, trying to compile a file that does #include <chrono> fails like this:

In file included from /home/runner/work/Panda3DS/Panda3DS/src/config.cpp:8:
In file included from /home/runner/work/Panda3DS/Panda3DS/third_party/toml11/toml.hpp:32:
In file included from /home/runner/work/Panda3DS/Panda3DS/third_party/toml11/toml/parser.hpp:9:
In file included from /home/runner/work/Panda3DS/Panda3DS/third_party/toml11/toml/combinator.hpp:16:
In file included from /home/runner/work/Panda3DS/Panda3DS/third_party/toml11/toml/result.hpp:5:
In file included from /home/runner/work/Panda3DS/Panda3DS/third_party/toml11/toml/traits.hpp:10:
/usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/chrono:2320:48: error: call to consteval function 'std::chrono::hh_mm_ss::_S_fractional_width' is not a constant expression
        static constexpr unsigned fractional_width = {_S_fractional_width()};
                                                      ^
/usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/chrono:2320:48: note: undefined function '_S_fractional_width' cannot be used in a constant expression
/usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/chrono:2275:2: note: declared here
        _S_fractional_width()
        ^
1 error generated.
gmake[2]: *** [CMakeFiles/Alber.dir/build.make:104: CMakeFiles/Alber.dir/src/config.cpp.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:588: CMakeFiles/Alber.dir/all] Error 2
gmake: *** [Makefile:136: all] Error 2

Should be fixable by using clang 17.

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• macOS 11
• macOS 12
• macOS 13
• Windows Server 2019
• Windows Server 2022

Image version and build link

2.310.2

Link to a failing build: https://github.com/wheremyfoodat/Panda3DS/actions/runs/6640518979/job/18041094730
Ignore the "Setting up newer Clang" step in this run, the compilation step was still using the stock ubuntu-latest clang

Is it regression?

Yes (worked fine until yesterday: https://github.com/wheremyfoodat/Panda3DS/actions/runs/6614729350/job/17965319784)

Expected behavior

Runner clang should probably be able to compile this

Actual behavior

Some (not all) ubuntu-latest runners fail to compile C++ files that do #include <chrono>

Repro steps

1. Have any C++ code that does #include <chrono>, such as the Panda3DS repo above (without updating the compiler like it does now)
2. Set up some ubuntu-latest actions for it.
3. Specific runners should fail with the error message listed in the "Description" field

[saran-t]

Also being hit by this: https://github.com/google-deepmind/mujoco_mpc/actions/runs/6644237828/job/18052982016

I think this is only affecting C++20 builds.

[wheremyfoodat]

Oh yeah, it does only affect C++20 builds, the STL headers shouldn't touch consteval and friends if building for an earlier standard version.

[Ozaq]

Same issue for us :/ https://github.com/PedestrianDynamics/jupedsim/actions/runs/6645042045/job/18055479382

I noticed that after the image upgrade clang-15 is available but not the default clang, is this related?

Also gradual rollout fine and dandy but how am I supposed to select a newly added clang version if it is random if I get an old or new image? Should I just apt install to ensure that there is a clang-15? And why are you doing the same for the ubuntu-22.04 image? Now I have no save version to fall back on...

[wheremyfoodat]

Same issue for us :/ https://github.com/PedestrianDynamics/jupedsim/actions/runs/6645042045/job/18055479382

I noticed that after the image upgrade clang-15 is available but not the default clang, is this related?

Also gradual rollout fine and dandy but how am I supposed to select a newly added clang version if it is random if I get an old or new image? Should I just apt install to ensure that there is a clang-15? And why are you doing the same for the ubuntu-22.04 image? Now I have no save version to fall back on...

You can use the LLVM Ubuntu repository as a workaround until this is fixed. You can make it install clang-17 like here: https://github.com/wheremyfoodat/Panda3DS/blob/master/.github/workflows/Linux_Build.yml

Note that the clang command will still invoke the default clang 14, you need to use clang-17 everywhere as well.

[ilia-shipitsin]

thank you for the report. we'll have a look

[vkottler]

Maybe an anecdote, but one solution I tried (that didn't work, but seems like it should) was:

Having clang-15 / clang++-15 use --gcc-toolchain=$(dirname $(gcc-12 -print-libgcc-file-name)), but even after that + adding a number of extra hard-coded -I / -L arguments, I couldn't get figure out /usr/bin/ld: cannot find crtbeginS.o: No such file or directory (during final linking, when all other includes and library arguments were located fine and crtbeginS.o and other startup objects are in the "toolchain"
directory).

Setting LD_LIBRARY_PATH to also point at the toolchain directory didn't seem to do anything (nor LIBRARY_PATH).

It seems that --gcc-toolchain= doesn't work out of the box with these Debian toolchain builds or something.

Anyway, since GCC 12 + 13 and clang 15 can currently be installed, pairing GCC 12 with clang 15 seems like an escape hatch but I wasn't personally able to get it working.

edit: this comment's suggestion to use clang 17 seems promising as an alternative

[jviotti]

Same issue here on C++20: https://github.com/sourcemeta/jsontoolkit/actions/runs/6646548576/job/18060229938. Interestingly enough, the problem is not always reproducible. Retrying the same job a couple times makes it go away.

[wheremyfoodat]

I'm not sure clang 15 can do much to fix this, Clang 17 is the one rated as having full consteval support

[ilia-shipitsin]

I'm not sure clang 15 can do much to fix this, Clang 17 is the one rated as having full consteval support

questions from our side

1. can clang 17 be installed from some kind of official repo ? (ubuntu preferrable)
2. are you already doing that in your own workflows ?
3. can/should we switch to clang 17 ?

[wheremyfoodat]

I'm not sure clang 15 can do much to fix this, Clang 17 is the one rated as having full consteval support

questions from our side

1. can clang 17 be installed from some kind of official repo ? (ubuntu preferrable)
2. are you already doing that in your own workflows ?
3. can/should we switch to clang 17 ?

1. Yes, from the official LLVM Ubuntu repository covering Ubuntu versions as early as 18.04. You can find a workflow using this and other details here: ubuntu-latest runners have an incompatible combination of clang and libstdc++ #8659 (comment)
2. Yep, see previously mentioned workflow (among others)
3. That's very subjective but I'd say yes honestly. It's a decently recent version of Clang with good C++20 support while being stable and tested instead of being bleeding edge. 14 is 4 majors behind by now.

[ilia-shipitsin]

I'm not sure clang 15 can do much to fix this, Clang 17 is the one rated as having full consteval support

questions from our side

1. can clang 17 be installed from some kind of official repo ? (ubuntu preferrable)
2. are you already doing that in your own workflows ?
3. can/should we switch to clang 17 ?

1. Yes, from the official LLVM Ubuntu repository covering Ubuntu versions as early as 18.04. You can find a workflow using this and other details here: ubuntu-latest runners have an incompatible combination of clang and libstdc++ #8659 (comment)
2. Yep, see previously mentioned workflow (among others)
3. That's very subjective but I'd say yes honestly. It's a decently recent version of Clang with good C++20 support while being stable and tested instead of being bleeding edge. 14 is 4 majors behind by now.

we definitely need someone with deep understanding of clang/llvm ecosystem to make a proper decision.

I've a question whether clang/llvm offers docker images (or maybe docker images from recognizable maintainer) ?

for pet project I use Fedora Rawhide image (to pick the latest clang) https://github.com/haproxy/haproxy/blob/master/.github/workflows/fedora-rawhide.yml#L14-L18

it would be really nice if clang would have been shipped like gcc: https://hub.docker.com/_/gcc

do you know are there similar images for clang/llvm ?

[wheremyfoodat]

I'm not sure clang 15 can do much to fix this, Clang 17 is the one rated as having full consteval support

questions from our side

1. can clang 17 be installed from some kind of official repo ? (ubuntu preferrable)
2. are you already doing that in your own workflows ?
3. can/should we switch to clang 17 ?

1. Yes, from the official LLVM Ubuntu repository covering Ubuntu versions as early as 18.04. You can find a workflow using this and other details here: ubuntu-latest runners have an incompatible combination of clang and libstdc++ #8659 (comment)
2. Yep, see previously mentioned workflow (among others)
3. That's very subjective but I'd say yes honestly. It's a decently recent version of Clang with good C++20 support while being stable and tested instead of being bleeding edge. 14 is 4 majors behind by now.

we definitely need someone with deep understanding of clang/llvm ecosystem to make a proper decision.

I've a question whether clang/llvm offers docker images (or maybe docker images from recognizable maintainer) ?

for pet project I use Fedora Rawhide image (to pick the latest clang) https://github.com/haproxy/haproxy/blob/master/.github/workflows/fedora-rawhide.yml#L14-L18

it would be really nice if clang would have been shipped like gcc: https://hub.docker.com/_/gcc

do you know are there similar images for clang/llvm ?

To my knowledge there's no official docker images, there's apt.llvm.org intended for providing various llvm versions via apt and then there's unofficial docker images like https://hub.docker.com/r/silkeh/clang

I have not used the latter so I can't vouch for it.

[ilia-shipitsin]

looks pretty popular

[randombit]

can/should we switch to clang 17 ?

For my usage that isn't helpful since one of the reasons we CI with Clang 14 is because that is the minimum Clang version that we support - largely because it is the default version of Clang included in 22.04. If we aren't actively testing with our minimum version we're very likely to introduce regressions, where something works with whatever Clang is in CI but not older versions.

[stenzek]

As far as I could tell, GCC 13/libstdc++ 13 getting pulled in was the result of the https://launchpad.net/~ubuntu-toolchain-r/+archive/ubuntu/test PPA getting added to the runner image. I don't know the sequence that the images are built in, but I'd guess that after the PPA was added, a general upgrade was performed, or installing something else resulted in an upgrade to gcc/libstdc++ and friends.

We've worked around this by removing the PPA, and reinstalling libstdc++, since we build AppImages from the workflows, and those were no longer compatible with Ubuntu 22.04. I'm not sure on the implications of doing so, though - perhaps the newer version was added for good reason, so if you want to do this, do so at your own risk. But it works well enough for us - see the linked PR above.

[Ozaq]

Hey GitHub Folks how come you don't rollback this change? Its ok for mistakes to happen but at this point you should probably roll back and test this internally first...

[SergiusTheBest]

I'm using https://github.com/mjp41/workaround8649 to temporary workaround the issue. You need to add the following step:

    # workaround for broken clang on ubuntu runner until https://github.com/actions/runner-images/issues/8659 get fixed
    - uses: mjp41/workaround8649@7929373c0fe5caf844d8115adccef39e3b5362e7
      with:
        os: ${{ matrix.os }}

[eladm-ultrawis]

@SergiusTheBest This is great thank you.

[KitsuneRal]

@ilia-shipitsin it's soon to be 2 months without a working Clang pipeline, and GitHub apparently just pushed a change breaking the workarounds without fixing the issue - as far as I just checked, at least. Should I move to another CI provider that has more attention for C++?

[BurningEnlightenment]

You can use the LLVM Ubuntu repository as a workaround until this is fixed. You can make it install clang-17 like here: https://github.com/wheremyfoodat/Panda3DS/blob/master/.github/workflows/Linux_Build.yml

Running scripts from a web location without any form of signature verification or hash pinning is an… interesting choice from a security perspective. I recommend adding the LLVM PubKey to your repository / CI and consuming their package feed directly:

sudo cp tools/llvm-snapshot.gpg.key.asc /etc/apt/trusted.gpg.d/llvm-snapshot.asc
sudo add-apt-repository "deb http://apt.llvm.org/jammy/ llvm-toolchain-jammy-16 main"
sudo apt-get -qq update
sudo apt-get -qq install clang-16

[stenzek]

If you're using the workaround from #8659 (comment), or a variant thereof, the version suffix of the libc6{-dev} package got bumped from 3.4 to 3.5 - changing that should allow it to go through again.

Shame this still isn't resolved - IMO the simplest solution is to not include gcc 13 from the testing repo/PPA, as far as I can tell, it's not a supported configuration for 22.04 anyway.

[mjp41]

FYI, I have packaged up the action from above which can be called like:

    - uses: mjp41/workaround8649@c8550b715ccdc17f89c8d5c28d7a48eeff9c94a8
      with:
        os: ${{ matrix.os }}

e.g. https://github.com/microsoft/snmalloc/blob/640cacf90ee83697d082c8fc758f0ea762e5e27a/.github/workflows/main.yml#L99-L101

where os is the platform, and applies the changes if it is not provided or set to ubuntu-latest or ubuntu-22.04. I also used a wildcard in the version, so hopefully should be resilient to other changes.

[Ozaq]

Microsoft / Github seems to not care. Bad optics.

[wheremyfoodat]

The real treasure is the friends we made along the way while waiting 3 months for a fix

[jviotti]

The real treasure is the friends we made along the way while waiting 3 months for a fix

Haha happy C++ coding everybody! We'll stand together even in the face of broken CI 😅 GitHub is us after all.

[Ozaq]

Why has this been turned into a discussion? Is there any debate about if this really is an issue?

[santiago0411]

Still waiting...

[ruifig]

Let's have a minute of silence for all the CI green check marks.

[Ozaq]

I still believe this IS an issue that needs to be addressed: #9641

[ruifig]

Looks like.
I had a workaround that stopped working eventually.
Just minutes ago I cleaned up my workflow file thinking that maybe defaults actually work again, but it's not the case. It's still broken. :(

[lkfortuna]

👋 Hey all, I am a PM on the Actions team. We are addressing this issue with this change: #9679

In accordance with our security policies, we are working to minimize the amount of third party repositories we install from. In response to this issue we will be removing the third party repo we are installing gcc 13 from, which will change the default gcc version installed on the Ubuntu 22 image to gcc 12.

[KitsuneRal]

That's nice @lkfortuna but what is the action to prevent such things from happening in the first place? Some tests to introduce for GitHub runners, to check sanity of the toolchain, maybe?