Docker incompatibility with glibc-2.34 #4193

lslezak · 2021-09-30T13:19:16Z

Description

The Problem

Using glibc-2.34 in GitHub Actions Docker images results in strange "permission error" failures in the jobs.

This new glibc has been introduced in openSUSE Tumbleweed recently (see the announcement). That means all Docker images based on the latest openSUSE Tumbleweed might fail with some strange errors.

This is quite similar to a recent problem with the faccessat2 syscall used in glibc-2.33 (link).

The problem is already fixed in the upstream Docker (moby/moby#42681) and there is a backport in progress (moby/moby#42836). This patch should be applied in the Docker included in the GitHub Action virtual environment.

Temporary Workaround

I found out that using --privileged Docker option makes the GitHub Action work correctly,
e.g. put this into the GitHub Action file:

container:
  image: foo:bar
  options: --privileged

Virtual environments affected

Image version and build link

https://github.com/lslezak/github_action_test/actions/runs/1291119026

Is it regression?

Yes, the Docker images with older glibc-2.33 work just fine.

Expected behavior

No crashes in Docker.

Actual behavior

Strange crashes when "fork" syscall is used.

Repro steps

See a complete GitHub Action example in https://github.com/lslezak/github_action_test/blob/main/.github/workflows/ci.yml.

That contains two failing jobs (one failing in actions/checkout@v2 action, the other one in the zypper install command) and there is a working job which uses the --privileged workaround). See the build results.

The text was updated successfully, but these errors were encountered:

miketimofeev · 2021-09-30T17:20:58Z

Hi @lslezak! Do you happen to know the exact docker version where the issue is fixed? I assume it will be 20.10.9 so all we can do here is to wait until the version is released

lslezak · 2021-10-01T10:15:09Z

I'm not involved in the Docker development, but that pull request is included in the 20.10.9 milestone so it should be fixed in that version.

mafalb · 2021-10-01T10:40:45Z

there is also an issue about fedora 35 #3812 - which is closed - but fedora 35 containers are still not working.

moby is used, not docker?
#3812 (comment)

catthehacker · 2021-10-01T10:58:33Z

moby is used, not docker?

Docker is just moby but released under different licence by Docker, Inc.

It adds the "--privileged" Docker option as a workaround for actions/runner-images#4193

catthehacker · 2021-10-04T20:02:06Z

It seems that fix will land in 20.10.10, see moby/moby#42836 (comment)

fedoradev disabled for now due to glibc vs docker clone3 issues on setup: Curl error (6): Couldn't resolve host name for https://mirrors.fedoraproject.org/metalink?repo=... [getaddrinfo() thread failed to start] actions/runner-images#4193

With recent glibc2.34 in ALTLinux CI fails: ``` /usr/bin/docker exec acdbb8c61a25424e068e81ffe7e18a00148d99a52833f8965dce034316c3d433 sh -c "cat /etc/*release | grep ^ID" 3 /__e/node12/bin/node[232]: ../src/node_platform.cc:63:std::unique_ptr<long unsigned int> node::WorkerThreadsTaskRunner::DelayedTaskScheduler::Start(): Assertion `(0) == (uv_thread_create(t.get(), start_thread, this))' failed. 4 1: 0x9da7c0 node::Abort() [/__e/node12/bin/node] 5 2: 0x9da847 [/__e/node12/bin/node] 6 3: 0xa49c8a node::WorkerThreadsTaskRunner::WorkerThreadsTaskRunner(int) [/__e/node12/bin/node] 7 4: 0xa49d4b node::NodePlatform::NodePlatform(int, node::tracing::TracingController*) [/__e/node12/bin/node] 8 5: 0x94e3da node::InitializeV8Platform(int) [/__e/node12/bin/node] 9 6: 0x9ac4c1 node::InitializeOncePerProcess(int, char**) [/__e/node12/bin/node] 10 7: 0x9ac6b3 node::Start(int, char**) [/__e/node12/bin/node] 11 8: 0x7f0fcd7b81dc [/lib64/libc.so.6] 12 9: 0x7f0fcd7b8288 __libc_start_main [/lib64/libc.so.6] 13 10: 0x94a055 [/__e/node12/bin/node] ``` actions/runner-images#4193 Related: #12

With recent glibc2.34 in ALTLinux CI fails: ``` /usr/bin/docker exec acdbb8c61a25424e068e81ffe7e18a00148d99a52833f8965dce034316c3d433 sh -c "cat /etc/*release | grep ^ID" 3 /__e/node12/bin/node[232]: ../src/node_platform.cc:63:std::unique_ptr<long unsigned int> node::WorkerThreadsTaskRunner::DelayedTaskScheduler::Start(): Assertion `(0) == (uv_thread_create(t.get(), start_thread, this))' failed. 4 1: 0x9da7c0 node::Abort() [/__e/node12/bin/node] 5 2: 0x9da847 [/__e/node12/bin/node] 6 3: 0xa49c8a node::WorkerThreadsTaskRunner::WorkerThreadsTaskRunner(int) [/__e/node12/bin/node] 7 4: 0xa49d4b node::NodePlatform::NodePlatform(int, node::tracing::TracingController*) [/__e/node12/bin/node] 8 5: 0x94e3da node::InitializeV8Platform(int) [/__e/node12/bin/node] 9 6: 0x9ac4c1 node::InitializeOncePerProcess(int, char**) [/__e/node12/bin/node] 10 7: 0x9ac6b3 node::Start(int, char**) [/__e/node12/bin/node] 11 8: 0x7f0fcd7b81dc [/lib64/libc.so.6] 12 9: 0x7f0fcd7b8288 __libc_start_main [/lib64/libc.so.6] 13 10: 0x94a055 [/__e/node12/bin/node] ``` actions/runner-images#4193 It is always safer to run unprivileged container but in this case it doesn't matter and can be easily rolled back on release of Docker 20.10.10. Related: #12 Signed-off-by: Stanislav Levin <slev@altlinux.org>

catthehacker · 2021-10-25T17:11:22Z

https://github.com/moby/moby/releases/tag/v20.10.10

Add support for clone3 syscall in the default seccomp policy to support running
containers based on recent versions of Fedora and Ubuntu.

moby-engine:
  Installed: (none)
  Candidate: 20.10.10+azure-1
  Version table:
     20.10.10+azure-1 500
        500 https://packages.microsoft.com/ubuntu/20.04/prod focal/main amd64 Packages
     20.10.9+azure-1 500
        500 https://packages.microsoft.com/ubuntu/20.04/prod focal/main amd64 Packages
     20.10.8+azure-3 500
        500 https://packages.microsoft.com/ubuntu/20.04/prod focal/main amd64 Packages

086de3c had to disable Tumbleweed due glibc-2.34 using new syscall clone3 which is not enabled in Docker seccomp filter [1]. Workaround is to disable Docker seccomp filtering, as we don't need this protection. This should help to avoid seccomp filtering in the future for all distros. [1] https://bugzilla.opensuse.org/show_bug.cgi?id=1190670 Fixes: actions/runner-images#4193 Signed-off-by: Petr Vorel <pvorel@suse.cz>

50b3bae had to disable Tumbleweed due glibc-2.34 using new syscall clone3 which is not enabled in Docker seccomp filter [1]. Workaround is to disable Docker seccomp filtering, as we don't need this protection. This should help to avoid seccomp filtering in the future for all distros. [1] https://bugzilla.opensuse.org/show_bug.cgi?id=1190670 Fixes: actions/runner-images#4193 Signed-off-by: Petr Vorel <pvorel@suse.cz> Acked-by: Cyril Hrubis <chrubis@suse.cz>

MehdiChinoune · 2021-11-01T20:45:30Z

An updated ubuntu-20.04 with a new docker-moby is rolling #4395

Docker-Moby Client 20.10.10+azure-1
Docker-Moby Server 20.10.10+azure-1

miketimofeev · 2021-11-03T12:56:51Z

@lslezak @MehdiChinoune the images with docker moby 20.10.10 have been deployed!
I'm going to close this issue, please feel free to contact us if you have any concerns.
Thank you!

lslezak · 2021-11-03T15:09:21Z

I have tested running a container without that --privileged workaround and it works fine with the latest image! Thank you! 👍

actions/runner-images#4193 reportedly fixed.

lslezak added the needs triage label Sep 30, 2021

al-cheb added Area: Containers investigate Collect additional information, like space on disk, other tool incompatibilities etc. OS: Ubuntu and removed needs triage labels Sep 30, 2021

lslezak added a commit to yast/helper_scripts that referenced this issue Oct 1, 2021

Added a script for modifying GitHub Actions globally

7218889

It adds the "--privileged" Docker option as a workaround for actions/runner-images#4193

miketimofeev self-assigned this Oct 4, 2021

miketimofeev added external bug Something isn't working and removed investigate Collect additional information, like space on disk, other tool incompatibilities etc. labels Oct 4, 2021

stanislavlevin mentioned this issue Oct 18, 2021

GH actions fails with containerized glibc2.34 stanislavlevin/tox-console-scripts#12

Closed

stanislavlevin mentioned this issue Oct 18, 2021

github actions: Run as privileged stanislavlevin/tox-console-scripts#13

Merged

catthehacker mentioned this issue Oct 18, 2021

glibc 2.34 updates break Container Images #4297

Closed

8 tasks

miketimofeev closed this as completed Nov 3, 2021

scop added a commit to scop/bash-completion that referenced this issue Nov 3, 2021

ci(images): re-enable fedoradev build

472e0e3

actions/runner-images#4193 reportedly fixed.

blitz mentioned this issue Apr 29, 2022

[21.11] docker: 2.10.9 -> 20.10.14 (for glibc 2.34 compatiblity!) NixOS/nixpkgs#170900

Merged

13 tasks

nathanweeks mentioned this issue Aug 3, 2023

Use full image for builder stage; update to bullseye wurmlab/sequenceserver#675

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docker incompatibility with glibc-2.34 #4193

Docker incompatibility with glibc-2.34 #4193

lslezak commented Sep 30, 2021

miketimofeev commented Sep 30, 2021

lslezak commented Oct 1, 2021

mafalb commented Oct 1, 2021

catthehacker commented Oct 1, 2021

catthehacker commented Oct 4, 2021 •

edited

catthehacker commented Oct 25, 2021 •

edited

MehdiChinoune commented Nov 1, 2021

miketimofeev commented Nov 3, 2021

lslezak commented Nov 3, 2021

Docker incompatibility with glibc-2.34 #4193

Docker incompatibility with glibc-2.34 #4193

Comments

lslezak commented Sep 30, 2021

Description

The Problem

Temporary Workaround

Virtual environments affected

Image version and build link

Is it regression?

Expected behavior

Actual behavior

Repro steps

miketimofeev commented Sep 30, 2021

lslezak commented Oct 1, 2021

mafalb commented Oct 1, 2021

catthehacker commented Oct 1, 2021

catthehacker commented Oct 4, 2021 • edited

catthehacker commented Oct 25, 2021 • edited

MehdiChinoune commented Nov 1, 2021

miketimofeev commented Nov 3, 2021

lslezak commented Nov 3, 2021

catthehacker commented Oct 4, 2021 •

edited

catthehacker commented Oct 25, 2021 •

edited