tonistiigi/binfmt:latest (qemu 9.2.0) segfault on ubuntu-24.04 / ubuntu-latest

[mayeut]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug, and:

• The documentation does not mention anything about my problem
• There are no open or closed issues that are related to my problem

Description

As mentioned in #188, the default image is unusable on ubuntu-24.04 since a kernel update a few weeks back.

As reported in that issue, moving to the qemu 8.1.5 image solved the issue for most users (since then, there has been a 9.2.0 / master image published).

It seems more & more unlikely that the latest image will be updated and more likely that setup-qemu-action should change the default image (possibly depending on the runner) to an image that works in most cases rather than what I imagine is almost none => hence this is a bug report setup-qemu-action can do something about without closing saying it's an upstream issue.

Multiple reproducers/logs/... are already available in that issue so not added new ones here.

Expected behaviour

No random segmentation faults.

Actual behaviour

Random segmentation faults in all the repositories I know of when the action is used with the default image on ubuntu-24.04 (latest)

Repository URL

No response

Workflow run URL

No response

YAML workflow

N/C

Workflow logs

No response

BuildKit logs

Additional info

No response

[kxc171]

This is impacting us as well, as of this morning most of our builds that use linux/arm64 result in a segmentation error. Pinning to tonistiigi/binfmt:qemu-v7.0.0-28 seems to have resolved the issue.

There is also an issue opened on tonistiigi/binfmt, tonistiigi/binfmt#240

[dylanTesouro]

This is also effecting us as well.

[goshander]

Same issue for linux/arm64

[crazy-max]

This is probably related to a kernel issue on Ubuntu: tonistiigi/binfmt#215 (comment)

[aeggerm]

A workaround that we found is to run our docker builds on ARM based VMs with Qemu emulating the AMD64.

Minimal modification was needed for our use case (CI runners).
Hope this will help someone :)

[giovaborgogno]

This is impacting us as well, as of this morning most of our builds that use linux/arm64 result in a segmentation error. Pinning to tonistiigi/binfmt:qemu-v7.0.0-28 seems to have resolved the issue.

There is also an issue opened on tonistiigi/binfmt, tonistiigi/binfmt#240

This worked for me, thanks

[bluecamel]

Here's a minimalish repro based on our build that was failing. Maybe worth noting is that our runner is ubicloud. Also, the same thing happens on the latest version of this action (even though the example below is v2) as well as the buildx action.

Dockerfile:

FROM ros:iron-ros-base-jammy

RUN apt-get update && \
    apt-get install -y \
        default-jdk \
        ros-iron-rmw-cyclonedds-cpp \
        libasio-dev \
        python3-pip \
        dumb-init && \
    rm -rf /var/lib/apt/lists/* && \
    apt-get clean

RUN apt-get update && \
    apt-get install -y \
        ros-iron-cv-bridge \
        ros-iron-image-transport \
        libusb-1.0-0-dev \
        libgstreamer1.0-dev \
        gstreamer1.0-tools \
        gstreamer1.0-plugins-bad \
        gstreamer1.0-plugins-ugly && \
    rm -rf /var/lib/apt/lists/*

Excerpt from our GitHub workflow:

jobs:
  docker_release:
    permissions:
      contents: read
      id-token: write
    runs-on: ubicloud
    steps:
      - name: Checkout project
        uses: actions/checkout@v3
        with:
          submodules: recursive
      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v2
        with:
          aws-region: ${{ inputs.AWS_REGION }}
          role-to-assume: arn:aws:iam::${{ inputs.AWS_ACCOUNT_ID }}:role/github-actions-${{ inputs.AWS_REGION }}-${{ github.repository_owner }}-${{ github.event.repository.name }}
      - name: Login to Amazon ECR
        id: login_ecr
        uses: aws-actions/amazon-ecr-login@v1
      - name: Setup QEMU
        uses: docker/setup-qemu-action@v2
      - name: Setup Docker Buildx
        uses: docker/setup-buildx-action@v2
      - name: Increment Version and Create Tag
        id: tag
        uses: mathieudutour/github-tag-action@v6.1
        with:
          github_token: ${{ secrets.CI_GITHUB_TOKEN }}
      - name: Get repo name with dashes instead of underscores
        id: get_repo_name_dashes
        run: |
          REPO_NAME_DASHES=${{ github.event.repository.name }}
          REPO_NAME_DASHES="${REPO_NAME_DASHES//_/-}"
          echo "REPO_NAME_DASHES=${REPO_NAME_DASHES}" >> ${GITHUB_OUTPUT}
      - name: Extract Docker Metadata (tags, labels)
        id: docker_metadata
        uses: docker/metadata-action@v4
        with:
          images: ${{ inputs.AWS_ACCOUNT_ID }}.dkr.ecr.${{ inputs.AWS_REGION }}.amazonaws.com/path/to/repo/${{ steps.get_repo_name_dashes.outputs.REPO_NAME_DASHES }}
          tags: |
            type=raw,value=latest,enable={{ is_default_branch }}
            type=semver,pattern={{version}},value=${{ steps.tag.outputs.new_tag }}
      - name: Build and Push Image
        id: container_image_build_push
        uses: docker/build-push-action@v4
        with:
          context: .
          file: docker/Dockerfile
          platforms: linux/amd64,linux/arm64/v8
          push: true
          tags: ${{ steps.docker_metadata.outputs.tags }}
          labels: ${{ steps.docker_metadata.outputs.labels }}

[tejashah88]

A workaround that we found is to run our docker builds on ARM based VMs with Qemu emulating the AMD64.

Minimal modification was needed for our use case (CI runners). Hope this will help someone :)

I mentioned it in another issue (here) but I found another workaround as well. If you set up your Dockerfile as a multi-stage build with multiarch/qemu-user-static as a base and inject the QEMU binaries into your target image, no segfaults will generate. Here's an example:

# Use multiarch/qemu-user-static only for non-native architectures. Ending tag is excluded due to specifying platform
# See https://docs.docker.com/reference/dockerfile/#automatic-platform-args-in-the-global-scope for more info
FROM --platform=$BUILDPLATFORM multiarch/qemu-user-static AS qemu

# Start with base image
FROM luxonis/depthai-library:latest

# Copy QEMU binary only when cross-compiling with ARM and ARM64
COPY --from=qemu /usr/bin/qemu-*-static /usr/bin/

# More build statements...

This was tested on Windows 10 Pro x64 host machine while building docker images to be used on Raspberry Pi 3s (linux/arm/v7) and 4s (linux/arm64/v8). To be clear, only the linux/arm64/v8 builds were failing.

I'd like to know why this works, as this seems too simple for my liking but I'm glad this is a decent workaround for now.

[shink]

Same issue

[DoingItNow]

At first it was intermitted failures now I consistently with fail every time when attempting to docker buildx

[booleanbetrayal]

We are mysteriously encountering this issue on ARM64 self-hosted GitHub Actions (ghcr.io/actions/actions-runner:2.322.0) based runners, beginning sometime over the past 72 hours, targeting AMD64. None of the workarounds have worked for us. Anyone else in the same boat?

[tair-itzhak]

We are mysteriously encountering this issue on ARM64 self-hosted GitHub Actions (ghcr.io/actions/actions-runner:2.322.0) based runners, beginning sometime over the past 72 hours, targeting AMD64. None of the workarounds have worked for us. Anyone else in the same boat?

We experience the same issue with actions-runners

[dyrnq]

Same issue for linux/arm64

[booleanbetrayal]

We are mysteriously encountering this issue on ARM64 self-hosted GitHub Actions (ghcr.io/actions/actions-runner:2.322.0) based runners, beginning sometime over the past 72 hours, targeting AMD64. None of the workarounds have worked for us. Anyone else in the same boat?

Specifically, this is happening during an addgroup RUN call in our Dockerfile.

[alphonsekoh]

For me, as a workaround I downgraded it to ubuntu-22.04 however the build time takes a huge impact; when building my docker image is 4x slower. Used to take around 10 mins when building using linux/arm64, now it take 40 mins.

[crazy-max]

https://github.com/tonistiigi/binfmt/releases/tag/deploy%2Fv9.2.2-52 (QEMU 9.2.2) has been released yesterday: tonistiigi/binfmt#215 (comment)

If you encounter something similar, open an issue on https://github.com/tonistiigi/binfmt/issues