Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

arm64 multiarch BuildKit compilation randomly segfaults #1309

Closed
3 tasks done
FacuM opened this issue Jan 26, 2025 · 6 comments
Closed
3 tasks done

arm64 multiarch BuildKit compilation randomly segfaults #1309

FacuM opened this issue Jan 26, 2025 · 6 comments

Comments

@FacuM
Copy link

FacuM commented Jan 26, 2025

Contributing guidelines

I've found a bug, and:

  • The documentation does not mention anything about my problem
  • There are no open or closed issues that are related to my problem

Description

Compilling my multiarch project started to randomly segfault on the action's QEMU binary while processing the Dockerfile steps.

As this is a public project, you can actually just take a look at the job output, but may also check the build log attached below. If you were to retry that job, it would fail on another random spot.

I don't know for how long this has been a thing since I haven't touched the image for about a month.

Expected behaviour

Getting the resulting image and publishing it to Docker Hub.

Actual behaviour

Random crashes and segfaults during image compilation for the target system, amd64 looks to be alright, but arm64 (which runs on top of QEMU) fails. I've already tried upgrading my dependencies, including both docker/setup-buildx-action and docker/build-push-action.

Repository URL

https://github.com/wprint3d/wprint3d

Workflow run URL

https://github.com/wprint3d/wprint3d/blob/alpha/.github/workflows/docker-image.yml

YAML workflow

name: Docker Image CI

on:
  push:
    branches: [ "alpha" ]
  pull_request:
    branches: [ "alpha" ]

concurrency:
  group: ${{ github.ref }}
  cancel-in-progress: false

jobs:

  build:
    runs-on: ubuntu-latest
    steps:
      -
        name: Checkout
        uses: actions/checkout@v3
      -
        name: Login to Docker Hub
        uses: docker/login-action@v2
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}
      -
        name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
      -
        name: Build and push
        uses: docker/build-push-action@v6
        with:
          context: .
          file: ./Dockerfile
          platforms: linux/amd64,linux/arm64
          push: true
          tags: ${{ secrets.DOCKERHUB_USERNAME }}/wprint3d:latest
          cache-from: type=gha
          cache-to: type=gha,mode=max

Workflow logs

logs_33538460101.zip

BuildKit logs

BuildKit logs are contained as part of the attached ZIP file and the link to the job.

Additional info

No response

@kahara
Copy link

kahara commented Jan 27, 2025

Observing the same thing here, I think:

https://github.com/kahara/vushf-exporter/actions/runs/12996988416/job/36247696081#step:10:401

On 2025-01-19, was able to run jobs to completion several times without issues. So perhaps things worked fine at least five seven days ago:

https://github.com/kahara/vushf-exporter/actions

Today there was first a single successful run but after that, each one has failed.

@FacuM
Copy link
Author

FacuM commented Jan 27, 2025

Observing the same thing here, I think:

https://github.com/kahara/vushf-exporter/actions/runs/12996988416/job/36247696081#step:10:401

On 2025-01-19, was able to run jobs to completion several times without issues. So perhaps things worked fine at least five seven days ago:

https://github.com/kahara/vushf-exporter/actions

Today there was first a single successful run but after that, each one has failed.

Okay, I have a temporary workaround! Try using ubuntu-22.04 instead of ubuntu-latest or ubuntu-24.04.

jobs:

  build:
    runs-on: ubuntu-22.04

Please try like that and let me know if it helps. :)

@crazy-max
Copy link
Member

Not build-push-action but QEMU related similar to docker/setup-qemu-action#188

This is tracked in tonistiigi/binfmt#215

You can try using qemu v8 to see if this fixes the issue for you: tonistiigi/binfmt#215 (comment)

- name: Set up QEMU
  uses: docker/setup-qemu-action@v3
  with:
    image: tonistiigi/binfmt:qemu-v8.1.5

Or switch to ubuntu-22.04 runner.

kahara added a commit to kahara/vushf-exporter that referenced this issue Jan 28, 2025
kahara added a commit to kahara/vushf-exporter that referenced this issue Jan 28, 2025
* cache image for docker/setup-qemu-action

* use v8 tonistiigi/binfmt

As seen in:

docker/build-push-action#1309 (comment)

tonistiigi/binfmt#215 (comment)
@kahara
Copy link

kahara commented Jan 28, 2025

Went with the "v8" approach, and that did it. Thanks everyone!

@FacuM
Copy link
Author

FacuM commented Feb 2, 2025

I do understand that this is caused by a broken external dependency, but why is it closed? This IS still an issue.

paulbourelly999 pushed a commit to usdot-fhwa-OPS/V2X-Hub that referenced this issue Feb 27, 2025
<!-- Thanks for the contribution, this is awesome. -->

# PR Details
## Description
This PR is to attempt to address the issue with segfault in arm64 build.
The fix is applied as suggested in this PR:
docker/build-push-action#1309
<!--- Describe your changes in detail -->

## Related Issue

<!--- This project only accepts pull requests related to open issues -->
<!--- If suggesting a new feature or change, please discuss it in an
issue first -->
<!--- If fixing a bug, there should be an issue describing it with steps
to reproduce -->
<!--- Please link to the issue here: -->

## Motivation and Context

<!--- Why is this change required? What problem does it solve? -->

## How Has This Been Tested?

<!--- Please describe in detail how you tested your changes. -->
<!--- Include details of your testing environment, and the tests you ran
to -->
<!--- see how your change affects other areas of the code, etc. -->

## Types of changes

<!--- What types of changes does your code introduce? Put an `x` in all
the boxes that apply: -->

- [x] Defect fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that cause existing functionality
to change)

## Checklist:

<!--- Go over all the following points, and put an `x` in all the boxes
that apply. -->
<!--- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->

- [ ] I have added any new packages to the sonar-scanner.properties file
- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
[V2XHUB Contributing
Guide](https://github.com/usdot-fhwa-OPS/V2X-Hub/blob/develop/Contributing.md)
- [ ] I have added tests to cover my changes.
- [ ] All new and existing tests passed.
jwillmartin pushed a commit to usdot-fhwa-OPS/V2X-Hub that referenced this issue Mar 4, 2025
<!-- Thanks for the contribution, this is awesome. -->

# PR Details
## Description
Temporary remove arm builds. Issue is similar to following 
<!--- Describe your changes in detail -->
docker/buildx#1169
docker/buildx#1170
might also be related to 
docker/build-push-action#1309
## Related Issue

<!--- This project only accepts pull requests related to open issues -->
<!--- If suggesting a new feature or change, please discuss it in an
issue first -->
<!--- If fixing a bug, there should be an issue describing it with steps
to reproduce -->
<!--- Please link to the issue here: -->

## Motivation and Context

<!--- Why is this change required? What problem does it solve? -->

## How Has This Been Tested?

<!--- Please describe in detail how you tested your changes. -->
<!--- Include details of your testing environment, and the tests you ran
to -->
<!--- see how your change affects other areas of the code, etc. -->

## Types of changes

<!--- What types of changes does your code introduce? Put an `x` in all
the boxes that apply: -->

- [ ] Defect fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that cause existing functionality
to change)

## Checklist:

<!--- Go over all the following points, and put an `x` in all the boxes
that apply. -->
<!--- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->

- [ ] I have added any new packages to the sonar-scanner.properties file
- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
[V2XHUB Contributing
Guide](https://github.com/usdot-fhwa-OPS/V2X-Hub/blob/develop/Contributing.md)
- [ ] I have added tests to cover my changes.
- [ ] All new and existing tests passed.
@paulbourelly999
Copy link

I still encounter this issue with ubuntu 22.04. Not sure why this issue is closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants