Error when using the new multi-arch Paketo buildpacks

[ThomasVitale]

The Paketo Buildpacks project is now publishing multi-arch buildpacks for Java and Java Native Image. They both works great to containerize Spring Boot applications (3.2.5, 3.3.0-M1, 3.3.0-SNAPSHOT) when using the pack CLI.

However, when using the Spring Boot plugin (Gradle or Maven), the build only works on ARM64 architectures whereas it gets stuck indefinitely when running on AMD64 architectures (at least on Linux. I hope I'll get to test this on Windows as well and update this issue with more info).

I have created a repo with two examples to demonstrate the problem: https://github.com/ThomasVitale/buildpacks-demo.

> Task :bootBuildImage
Building image 'docker.io/library/demo:latest'

 > Pulling builder image 'docker.io/paketobuildpacks/builder-jammy-buildpackless-tiny:latest' ..................................................
 > Pulled builder image 'paketobuildpacks/builder-jammy-buildpackless-tiny@sha256:589ca95ba0d64b32b5c2e7b65768e2d22c1bbafe61f8348280e6a9064fb8185f'
 > Pulling run image 'docker.io/paketobuildpacks/run-jammy-tiny:latest' ..................................................
 > Pulled run image 'paketobuildpacks/run-jammy-tiny@sha256:c6c34fd722defe44d4630061eec4d8fa8188508508016b7682388adb68a57b42'
 > Pulling buildpack image 'gcr.io/paketo-buildpacks/java:latest' ..................................................
 > Pulled buildpack image 'paketobuildpacks/java@sha256:33b879227101ce429cf6f156178abdb30920ffa9b14435dc1c460b9dcb8121e0'

The task gets stuck after pulling the buildpack image. It will only stop when forcefully shut down (on GitHub Actions that happens after 6 hours by default).

[mhalbritter]

It works on my machine, though. Both projects work fine with g bootBuildImage.

Env:

> docker version
Client: Docker Engine - Community
 Version:           26.1.2
 API version:       1.45
 Go version:        go1.21.10
 Git commit:        211e74b
 Built:             Wed May  8 14:01:38 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          26.1.2
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.10
  Git commit:       ef1912d
  Built:            Wed May  8 13:59:55 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.31
  GitCommit:        e377cd56a71523140ca6ae87e30244719194a521
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

> uname -a
Linux workstation 6.8.9-200.fc39.x86_64 #1 SMP PREEMPT_DYNAMIC Thu May  2 18:44:19 UTC 2024 x86_64 GNU/Linux

log.txt

[ThomasVitale]

Thanks @mhalbritter for the feedback. I've done some more testing. I have tried the following scenarios and they all worked fine, based on Docker Desktop 4.30.0 (Docker Engine 26.1.1).

• macOS (ARM64)
• Windows 10 (AMD64)
• Ubuntu 22.04 on WSL2/Windows 10 (AMD64)

I get the error in the following scenarios. I wonder if it's something related to the older Docker version or something specific to the VM images used by GitHub.

• ubuntu-22.04 GitHub Runners using Docker Engine 24.0.9 (example: https://github.com/ThomasVitale/supply-chain-security-java/actions/runs/9038631465/job/24840127488). Runner configuration: https://github.com/actions/runner-images/blob/main/images/ubuntu/Ubuntu2204-Readme.md
• Ubuntu 20.04 on GitHub Codespaces using Docker Engine 24.0.9 (it can be tested from this repo: https://github.com/ThomasVitale/buildpacks-demo).

[mhalbritter]

Thanks for the feedback. The failures on CI are reproducible, and not some flaky internet connection, right?

[ThomasVitale]

@mhalbritter yes, the failure on GitHub Actions (Ubuntu 22.04) and GitHub Codespaces (Ubuntu 20.04) is reproducible and happens consistently.

[ThomasVitale]

I also tried the build on newer GitHub runners (Ubuntu 24.04, Docker 26.1.0) and the outcome is the same: the build hangs indefinitely. Example: https://github.com/ThomasVitale/supply-chain-security-java/actions/runs/9059907830/job/25092379764

[dashaun]

I'm working with @NotBjoggisAtAll I'm validating things on my side as well, I'm not running into the issue on any of my environments.

Here is my Spring Boot Maven Plugin config:

  <build>
    <plugins>
      <plugin>
        <groupId>org.graalvm.buildtools</groupId>
        <artifactId>native-maven-plugin</artifactId>
      </plugin>
      <plugin>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-maven-plugin</artifactId>
        <executions>
          <execution>
            <id>build-info</id>
            <goals>
              <goal>build-info</goal>
            </goals>
          </execution>
        </executions>
        <configuration>
          <image>
            <builder>paketobuildpacks/builder-jammy-buildpackless-tiny</builder>
            <buildpacks>
              <buildpack>paketobuildpacks/java-native-image</buildpack>
            </buildpacks>
            <name>dashaun/${project.name}:v${project.version}-${os.detected.arch}</name>
            <env>
              <BP_JVM_VERSION>22</BP_JVM_VERSION>
              <BP_NATIVE_IMAGE_BUILD_ARGUMENTS>-H:-AddAllFileSystemProviders</BP_NATIVE_IMAGE_BUILD_ARGUMENTS>
            </env>
          </image>
        </configuration>
      </plugin>
    </plugins>
  </build>

It's working for me on Github with this pipeline, using both Java 21 and Java 22:

name: Native-AMD64

on:
  push:
    tags:
      - "v*"

env:
  IMAGE_NAME: dashaun/io.dashaun.service.gateway

jobs:
  build:

    runs-on: ubuntu-latest

    steps:
      #Login to DockerHub
      - name: Login to DockerHub
        uses: docker/login-action@v2
        with:
          username: dashaun
          password: ${{ secrets.DOCKERHUB_TOKEN }}
      - uses: actions/setup-java@v2
        with:
          distribution: 'liberica' # See 'Supported distributions' for available options
          java-version: '22'
      - name: Checkout master
        uses: actions/checkout@v3
        with:
          submodules: true
      #Build Image
      - name: Build Image
        run: ./mvnw -Pnative spring-boot:build-image
      #Deploy the image to the Docker registry
      - name: Push Images to Docker Registry
        run: docker push -a $IMAGE_NAME


  manifest:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - name: Login to DockerHub
        uses: docker/login-action@v2
        with:
          username: dashaun
          password: ${{ secrets.DOCKERHUB_TOKEN }}
      - name: pull-arm64
        uses: nick-fields/retry@v2
        with:
          timeout_minutes: 5
          retry_wait_seconds: 60
          max_attempts: 6
          command: docker pull $IMAGE_NAME:$GITHUB_REF_NAME-aarch_64
      - name: create-manifest
        run: |
          docker manifest create $IMAGE_NAME:$GITHUB_REF_NAME --amend $IMAGE_NAME:$GITHUB_REF_NAME-x86_64 --amend $IMAGE_NAME:$GITHUB_REF_NAME-aarch_64
          docker manifest push $IMAGE_NAME:$GITHUB_REF_NAME
          docker manifest create $IMAGE_NAME:latest --amend $IMAGE_NAME:$GITHUB_REF_NAME-x86_64 --amend $IMAGE_NAME:$GITHUB_REF_NAME-aarch_64
          docker manifest push $IMAGE_NAME:latest

It's also working for me consistently on CircleCI (ARM64) with this config, Java 21 and Java 22:

version: 2.1

orbs:
  docker: circleci/docker@2.4.0
  sdkman: joshdholtz/sdkman@0.2.0

jobs:
  arm64-native:
    machine:
      image: ubuntu-2004:current
      resource_class: arm.medium
    steps:
      - checkout
      - sdkman/setup-sdkman
      - sdkman/sdkman-install:
          candidate: java
          version: 24.0.1.r22-nik
      - run:
          name: "mvnw -Pnative spring-boot:build-image"
          command: "./mvnw -Pnative spring-boot:build-image"
      - docker/check:
          docker-username: DOCKER_LOGIN
          docker-password: DOCKERHUB_PASSWORD
      - docker/push:
          image: dashaun/$CIRCLE_PROJECT_REPONAME
          tag: $CIRCLE_TAG-aarch_64

workflows:
  arm64-native-workflow:
    jobs:
      - arm64-native:
          context:
            - dashaun-dockerhub
          filters:
            tags:
              only: /^v.*/
            branches:
              ignore: /.*/

It also works locally on all of my machines, I haven't been able to reproduce this issue yet.

I'm using this repo to experiment with: https://github.com/dashaun-io/io.dashaun.service.gateway

@ThomasVitale Lets find some time to pair on this!

[NotBjoggisAtAll]

I have a pipeline using the new buildpacks and I have issues with one of them.

Works fine on Windows 11.
My Github action runner (ubuntu-latest) works fine.
My selfhosted runner running on ARM64 Ubuntu 22.04.4 LTS (docker engine version 26.1.3) is not working.
https://github.com/jonasandersen-no/jonasandersen-admin/actions/runs/9429070241

Github action workflow

name: Build latest master and deploy

on:
  push:
    branches:
      - master
      - java22

jobs:
  build:
    strategy:
      matrix:
        os: [ ubuntu-latest, self-hosted ]
    runs-on: ${{ matrix.os }}
    permissions:
      contents: read
      packages: write
    steps:
      - name: Setup Maven Action
        uses: s4u/setup-maven-action@v1.12.0
        with:
          java-version: 21
          java-distribution: liberica
          checkout-persist-credentials: true
          cache-enabled: false
      - name: Log in to the Container registry
        uses: docker/login-action@v3
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}

      - name: Build with Maven
        run: mvn clean package -Pnative

      - name: Push Images to Docker Registry
        run: docker push -a bjoggis/jonasandersen-admin
  manifest:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - name: Login to DockerHub
        uses: docker/login-action@v2
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}
      - name: create-manifest
        run: |
          docker manifest create bjoggis/jonasandersen-admin:v0.0.0 --amend bjoggis/jonasandersen-admin:v0.0.0-x86_64 --amend bjoggis/jonasandersen-admin:v0.0.0-aarch_64
          docker manifest push bjoggis/jonasandersen-admin:v0.0.0
          docker manifest create bjoggis/jonasandersen-admin:latest --amend bjoggis/jonasandersen-admin:v0.0.0-x86_64 --amend bjoggis/jonasandersen-admin:v0.0.0-aarch_64
          docker manifest push bjoggis/jonasandersen-admin:latest

pom plugin setup:

      <plugin>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-maven-plugin</artifactId>
        <executions>
          <execution>
            <id>build-info</id>
            <goals>
              <goal>build-info</goal>
            </goals>
          </execution>
          <execution>
            <goals>
              <goal>build-image-no-fork</goal>
            </goals>
          </execution>
        </executions>
        <configuration>
          <image>
            <builder>paketobuildpacks/builder-jammy-buildpackless-tiny</builder>
            <name>bjoggis/${project.artifactId}:v${project.version}-${os.detected.arch}</name>
            <buildpacks>
              <buildpack>paketobuildpacks/java-native-image</buildpack>
            </buildpacks>
            <env>
              <BP_JVM_VERSION>21</BP_JVM_VERSION>
            </env>
          </image>
        </configuration>
      </plugin>

[ThomasVitale]

I have extended my sample repo to include GitHub Actions pipelines on Ubuntu 20.04, Ubuntu 22.04 and 24.04 (AMD64).

The build works on 20.04 and 24.04, but it fails on 22.04.
Example: https://github.com/ThomasVitale/buildpacks-demo/actions/runs/9429404583/job/25975753286

GitHub Runners Specs:

• ubuntu-20.04
• ubuntu-22.04
• ubuntu-24.04

I have also added a DevContainer specification to easily open the project in GitHub Codespaces, which use Debian 11 Bullseye (and Docker 24). The build fails there, too.

Notice: it's only the regular Java build that has issues. When building a native image, I haven't experienced any issue.
Furthermore, I have only found issues on Linux AMD64 systems.

@dashaun I'm available to pair on this!