Buildpacks do not support Docker with containerd image store

[n-g]

Tested with Spring Boot version 3.2.4.

We have found that Spring Boot with Buildpacks fails when enabling the upcoming containerd image store in Docker.

How to reproduce:

1. Enable the containerd image store in Docker Desktop:
   https://docs.docker.com/desktop/containerd/#enable-the-containerd-image-store
2. Clone an example
   https://github.com/gavlyukovskiy/benchmark-spring-application
3. Build the example

% ./gradlew :spring-web:build :spring-web:bootBuildImage

> Task :spring-web:bootBuildImage
Building image 'docker.io/library/spring-web-tomcat:latest'

 > Pulling builder image 'docker.io/paketobuildpacks/builder-jammy-base:latest' ..................................................
 > Pulled builder image 'paketobuildpacks/builder-jammy-base@sha256:a913462b288209172cb7626c2ded55843002a7aa1a6bc87e9af52be1ab5511fc'
 > Pulling run image 'docker.io/paketobuildpacks/run-jammy-base:latest' ..................................................
 > Pulled run image 'paketobuildpacks/run-jammy-base@sha256:8431203470391fc58454b71bdb917f53c20f403892fbb447f4ea5265a8d7cf49'
 > Pulling buildpack image 'gcr.io/paketo-buildpacks/adoptium@sha256:86d031571027360c4c35d31b1d37e7a1fc36c24770fa0400d8c2eac5c73863c2' ..................................................
 > Pulled buildpack image 'gcr.io/paketo-buildpacks/adoptium@sha256:86d031571027360c4c35d31b1d37e7a1fc36c24770fa0400d8c2eac5c73863c2'

> Task :spring-web:bootBuildImage FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':spring-web:bootBuildImage'.
> Error pulling buildpack image 'gcr.io/paketo-buildpacks/adoptium@sha256:86d031571027360c4c35d31b1d37e7a1fc36c24770fa0400d8c2eac5c73863c2'

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
> Get more help at https://help.gradle.org.

BUILD FAILED in 3s
8 actionable tasks: 4 executed, 4 up-to-date

The build succeeds when disabling the containerd image store

What we found:
When exporting layers, the Spring Boot Buildpacks integration assumes that each layer is stored as an uncompressed TAR
source

When enabling the containerd image store, Docker serves the image with one gzip compressed layer.

This internally leads to

Caused by: java.io.IOException: Corrupted TAR archive.
    at org.apache.commons.compress.archivers.tar.TarArchiveEntry.parseTarHeader(TarArchiveEntry.java:1597)
    at org.apache.commons.compress.archivers.tar.TarArchiveEntry.<init>(TarArchiveEntry.java:556)
    at org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:379)
    at org.springframework.boot.buildpack.platform.build.ImageBuildpack$ExportedLayers.copyLayerTar(ImageBuildpack.java:134)
    at org.springframework.boot.buildpack.platform.build.ImageBuildpack$ExportedLayers.copyToTemp(ImageBuildpack.java:125)
    at org.springframework.boot.buildpack.platform.build.ImageBuildpack$ExportedLayers.lambda$new$0(ImageBuildpack.java:118)
    at org.springframework.boot.buildpack.platform.docker.DockerApi$ImageApi.exportLayerFiles(DockerApi.java:302)
    at org.springframework.boot.buildpack.platform.build.Builder$BuilderResolverContext.exportImageLayers(Builder.java:277)
    at org.springframework.boot.buildpack.platform.build.ImageBuildpack$ExportedLayers.<init>(ImageBuildpack.java:118)
    at org.springframework.boot.buildpack.platform.build.ImageBuildpack.<init>(ImageBuildpack.java:65)
    ... 128 more
Caused by: java.lang.IllegalArgumentException: Invalid byte 38 at offset 0 in '&���[' len=8
    at org.apache.commons.compress.archivers.tar.TarUtils.parseOctal(TarUtils.java:153)
    at org.apache.commons.compress.archivers.tar.TarUtils.parseOctalOrBinary(TarUtils.java:183)
    at org.apache.commons.compress.archivers.tar.TarArchiveEntry.parseOctalOrBinary(TarArchiveEntry.java:1698)
    at org.apache.commons.compress.archivers.tar.TarArchiveEntry.parseTarHeaderUnwrapped(TarArchiveEntry.java:1609)
    at org.apache.commons.compress.archivers.tar.TarArchiveEntry.parseTarHeader(TarArchiveEntry.java:1595)

Background:
Docker is moving the image store to containerd to enable a couple of new use cases.
See https://docs.docker.com/desktop/containerd/#image-store
The containerd image store will eventually become the default.

As per
https://github.com/opencontainers/image-spec/blob/main/media-types.md#oci-image-media-types
layers may be stored in three types:

• application/vnd.oci.image.layer.v1.tar: "Layer", as a tar archive
• application/vnd.oci.image.layer.v1.tar+gzip: "Layer", as a tar archive compressed with gzip
• application/vnd.oci.image.layer.v1.tar+zstd: "Layer", as a tar archive compressed with zstd

Hence, for full compatibility with the image spec, the layer exporting code needs to cover the uncompressed case, as well as gzip and zstd.

[philwebb]

Although this is a bug, we're nervous about fixing it in 3.1 since we're about to move out of OSS support. Instead we're only going to fix this in 3.2.x and forward.

[n-g]

Thank you for the fix and the update!