Invalid digest error when running the plugin on Ubuntu 24.04

[lovetodream]

Description

When using the plugin on a machine with Ubuntu 24.04, I am receiving the following error on upload:

Build of product 'App' complete! (104.98s)
[ContainerImageBuilder] Found base image manifest: sha256:3d417a61bb39f2147c8f8e9ca33885955aa17f7c218bbc33ebe0aaf475041b51
[ContainerImageBuilder] Found base image configuration: sha256:2a486c7915a39ccee40fb774503ed64d3ff95575cbf19b21f82cecc3537b121d
[ContainerImageBuilder]
[ContainerImageBuilder] Built application layer
[ContainerImageBuilder]
[ContainerImageBuilder] Uploading application layer
[ContainerImageBuilder]
[ContainerImageBuilder] E
[ContainerImageBuilder] r
[ContainerImageBuilder] r
[ContainerImageBuilder] o
[ContainerImageBuilder] r: DIGEST_INVALID: provided digest did not match uploaded conten

I also tested the HelloWorldHummingbird example, the issue happens there too. It worked on a macOS Sequoia Machine.

The registry I am trying to upload to is an internally hosted version of https://hub.docker.com/_/registry (v2).

Reproduction

(Optionally) start a registry server on your network with a reverse proxy in front of it.

docker run -d -p 5000:5000 --restart always --name registry registry:2

Run the plugin command (using the Examples/HelloWorldHummingbird).

swift package --swift-sdk x86_64-swift-linux-musl build-container-image --from swift:slim --repository registry.domain.local/hello

Package versions

Plugin Version: 0.2.0

Expected behavior

I'd expect a successful upload.

Environment

Ubuntu: Ubuntu 24.04.1 LTS
Swift version 6.0.1 (swift-6.0.1-RELEASE)
Target: x86_64-unknown-linux-gnu
Static SDK: swift-6.0.1-RELEASE_static-linux-0.0.1

Additional information

Maybe there is a difference in checksum calculation when running on macOS vs Linux.

Registry Log:

172.17.0.1 - - [25/Oct/2024:09:58:32 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "containertool-tool (unknown version) curl/8.5.0"
time="2024-10-25T09:58:42.233143641Z" level=info msg="response completed" go.version=go1.20.8 http.request.host=registry.domain.local http.request.id=ab2c35b8-14f5-4675-a394-084517da7df1 http.request.method=POST http.request.remoteaddr=10.70.99.46 http.request.uri="/v2/hello/blobs/uploads/" http.request.useragent="containertool-tool (unknown version) curl/8.5.0" http.response.duration=7.204548ms http.response.status=202 http.response.written=0
172.17.0.1 - - [25/Oct/2024:09:58:42 +0000] "POST /v2/hello/blobs/uploads/ HTTP/1.1" 202 0 "" "containertool-tool (unknown version) curl/8.5.0"
time="2024-10-25T09:58:42.476300767Z" level=error msg="canonical digest does match provided digest" canonical=sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 go.version=go1.20.8 http.request.contenttype="application/octet-stream" http.request.host=registry.domain.local http.request.id=3ee4ddf6-0f4f-413f-b752-012688f56f39 http.request.method=PUT http.request.remoteaddr=10.70.99.46 http.request.uri="/v2/hello/blobs/uploads/fffe86d9-eeb0-4da3-8337-8fe24094300c?_state=q3veN8OkuhakmA670HatR5kzlj_c4DGhHtaBCSizdJd7Ik5hbWUiOiJoZWxsbyIsIlVVSUQiOiJmZmZlODZkOS1lZWIwLTRkYTMtODMzNy04ZmUyNDA5NDMwMGMiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjQtMTAtMjVUMDk6NTg6NDIuMjI3MjE4OTI4WiJ9&digest=sha256:a171af53824f3224ea2848f2bc07c9e7ffabec60a628cb8631eab510790b07cc" http.request.useragent="containertool-tool (unknown version) curl/8.5.0" provided=sha256:a171af53824f3224ea2848f2bc07c9e7ffabec60a628cb8631eab510790b07cc vars.name=hello vars.uuid=fffe86d9-eeb0-4da3-8337-8fe24094300c
time="2024-10-25T09:58:42.476849465Z" level=error msg="error canceling upload after error: already closed" go.version=go1.20.8 http.request.contenttype="application/octet-stream" http.request.host=registry.domain.local http.request.id=3ee4ddf6-0f4f-413f-b752-012688f56f39 http.request.method=PUT http.request.remoteaddr=10.70.99.46 http.request.uri="/v2/hello/blobs/uploads/fffe86d9-eeb0-4da3-8337-8fe24094300c?_state=q3veN8OkuhakmA670HatR5kzlj_c4DGhHtaBCSizdJd7Ik5hbWUiOiJoZWxsbyIsIlVVSUQiOiJmZmZlODZkOS1lZWIwLTRkYTMtODMzNy04ZmUyNDA5NDMwMGMiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjQtMTAtMjVUMDk6NTg6NDIuMjI3MjE4OTI4WiJ9&digest=sha256:a171af53824f3224ea2848f2bc07c9e7ffabec60a628cb8631eab510790b07cc" http.request.useragent="containertool-tool (unknown version) curl/8.5.0" vars.name=hello vars.uuid=fffe86d9-eeb0-4da3-8337-8fe24094300c
time="2024-10-25T09:58:42.479405139Z" level=error msg="response completed with error" err.code="digest invalid" err.detail="invalid digest for referenced layer: sha256:a171af53824f3224ea2848f2bc07c9e7ffabec60a628cb8631eab510790b07cc, content does not match digest" err.message="provided digest did not match uploaded content" go.version=go1.20.8 http.request.contenttype="application/octet-stream" http.request.host=registry.domain.local http.request.id=3ee4ddf6-0f4f-413f-b752-012688f56f39 http.request.method=PUT http.request.remoteaddr=10.70.99.46 http.request.uri="/v2/hello/blobs/uploads/fffe86d9-eeb0-4da3-8337-8fe24094300c?_state=q3veN8OkuhakmA670HatR5kzlj_c4DGhHtaBCSizdJd7Ik5hbWUiOiJoZWxsbyIsIlVVSUQiOiJmZmZlODZkOS1lZWIwLTRkYTMtODMzNy04ZmUyNDA5NDMwMGMiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjQtMTAtMjVUMDk6NTg6NDIuMjI3MjE4OTI4WiJ9&digest=sha256:a171af53824f3224ea2848f2bc07c9e7ffabec60a628cb8631eab510790b07cc" http.request.useragent="containertool-tool (unknown version) curl/8.5.0" http.response.contenttype="application/json; charset=utf-8" http.response.duration=7.335909ms http.response.status=400 http.response.written=204 vars.name=hello vars.uuid=fffe86d9-eeb0-4da3-8337-8fe24094300c
172.17.0.1 - - [25/Oct/2024:09:58:42 +0000] "PUT /v2/hello/blobs/uploads/fffe86d9-eeb0-4da3-8337-8fe24094300c?_state=q3veN8OkuhakmA670HatR5kzlj_c4DGhHtaBCSizdJd7Ik5hbWUiOiJoZWxsbyIsIlVVSUQiOiJmZmZlODZkOS1lZWIwLTRkYTMtODMzNy04ZmUyNDA5NDMwMGMiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjQtMTAtMjVUMDk6NTg6NDIuMjI3MjE4OTI4WiJ9&digest=sha256:a171af53824f3224ea2848f2bc07c9e7ffabec60a628cb8631eab510790b07cc HTTP/1.1" 400 204 "" "containertool-tool (unknown version) curl/8.5.0"

[euanh]

Hi @lovetodream

I'm afraid haven't been able to reproduce this problem yet. From the logs it looks like:

• The plugin found the swift:slim base image.
• It built the application image layer which will be stacked on top of swift:slim.
• It failed while it was trying to upload the application image layer. It's hard to tell exactly which sub-step of this upload failed:
  • First, it uploads the image layer, containing the executable.
  • Then it uploads the image configuration metadata.
  • Finally it uploads a manifest object.
• It did not get to the stage of uploading the base image layers; this only happens after the application image layer has been uploaded.

Could you please give me some more information?

• Could you please re-run your example passing the --verbose flag?

  swift package --swift-sdk x86_64-swift-linux-musl build-container-image --verbose --from swift:slim --repository registry.domain.local/hello
  

• Are you running your Ubuntu build on an Ubuntu host/VM, or in an Ubuntu container? If you're running in a publicly available container, can you post the image reference?

• In your reproduction notes you say (Optionally) start a registry server on your network with a reverse proxy in front of it.
  Which reverse proxy are you using, and how is it configured?

• Could you please try the following commands on your Ubuntu machine:

  # Check out the plugin repository
  git clone https://github.com/apple/swift-container-plugin --branch 0.2.0 && cd swift-container-plugin
  
  # Run the helper tool which packages and uploads the container.
  # In this case "README.md" will be used as the payload, so the container image won't be able to run.
  # This is just to exercise the digest calculation.
  swift run containertool --verbose --repository registry.domain.local/hello --from swift:slim README.md
  
  # Run the helper tool again.   This time, do not include a base image.   This excludes the problem 
  swift run containertool --verbose --repository registry.domain.local/hello --from scratch README.md