BUG: Pushing to Google artifact registry fails

[abbottdev]

Hi,

Trying to use container publishing with Google container (aka Artifact) registry always fails. I've tried playing around with the env vars DOTNET_CONTAINER_DEBUG_REGISTRY_FORCE_CHUNKED_UPLOAD and DOTNET_CONTAINER_REGISTRY_CHUNKED_UPLOAD_SIZE_BYTES - neither setting changed the behaviour.

With a csproj that looks like this:

    <ContainerBaseImage>us-docker.pkg.dev/my-project/images/dotnet-minimal:9.0-main</ContainerBaseImage>
    <ContainerRuntimeIdentifier>linux-x64</ContainerRuntimeIdentifier>
    <ContainerRegistry>us-east1-docker.pkg.dev</ContainerRegistry>
    <ContainerRepository>my-project/images/my-app</ContainerRepository>

I always receive the following build output:

my-app failed with 1 error(s) (10.0s) → src/my-app/bin/Release/net9.0/publish/
    Building image 'my-app' with tags 'latest' on top of base image 'us-docker.pkg.dev/my-project/images/dotnet-minimal:9.0-main'.
    Uploading layer 'sha256:6e909acdb790c5a1989d9cfc795fda5a246ad6664bb27b5c688e2b734b2c5fad' to 'us-east1-docker.pkg.dev'.
    Uploading layer 'sha256:a1da9029f4cb74fb65ddf4fe6670ae679ccd703de242c9e771959c3c215ef22c' to 'us-east1-docker.pkg.dev'.
    Uploading layer 'sha256:fba2434d71078090dc234e052b14b4a1258e256932e87eeca015b355a387c4a5' to 'us-east1-docker.pkg.dev'.
    Uploading layer 'sha256:927305f5e1552617547402d73b787b09549b5220d88277bfb0037644ef834c6c' to 'us-east1-docker.pkg.dev'.
    Uploading layer 'sha256:d905ba5755a11a670fda6f917ae3ec4f02b9c76db01ece30d659a6aba61edb5e' to 'us-east1-docker.pkg.dev'.
    Uploading layer 'sha256:5ee0d36afba19b0f7680fecffe262f6feddbcea1da710dbb25c29a3a74703d94' to 'us-east1-docker.pkg.dev'.
    Uploading layer 'sha256:e7183ae1137d9d79bbf1d0c19631e6c89a9372d4698412315b5373c433ee50b3' to 'us-east1-docker.pkg.dev'.
    Uploading layer 'sha256:6594ce4977ae2ff2a1392a90b72090c68535288dc8c931bd66fb24678e1c7a22' to 'us-east1-docker.pkg.dev'.
    Uploading layer 'sha256:e10594a5f32ed26a7e42d2a2c63851d8ddde1ff8af8d6fd5c0b47f8eea2b65ba' to 'us-east1-docker.pkg.dev'.
    /usr/local/share/dotnet/sdk/9.0.303/Containers/build/Microsoft.NET.Build.Containers.targets(250,5): error : 
      Containerize: error CONTAINER004: CONTAINER1014: Manifest pull failed.
      URI: https://us-east1-docker.pkg.dev/v2/my-app/blobs/sha256:fba2434d71078090dc234e052b14b4a1258e256932e87eeca015b355a387c4a5
      HTTP status code: BadRequest
         at Microsoft.NET.Build.Containers.DefaultBlobOperations.LogAndThrowContainerHttpException[T](HttpResponseMessage response, CancellationToken cancellationToken) in /_/src/Containers/Microsoft.NET.Build.Containers/Registry/DefaultBlobOperations.cs:line 
      79
         at Microsoft.NET.Build.Containers.DefaultBlobOperations.ExistsAsync(String repositoryName, String digest, CancellationToken cancellationToken) in /_/src/Containers/Microsoft.NET.Build.Containers/Registry/DefaultBlobOperations.cs:line 39
         at Microsoft.NET.Build.Containers.Registry.<>c__DisplayClass52_0.<<PushAsync>b__0>d.MoveNext() in /_/src/Containers/Microsoft.NET.Build.Containers/Registry/Registry.cs:line 586
      --- End of stack trace from previous location ---
         at Microsoft.NET.Build.Containers.Registry.PushAsync(BuiltImage builtImage, SourceImageReference source, DestinationImageReference destination, Boolean pushTags, CancellationToken cancellationToken) in /_/src/Containers/Microsoft.NET.Build.Containers/
      Registry/Registry.cs:line 614
         at Microsoft.NET.Build.Containers.ImagePublisher.PushToRemoteRegistryAsync[T](T image, SourceImageReference sourceImageReference, DestinationImageReference destinationImageReference, TaskLoggingHelper Log, CancellationToken cancellationToken, Func`5 p
      ushFunc, String successMessage) in /_/src/Containers/Microsoft.NET.Build.Containers/ImagePublisher.cs:line 145

Build failed with 1 error(s) in 12.7s

Looking at the logs, it looks like the way the SDK publishing is trying to push to GCR/Artifact registry is incorrect. With verbose logging enabled I can see each HTTP request for each layer it's trying to push returns a HTTP 400 Bad Request:

Verbose logs:

                     Building image 'my-app' with tags 'latest' on top of base image 'us-docker.pkg.dev/my-project/images/dotnet-minimal:9.0-main'. (TaskId:332)
                     Setting user from APP_UID environment variable (TaskId:332)
                     Setting ports from ASPNETCORE_HTTP_PORTS environment variable (TaskId:332)
                     Added port 8080 (TaskId:332)
                     Uploading layer 'sha256:6e909acdb790c5a1989d9cfc795fda5a246ad6664bb27b5c688e2b734b2c5fad' to 'us-east1-docker.pkg.dev'. (TaskId:332)
                     Uploading layer 'sha256:a1da9029f4cb74fb65ddf4fe6670ae679ccd703de242c9e771959c3c215ef22c' to 'us-east1-docker.pkg.dev'. (TaskId:332)
                     Uploading layer 'sha256:fba2434d71078090dc234e052b14b4a1258e256932e87eeca015b355a387c4a5' to 'us-east1-docker.pkg.dev'. (TaskId:332)
                     Uploading layer 'sha256:927305f5e1552617547402d73b787b09549b5220d88277bfb0037644ef834c6c' to 'us-east1-docker.pkg.dev'. (TaskId:332)
                     Uploading layer 'sha256:d905ba5755a11a670fda6f917ae3ec4f02b9c76db01ece30d659a6aba61edb5e' to 'us-east1-docker.pkg.dev'. (TaskId:332)
                     Uploading layer 'sha256:5ee0d36afba19b0f7680fecffe262f6feddbcea1da710dbb25c29a3a74703d94' to 'us-east1-docker.pkg.dev'. (TaskId:332)
                     Uploading layer 'sha256:e7183ae1137d9d79bbf1d0c19631e6c89a9372d4698412315b5373c433ee50b3' to 'us-east1-docker.pkg.dev'. (TaskId:332)
                     Uploading layer 'sha256:6594ce4977ae2ff2a1392a90b72090c68535288dc8c931bd66fb24678e1c7a22' to 'us-east1-docker.pkg.dev'. (TaskId:332)
                     Uploading layer 'sha256:ef905e37b3abae70085dea991ac06a670fc4ec23610e6e05d1236841154bf88f' to 'us-east1-docker.pkg.dev'. (TaskId:332)
                     Request URI: HEAD https://us-east1-docker.pkg.dev/v2/my-app/blobs/sha256:927305f5e1552617547402d73b787b09549b5220d88277bfb0037644ef834c6c
                     Status code: BadRequest
                     Response headers:
                     Docker-Distribution-Api-Version: registry/2.0
                     X-Content-Type-Options: nosniff
                     X-Frame-Options: SAMEORIGIN
                     X-XSS-Protection: 0
                     Date: Wed, 06 Aug 2025 09:08:18 GMT
                     
                     Response content: <empty>
                      (TaskId:332)
                     Request URI: HEAD https://us-east1-docker.pkg.dev/v2/my-app/blobs/sha256:6e909acdb790c5a1989d9cfc795fda5a246ad6664bb27b5c688e2b734b2c5fad
                     Status code: BadRequest
                     Response headers:
                     Docker-Distribution-Api-Version: registry/2.0
                     X-Content-Type-Options: nosniff
                     X-Frame-Options: SAMEORIGIN
                     X-XSS-Protection: 0
                     Date: Wed, 06 Aug 2025 09:08:18 GMT
                     
                     Response content: <empty>
                      (TaskId:332)
                     Request URI: HEAD https://us-east1-docker.pkg.dev/v2/my-app/blobs/sha256:ef905e37b3abae70085dea991ac06a670fc4ec23610e6e05d1236841154bf88f
                     Status code: BadRequest
                     Response headers:
                     Docker-Distribution-Api-Version: registry/2.0
                     X-Content-Type-Options: nosniff
                     X-Frame-Options: SAMEORIGIN
                     X-XSS-Protection: 0
                     Date: Wed, 06 Aug 2025 09:08:18 GMT
                     
                     Response content: <empty>
                      (TaskId:332)
                     Request URI: HEAD https://us-east1-docker.pkg.dev/v2/my-app/blobs/sha256:fba2434d71078090dc234e052b14b4a1258e256932e87eeca015b355a387c4a5
                     Status code: BadRequest
                     Response headers:
                     Docker-Distribution-Api-Version: registry/2.0
                     X-Content-Type-Options: nosniff
                     X-Frame-Options: SAMEORIGIN
                     X-XSS-Protection: 0
                     Date: Wed, 06 Aug 2025 09:08:18 GMT
                     
                     Response content: <empty>
                      (TaskId:332)
                     Request URI: HEAD https://us-east1-docker.pkg.dev/v2/my-app/blobs/sha256:5ee0d36afba19b0f7680fecffe262f6feddbcea1da710dbb25c29a3a74703d94
                     Status code: BadRequest
                     Response headers:
                     Docker-Distribution-Api-Version: registry/2.0
                     X-Content-Type-Options: nosniff
                     X-Frame-Options: SAMEORIGIN
                     X-XSS-Protection: 0
                     Date: Wed, 06 Aug 2025 09:08:18 GMT
                     
                     Response content: <empty>
                      (TaskId:332)
                     Request URI: HEAD https://us-east1-docker.pkg.dev/v2/my-app/blobs/sha256:d905ba5755a11a670fda6f917ae3ec4f02b9c76db01ece30d659a6aba61edb5e
                     Status code: BadRequest
                     Response headers:
                     Docker-Distribution-Api-Version: registry/2.0
                     X-Content-Type-Options: nosniff
                     X-Frame-Options: SAMEORIGIN
                     X-XSS-Protection: 0
                     Date: Wed, 06 Aug 2025 09:08:18 GMT
                     
                     Response content: <empty>
                      (TaskId:332)
                     Request URI: HEAD https://us-east1-docker.pkg.dev/v2/my-app/blobs/sha256:a1da9029f4cb74fb65ddf4fe6670ae679ccd703de242c9e771959c3c215ef22c
                     Status code: BadRequest
                     Response headers:
                     Docker-Distribution-Api-Version: registry/2.0
                     X-Content-Type-Options: nosniff
                     X-Frame-Options: SAMEORIGIN
                     X-XSS-Protection: 0
                     Date: Wed, 06 Aug 2025 09:08:18 GMT
                     
                     Response content: <empty>
                      (TaskId:332)
                     Request URI: HEAD https://us-east1-docker.pkg.dev/v2/my-app/blobs/sha256:6594ce4977ae2ff2a1392a90b72090c68535288dc8c931bd66fb24678e1c7a22
                     Status code: BadRequest
                     Response headers:
                     Docker-Distribution-Api-Version: registry/2.0
                     X-Content-Type-Options: nosniff
                     X-Frame-Options: SAMEORIGIN
                     X-XSS-Protection: 0
                     Date: Wed, 06 Aug 2025 09:08:18 GMT
                     
                     Response content: <empty>
                      (TaskId:332)
                     Request URI: HEAD https://us-east1-docker.pkg.dev/v2/my-app/blobs/sha256:e7183ae1137d9d79bbf1d0c19631e6c89a9372d4698412315b5373c433ee50b3
                     Status code: BadRequest
                     Response headers:
                     Docker-Distribution-Api-Version: registry/2.0
                     X-Content-Type-Options: nosniff
                     X-Frame-Options: SAMEORIGIN
                     X-XSS-Protection: 0
                     Date: Wed, 06 Aug 2025 09:08:18 GMT
                     
                     Response content: <empty>
                      (TaskId:332)