Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

podman push does not work, but docker push works on heroku registry #10309

Closed
damnms opened this issue May 11, 2021 · 9 comments
Closed

podman push does not work, but docker push works on heroku registry #10309

damnms opened this issue May 11, 2021 · 9 comments
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.

Comments

@damnms
Copy link

damnms commented May 11, 2021

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description
I visited a devops course and in the last lesson the teacher used docker to push an image to heroku's image registry and let this run. I wanted to use podman because they should be compatible. The docker push worked whereas the podman push did not.

Steps to reproduce the issue:

  1. create a heroku account

  2. checkout https://github.com/damnms/DevOpsPath

  3. docker build -t zhaw-mosa/devopspath .

  4. install or download heroku cli

  5. copy /usr/bin/podman to /usr/bin/docker //otherwise heroku will complain...

  6. heroku login

  7. heroku container:login //guess this could be done with podman login too

  8. heroku create --region eu

  9. heroku container:push web or podman push registry.heroku.com/$THE_NAME_FROM_STEP8/web

Describe the results you received:
[oli@lucy DevOpsPath]$podman push registry.heroku.com/afternoon-garden-35911/web
Getting image source signatures
Copying blob 1e77dd81f9fa skipped: already exists
Copying blob 030309cad0ba skipped: already exists
Copying blob 557433955211 skipped: already exists
Copying blob 6f15325cc380 skipped: already exists
Copying blob ca85d5f63d9d skipped: already exists
Copying blob ae69bc9df648 skipped: already exists
Copying blob c6e64e7942e5 skipped: already exists
Copying blob 5f70bf18a086 skipped: already exists
Copying blob 8b10c0407caf skipped: already exists
Copying blob 9dcb91c9af71 [--------------------------------------] 0.0b / 0.0b
Copying config b18c874e7d [--------------------------------------] 0.0b / 5.5KiB
Writing manifest to image destination
Error: Error writing manifest "{"schemaVersion":2,"config":{"mediaType":"application/vnd.oci.image.config.v1+json","digest":"sha256:b18c874e7dc895c5e80ea8472e0f1d7ca6ebe818269641369f39c6b7ddb84a65","size":5591},"layers":[{"mediaType":"application/vnd.oci.image.layer.v1.tar+gzip","digest":"sha256:2eab874737fe4c4a4996b2b12e8ac379bca8703588e63a54297279f3711f1a4d","size":27928748},{"mediaType":"application/vnd.oci.image.layer.v1.tar+gzip","digest":"sha256:8581af0d7e7379e3d92ac56384a0c197c9425fb360d5b00836e5337419a183ad","size":921},{"mediaType":"application/vnd.oci.image.layer.v1.tar+gzip","digest":"sha256:a6ff4aede508e7d435f63d6bbaafd8f5c6841ac4b136871530d0929d7546bc69","size":204},{"mediaType":"application/vnd.oci.image.layer.v1.tar+gzip","digest":"sha256:b3e208ec4920de022c3e222c74e79ae14084ee99d356d040b91d4659d6f93747","size":232527432},{"mediaType":"application/vnd.oci.image.layer.v1.tar+gzip","digest":"sha256:7a758745be8c71712555423328b627edf74355c6eabb0b6497023d78ff0d1437","size":86561314},{"mediaType":"application/vnd.oci.image.layer.v1.tar+gzip","digest":"sha256:16fe1e77ef1aae905e67bccbb479c2bb458357f7f6d391a7541a653e9cff9a0c","size":129519078},{"mediaType":"application/vnd.oci.image.layer.v1.tar+gzip","digest":"sha256:443602cc0ea7a4fd199f28fb1fb318308c154985acbd976561d57d7453ac1665","size":221448761},{"mediaType":"application/vnd.oci.image.layer.v1.tar+gzip","digest":"sha256:d49f5b603640d3acefdc4250d2d952ace07eead5b95b36e92d2c0a2afe852525","size":8380946},{"mediaType":"application/vnd.oci.image.layer.v1.tar+gzip","digest":"sha256:4ca545ee6d5db5c1170386eeb39b2ffe3bd46e5d4a73a9acbebc805f19607eb3","size":42},{"mediaType":"application/vnd.oci.image.layer.v1.tar+gzip","digest":"sha256:d4fc7d95db290624cfe1a353659b92289eb3a321d25ea4e8e5eafc6b2f34dc77","size":386801878}]}": Error uploading manifest latest to registry.heroku.com/afternoon-garden-35911/web: unsupported

Describe the results you expected:
when i run the same with docker, it works:
root@jitsii:/home/oli/DevOpsPath# docker push registry.heroku.com/guarded-springs-24685/web
Using default tag: latest
The push refers to repository [registry.heroku.com/guarded-springs-24685/web]
29064dc73b4e: Pushing [=====================> ] 190.3MB/438.2MB
d501667405ca: Layer already exists
48f22651fab0: Layer already exists
5de2531fb0cb: Pushing [=================> ] 201MB/583.8MB
5cf0446ac026: Layer already exists
5c35c950784e: Layer already exists
ae48844230f4: Pushing [=====================================> ] 149.7MB/198.9MB
ae69bc9df648: Pushing [====================> ] 160.5MB/387.6MB
6f15325cc380: Layer already exists
1e77dd81f9fa: Pushed
030309cad0ba: Pushing [================================================> ] 61.18MB/63.26MB
^C

Additional information you deem important (e.g. issue happens only occasionally):
i could imagine that this has something to do with authentication! when i use docker login, it creates a $HOME/.docker/config.json which i do not have when logging in with podman

Output of podman version:

podman version 3.2.0-rc1

Output of podman info --debug:

[oli@lucy heroku]$ podman info --debug
host:
  arch: amd64
  buildahVersion: 1.20.2-dev
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.27-2.fc34.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.27, commit: '
  cpus: 16
  distribution:
    distribution: fedora
    version: "34"
  eventLogger: journald
  hostname: lucy
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.11.18-300.fc34.x86_64
  linkmode: dynamic
  memFree: 4691214336
  memTotal: 16697237504
  ociRuntime:
    name: crun
    package: crun-0.19.1-2.fc34.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.19.1
      commit: 1535fedf0b83fb898d449f9680000f729ba719f5
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.9-1.fc34.x86_64
    version: |-
      slirp4netns version 1.1.8+dev
      commit: 6dc0186e020232ae1a6fcc1f7afbc3ea02fd3876
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 8579182592
  swapTotal: 8589930496
  uptime: 7h 16m 25.05s (Approximately 0.29 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/oli/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 1
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.5.0-1.fc34.x86_64
      Version: |-
        fusermount3 version: 3.10.2
        fuse-overlayfs: version 1.5
        FUSE library version 3.10.2
        using FUSE kernel interface version 7.31
  graphRoot: /home/oli/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 24
  runRoot: /run/user/1000/containers
  volumePath: /home/oli/.local/share/containers/storage/volumes
version:
  APIVersion: 3.2.0-rc1
  Built: 1620248626
  BuiltTime: Wed May  5 23:03:46 2021
  GitCommit: ""
  GoVersion: go1.16.3
  OsArch: linux/amd64
  Version: 3.2.0-rc1


Package info (e.g. output of rpm -q podman or apt list podman):

[oli@lucy heroku]$ rpm -qa | grep podman
podman-plugins-3.2.0-0.1.rc1.fc34.x86_64
podman-3.2.0-0.1.rc1.fc34.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):
i have a little server that has a docker install (debian 11) and my workstation (fedora 34).
therefore i created 2 heroku app-spaces, one is guarded springs and the other is the afternoon garden.

@openshift-ci openshift-ci bot added the kind/bug Categorizes issue or PR as related to a bug. label May 11, 2021
@damnms damnms closed this as completed May 11, 2021
@damnms damnms reopened this May 11, 2021
@nalind
Copy link
Member

nalind commented May 11, 2021

It looks like we're attempting to push the image in OCI format, and the registry is replying with "unsupported". My guess is that the details it's giving (if it's giving any -- trying again with --log-level=debug might shed some light on that) don't look like what we'd normally interpret as "maybe try again in a different format?" guidance, which would normally trigger an attempt to convert to Docker format on the fly to see if the registry would accept the image that way.

If you remove the image and rebuild it with the "BUILDAH_FORMAT" variable in your environment set to "docker", the built image should start out in Docker format, and if pushing that image succeeds, then we'll know for sure it's a matter of the "try pushing in a different format" logic not being triggered when it should be.

@damnms
Copy link
Author

damnms commented May 11, 2021

[oli@lucy DevOpsPath]$ podman --log-level=debug push localhost/zhaw-mosa/devopspath registry.heroku.com/afternoon-garden-35911/web
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called push.PersistentPreRunE(podman --log-level=debug push localhost/zhaw-mosa/devopspath registry.heroku.com/afternoon-garden-35911/web) 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf" 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /home/oli/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/oli/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /home/oli/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /home/oli/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] Not configuring container store              
DEBU[0000] Initializing event backend journald          
DEBU[0000] configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument 
DEBU[0000] configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
DEBU[0000] Default CNI network name podman is unchangeable 
INFO[0000] Setting parallel job count to 49             
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called push.PersistentPreRunE(podman --log-level=debug push localhost/zhaw-mosa/devopspath registry.heroku.com/afternoon-garden-35911/web) 
DEBU[0000] overlay storage already configured with a mount-program 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf" 
DEBU[0000] overlay storage already configured with a mount-program 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /home/oli/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/oli/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /home/oli/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /home/oli/.local/share/containers/storage/volumes 
DEBU[0000] overlay storage already configured with a mount-program 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=btrfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument 
DEBU[0000] configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
DEBU[0000] Default CNI network name podman is unchangeable 
INFO[0000] Setting parallel job count to 49             
DEBU[0000] Looking up image "localhost/zhaw-mosa/devopspath" in local containers storage 
DEBU[0000] Trying "localhost/zhaw-mosa/devopspath" ...  
DEBU[0000] Trying "localhost/zhaw-mosa/devopspath:latest" ... 
DEBU[0000] parsed reference into "[overlay@/home/oli/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@06678cbeb278f7967cd096e3d8d4d5640bc932b532ec161c9fc3ff055afe212a" 
DEBU[0000] exporting opaque data as blob "sha256:06678cbeb278f7967cd096e3d8d4d5640bc932b532ec161c9fc3ff055afe212a" 
DEBU[0000] Found image "localhost/zhaw-mosa/devopspath" as "localhost/zhaw-mosa/devopspath:latest" in local containers storage ([overlay@/home/oli/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@06678cbeb278f7967cd096e3d8d4d5640bc932b532ec161c9fc3ff055afe212a) 
DEBU[0000] Pushing image localhost/zhaw-mosa/devopspath to registry.heroku.com/afternoon-garden-35911/web 
DEBU[0000] Copying source image [overlay@/home/oli/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@06678cbeb278f7967cd096e3d8d4d5640bc932b532ec161c9fc3ff055afe212a to destination image //registry.heroku.com/afternoon-garden-35911/web:latest 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/000-shortnames.conf" 
DEBU[0000] Found credentials for registry.heroku.com in credential helper containers-auth.json 
DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU[0000]  Using "default-docker" configuration        
DEBU[0000]   Using file:///var/lib/containers/sigstore  
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.heroku.com 
DEBU[0000] Using blob info cache at /home/oli/.local/share/containers/storage/cache/blob-info-cache-v1.boltdb 
DEBU[0000] IsRunningImageAllowed for image containers-storage:[overlay@/home/oli/.local/share/containers/storage]@06678cbeb278f7967cd096e3d8d4d5640bc932b532ec161c9fc3ff055afe212a 
DEBU[0000]  Using default policy section                
DEBU[0000]  Requirement 0: allowed                      
DEBU[0000] Overall: allowed                             
Getting image source signatures
DEBU[0000] Manifest has MIME type application/vnd.oci.image.manifest.v1+json, ordered candidate list [application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.docker.distribution.manifest.v1+json] 
DEBU[0000] ... will first try using the original manifest unmodified 
DEBU[0000] Checking /v2/afternoon-garden-35911/web/blobs/sha256:ae69bc9df648767cb8974730c19b9139f81fcb9ba19df3c772c41b58e976a83b 
DEBU[0000] Checking /v2/afternoon-garden-35911/web/blobs/sha256:9ad5c6bbf51bbb4d90939d440bf48c311b4da6cb4138eb73c3d6ca1ec73fb4d9 
DEBU[0000] GET https://registry.heroku.com/v2/          
DEBU[0000] Checking /v2/afternoon-garden-35911/web/blobs/sha256:ca85d5f63d9df3d9747c29fcb4e88d34ef1caaef151faaf30057b3db4b925d0c 
DEBU[0000] Checking /v2/afternoon-garden-35911/web/blobs/sha256:1e77dd81f9fa12f3034fa1ed55bc2d1f7a316b450ae53f36beb52af2dd83b78f 
DEBU[0000] Checking /v2/afternoon-garden-35911/web/blobs/sha256:030309cad0ba82b098939419dcb5e0a95c77d2427d99c44a690ecab59f80a487 
DEBU[0000] Checking /v2/afternoon-garden-35911/web/blobs/sha256:6f15325cc380f8fc8fa0cdffc5cc7e38c5beb155e09ab3e0edbb1e5a842c46cc 
DEBU[0000] Ping https://registry.heroku.com/v2/ status 401 
DEBU[0000] HEAD https://registry.heroku.com/v2/afternoon-garden-35911/web/blobs/sha256:ae69bc9df648767cb8974730c19b9139f81fcb9ba19df3c772c41b58e976a83b 
DEBU[0000] HEAD https://registry.heroku.com/v2/afternoon-garden-35911/web/blobs/sha256:9ad5c6bbf51bbb4d90939d440bf48c311b4da6cb4138eb73c3d6ca1ec73fb4d9 
DEBU[0000] HEAD https://registry.heroku.com/v2/afternoon-garden-35911/web/blobs/sha256:ca85d5f63d9df3d9747c29fcb4e88d34ef1caaef151faaf30057b3db4b925d0c 
DEBU[0000] HEAD https://registry.heroku.com/v2/afternoon-garden-35911/web/blobs/sha256:1e77dd81f9fa12f3034fa1ed55bc2d1f7a316b450ae53f36beb52af2dd83b78f 
DEBU[0000] HEAD https://registry.heroku.com/v2/afternoon-garden-35911/web/blobs/sha256:030309cad0ba82b098939419dcb5e0a95c77d2427d99c44a690ecab59f80a487 
DEBU[0000] HEAD https://registry.heroku.com/v2/afternoon-garden-35911/web/blobs/sha256:6f15325cc380f8fc8fa0cdffc5cc7e38c5beb155e09ab3e0edbb1e5a842c46cc 
DEBU[0001] ... not present                              
DEBU[0001] Trying to reuse cached location sha256:7a758745be8c71712555423328b627edf74355c6eabb0b6497023d78ff0d1437 compressed with gzip in registry.heroku.com/afternoon-garden-35911/web 
DEBU[0001] Checking /v2/afternoon-garden-35911/web/blobs/sha256:7a758745be8c71712555423328b627edf74355c6eabb0b6497023d78ff0d1437 
DEBU[0001] HEAD https://registry.heroku.com/v2/afternoon-garden-35911/web/blobs/sha256:7a758745be8c71712555423328b627edf74355c6eabb0b6497023d78ff0d1437 
DEBU[0001] ... not present                              
DEBU[0001] Trying to reuse cached location sha256:b3e208ec4920de022c3e222c74e79ae14084ee99d356d040b91d4659d6f93747 compressed with gzip in registry.heroku.com/afternoon-garden-35911/web 
DEBU[0001] Checking /v2/afternoon-garden-35911/web/blobs/sha256:b3e208ec4920de022c3e222c74e79ae14084ee99d356d040b91d4659d6f93747 
DEBU[0001] HEAD https://registry.heroku.com/v2/afternoon-garden-35911/web/blobs/sha256:b3e208ec4920de022c3e222c74e79ae14084ee99d356d040b91d4659d6f93747 
DEBU[0001] ... not present                              
DEBU[0001] Trying to reuse cached location sha256:a30a86627e11d7a794eabc072d925c01438ad6da9f5c5337fa246f3cf22392e9 compressed with gzip in registry.heroku.com/afternoon-garden-35911/web 
DEBU[0001] Checking /v2/afternoon-garden-35911/web/blobs/sha256:a30a86627e11d7a794eabc072d925c01438ad6da9f5c5337fa246f3cf22392e9 
DEBU[0001] HEAD https://registry.heroku.com/v2/afternoon-garden-35911/web/blobs/sha256:a30a86627e11d7a794eabc072d925c01438ad6da9f5c5337fa246f3cf22392e9 
DEBU[0001] ... not present                              
DEBU[0001] Trying to reuse cached location sha256:2eab874737fe4c4a4996b2b12e8ac379bca8703588e63a54297279f3711f1a4d compressed with gzip in registry.heroku.com/afternoon-garden-35911/web 
DEBU[0001] Checking /v2/afternoon-garden-35911/web/blobs/sha256:2eab874737fe4c4a4996b2b12e8ac379bca8703588e63a54297279f3711f1a4d 
DEBU[0001] HEAD https://registry.heroku.com/v2/afternoon-garden-35911/web/blobs/sha256:2eab874737fe4c4a4996b2b12e8ac379bca8703588e63a54297279f3711f1a4d 
DEBU[0001] ... not present                              
DEBU[0001] Trying to reuse cached location sha256:a6ff4aede508e7d435f63d6bbaafd8f5c6841ac4b136871530d0929d7546bc69 compressed with gzip in registry.heroku.com/afternoon-garden-35911/web 
DEBU[0001] Checking /v2/afternoon-garden-35911/web/blobs/sha256:a6ff4aede508e7d435f63d6bbaafd8f5c6841ac4b136871530d0929d7546bc69 
DEBU[0001] HEAD https://registry.heroku.com/v2/afternoon-garden-35911/web/blobs/sha256:a6ff4aede508e7d435f63d6bbaafd8f5c6841ac4b136871530d0929d7546bc69 
DEBU[0001] ... not present                              
DEBU[0001] Trying to reuse cached location sha256:8581af0d7e7379e3d92ac56384a0c197c9425fb360d5b00836e5337419a183ad compressed with gzip in registry.heroku.com/afternoon-garden-35911/web 
DEBU[0001] Checking /v2/afternoon-garden-35911/web/blobs/sha256:8581af0d7e7379e3d92ac56384a0c197c9425fb360d5b00836e5337419a183ad 
DEBU[0001] HEAD https://registry.heroku.com/v2/afternoon-garden-35911/web/blobs/sha256:8581af0d7e7379e3d92ac56384a0c197c9425fb360d5b00836e5337419a183ad 
DEBU[0001] ... already exists                           
DEBU[0001] ... already exists                           
DEBU[0001] ... already exists                           
DEBU[0001] ... already exists                           
DEBU[0001] Skipping blob sha256:030309cad0ba82b098939419dcb5e0a95c77d2427d99c44a690ecab59f80a487 (already present): 
Copying blob 030309cad0ba skipped: already exists  
Copying blob 030309cad0ba skipped: already exists  
Copying blob 030309cad0ba skipped: already exists  
Copying blob ae69bc9df648 skipped: already exists  
Copying blob ca85d5f63d9d skipped: already exists  
Copying blob 030309cad0ba skipped: already exists  
Copying blob ae69bc9df648 skipped: already exists  
Copying blob ca85d5f63d9d skipped: already exists  
Copying blob 9ad5c6bbf51b [--------------------------------------] 0.0b / 0.0b
Copying blob 030309cad0ba skipped: already exists  
Copying blob ae69bc9df648 skipped: already exists  
Copying blob 030309cad0ba skipped: already exists  
Copying blob ae69bc9df648 skipped: already exists  
Copying blob 030309cad0ba skipped: already exists  
Copying blob ae69bc9df648 skipped: already exists  
Copying blob 030309cad0ba skipped: already exists  
Copying blob ae69bc9df648 skipped: already exists  
Copying blob ca85d5f63d9d skipped: already exists  
Copying blob 9ad5c6bbf51b skipped: already exists  
Copying blob 6f15325cc380 skipped: already exists  
Copying blob 1e77dd81f9fa skipped: already exists  
DEBU[0002] ... not present                              
DEBU[0002] Trying to reuse cached location sha256:d5534e229174a467597c304c5f82c36824f1349812f0bd0d309f65dbc58f8f47 compressed with gzip in registry.heroku.com/afternoon-garden-35911/web 
DEBU[0002] Checking /v2/afternoon-garden-35911/web/blobs/sha256:d5534e229174a467597c304c5f82c36824f1349812f0bd0d309f65dbc58f8f47 
DEBU[0002] HEAD https://registry.heroku.com/v2/afternoon-garden-35911/web/blobs/sha256:d5534e229174a467597c304c5f82c36824f1349812f0bd0d309f65dbc58f8f47 
DEBU[0002] ... not present                              
DEBU[0002] Trying to reuse cached location sha256:b387946ca2b14c0e1beee84ea3de94419ce31eecaea94c30b800395599ff30d5 compressed with gzip in registry.heroku.com/afternoon-garden-35911/web 
Copying blob 030309cad0ba skipped: already exists  
Copying blob ae69bc9df648 skipped: already exists  
Copying blob ca85d5f63d9d skipped: already exists  
Copying blob 9ad5c6bbf51b skipped: already exists  
Copying blob 030309cad0ba skipped: already exists  
Copying blob ae69bc9df648 skipped: already exists  
Copying blob 030309cad0ba skipped: already exists  
Copying blob ae69bc9df648 skipped: already exists  
Copying blob 030309cad0ba skipped: already exists  
Copying blob ae69bc9df648 skipped: already exists  
Copying blob 030309cad0ba skipped: already exists  
Copying blob ae69bc9df648 skipped: already exists  
Copying blob 030309cad0ba skipped: already exists  
Copying blob ae69bc9df648 skipped: already exists  
Copying blob ca85d5f63d9d skipped: already exists  
Copying blob 9ad5c6bbf51b skipped: already exists  
Copying blob 6f15325cc380 skipped: already exists  
Copying blob 1e77dd81f9fa skipped: already exists  
Copying blob fe3990b1ba7d skipped: already exists  
Copying blob c22b83219c54 skipped: already exists  
Copying blob 5f70bf18a086 skipped: already exists  
Copying blob 261b20bb325b [--------------------------------------] 0.0b / 0.0b
DEBU[0003] exporting opaque data as blob "sha256:06678cbeb278f7967cd096e3d8d4d5640bc932b532ec161c9fc3ff055afe212a" 
DEBU[0003] No compression detected                      
DEBU[0003] Using original blob without modification     
DEBU[0003] Checking /v2/afternoon-garden-35911/web/blobs/sha256:06678cbeb278f7967cd096e3d8d4d5640bc932b532ec161c9fc3ff055afe212a 
DEBU[0003] HEAD https://registry.heroku.com/v2/afternoon-garden-35911/web/blobs/sha256:06678cbeb278f7967cd096e3d8d4d5640bc932b532ec161c9fc3ff055afe212a 
Copying config 06678cbeb2 [--------------------------------------] 0.0b / 5.5KiB
DEBU[0003] ... already exists                           
Writing manifest to image destination
DEBU[0003] PUT https://registry.heroku.com/v2/afternoon-garden-35911/web/manifests/latest 
DEBU[0004] Writing manifest using preferred type application/vnd.oci.image.manifest.v1+json failed: Error writing manifest "{\"schemaVersion\":2,\"config\":{\"mediaType\":\"application/vnd.oci.image.config.v1+json\",\"digest\":\"sha256:06678cbeb278f7967cd096e3d8d4d5640bc932b532ec161c9fc3ff055afe212a\",\"size\":5590},\"layers\":[{\"mediaType\":\"application/vnd.oci.image.layer.v1.tar+gzip\",\"digest\":\"sha256:2eab874737fe4c4a4996b2b12e8ac379bca8703588e63a54297279f3711f1a4d\",\"size\":27928748},{\"mediaType\":\"application/vnd.oci.image.layer.v1.tar+gzip\",\"digest\":\"sha256:8581af0d7e7379e3d92ac56384a0c197c9425fb360d5b00836e5337419a183ad\",\"size\":921},{\"mediaType\":\"application/vnd.oci.image.layer.v1.tar+gzip\",\"digest\":\"sha256:a6ff4aede508e7d435f63d6bbaafd8f5c6841ac4b136871530d0929d7546bc69\",\"size\":204},{\"mediaType\":\"application/vnd.oci.image.layer.v1.tar+gzip\",\"digest\":\"sha256:b3e208ec4920de022c3e222c74e79ae14084ee99d356d040b91d4659d6f93747\",\"size\":232527432},{\"mediaType\":\"application/vnd.oci.image.layer.v1.tar+gzip\",\"digest\":\"sha256:7a758745be8c71712555423328b627edf74355c6eabb0b6497023d78ff0d1437\",\"size\":86561314},{\"mediaType\":\"application/vnd.oci.image.layer.v1.tar+gzip\",\"digest\":\"sha256:a30a86627e11d7a794eabc072d925c01438ad6da9f5c5337fa246f3cf22392e9\",\"size\":129531783},{\"mediaType\":\"application/vnd.oci.image.layer.v1.tar+gzip\",\"digest\":\"sha256:b387946ca2b14c0e1beee84ea3de94419ce31eecaea94c30b800395599ff30d5\",\"size\":221449469},{\"mediaType\":\"application/vnd.oci.image.layer.v1.tar+gzip\",\"digest\":\"sha256:2a92b839348f3e21e46102995abb833e7edf4c8e7a270a9b91d40668faa09c5f\",\"size\":8380941},{\"mediaType\":\"application/vnd.oci.image.layer.v1.tar+gzip\",\"digest\":\"sha256:4ca545ee6d5db5c1170386eeb39b2ffe3bd46e5d4a73a9acbebc805f19607eb3\",\"size\":42},{\"mediaType\":\"application/vnd.oci.image.layer.v1.tar+gzip\",\"digest\":\"sha256:d5534e229174a467597c304c5f82c36824f1349812f0bd0d309f65dbc58f8f47\",\"size\":386812071}]}": Error uploading manifest latest to registry.heroku.com/afternoon-garden-35911/web: unsupported 
Error: Error writing manifest "{\"schemaVersion\":2,\"config\":{\"mediaType\":\"application/vnd.oci.image.config.v1+json\",\"digest\":\"sha256:06678cbeb278f7967cd096e3d8d4d5640bc932b532ec161c9fc3ff055afe212a\",\"size\":5590},\"layers\":[{\"mediaType\":\"application/vnd.oci.image.layer.v1.tar+gzip\",\"digest\":\"sha256:2eab874737fe4c4a4996b2b12e8ac379bca8703588e63a54297279f3711f1a4d\",\"size\":27928748},{\"mediaType\":\"application/vnd.oci.image.layer.v1.tar+gzip\",\"digest\":\"sha256:8581af0d7e7379e3d92ac56384a0c197c9425fb360d5b00836e5337419a183ad\",\"size\":921},{\"mediaType\":\"application/vnd.oci.image.layer.v1.tar+gzip\",\"digest\":\"sha256:a6ff4aede508e7d435f63d6bbaafd8f5c6841ac4b136871530d0929d7546bc69\",\"size\":204},{\"mediaType\":\"application/vnd.oci.image.layer.v1.tar+gzip\",\"digest\":\"sha256:b3e208ec4920de022c3e222c74e79ae14084ee99d356d040b91d4659d6f93747\",\"size\":232527432},{\"mediaType\":\"application/vnd.oci.image.layer.v1.tar+gzip\",\"digest\":\"sha256:7a758745be8c71712555423328b627edf74355c6eabb0b6497023d78ff0d1437\",\"size\":86561314},{\"mediaType\":\"application/vnd.oci.image.layer.v1.tar+gzip\",\"digest\":\"sha256:a30a86627e11d7a794eabc072d925c01438ad6da9f5c5337fa246f3cf22392e9\",\"size\":129531783},{\"mediaType\":\"application/vnd.oci.image.layer.v1.tar+gzip\",\"digest\":\"sha256:b387946ca2b14c0e1beee84ea3de94419ce31eecaea94c30b800395599ff30d5\",\"size\":221449469},{\"mediaType\":\"application/vnd.oci.image.layer.v1.tar+gzip\",\"digest\":\"sha256:2a92b839348f3e21e46102995abb833e7edf4c8e7a270a9b91d40668faa09c5f\",\"size\":8380941},{\"mediaType\":\"application/vnd.oci.image.layer.v1.tar+gzip\",\"digest\":\"sha256:4ca545ee6d5db5c1170386eeb39b2ffe3bd46e5d4a73a9acbebc805f19607eb3\",\"size\":42},{\"mediaType\":\"application/vnd.oci.image.layer.v1.tar+gzip\",\"digest\":\"sha256:d5534e229174a467597c304c5f82c36824f1349812f0bd0d309f65dbc58f8f47\",\"size\":386812071}]}": Error uploading manifest latest to registry.heroku.com/afternoon-garden-35911/web: unsupported

@damnms
Copy link
Author

damnms commented May 11, 2021

i can confirm that it worked with BUILDAH_FORMAT=docker - thanks!

@rhatdan
Copy link
Member

rhatdan commented May 11, 2021

Please report this as a bug to Heroku, that they should support OCI Images.

@nalind
Copy link
Member

nalind commented May 11, 2021

We may also want to adjust something in the neighborhood of https://github.com/containers/image/blob/56ddc22a8b0b1cef6fc1c5f321285fbde7f4274b/docker/docker_image_dest.go#L473 to get the try-converting-it behavior we'd expect.

@TomSweeneyRedHat
Copy link
Member

If we get the unsupported error message on the push, could we possibly end the text with something like, "The registry believes the image is an unsupported type. Try rebuilding the image using --format=docker and retry the push. "

We seem to keep hitting issues like this every few weeks.

@nalind
Copy link
Member

nalind commented May 11, 2021

Under normal operation, the intent of the code is that if we can determine that a different format would be accepted by the registry, we should be trying an on-the-fly conversion automatically. If we can do that here, that's highly preferable to forcing users to rebuild their images.

@rhatdan
Copy link
Member

rhatdan commented May 11, 2021

So should this be done via Podman or containers/image?

@nalind
Copy link
Member

nalind commented May 11, 2021

That decision gets made in the image library's registry client logic, more or less boiling down to whether https://github.com/containers/image/blob/56ddc22a8b0b1cef6fc1c5f321285fbde7f4274b/docker/docker_image_dest.go#L449 returns true or not.

@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 22, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 22, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Development

No branches or pull requests

4 participants