buildah image - podman pull: unsupported docker v2s2 media type: ""

[uwegeercken]

I followed the buildah tutorial for building an image and pushed the image to my local Nexus v3.22 artifactory - like this:

buildah push --tls-verify=false silent1:8082/alpine-basic docker://silent1:8083/uwe/alpine-basic:latest

The image is visible in Nexus artifactory.

When I now go and remove the image locally and try to pull the image from Nexus

podman pull silent1:8082/uwe/alpine-basic:latest

I get following error:

podman pull silent1:8082/uwe/alpine-basic:latest
Trying to pull silent1:8082/uwe/alpine-basic:latest...
unsupported docker v2s2 media type: ""
Error: error pulling image "silent1:8082/uwe/alpine-basic:latest": unable to pull silent1:8082/uwe/alpine-basic:latest: unable to pull image: Error initializing image from source docker://silent1:8082/uwe/alpine-basic:latest: unsupported docker v2s2 media type: ""

Note: that I can pull images from docker hub using podman and through Nexus like this without a problem:
podman pull silent1:8082/httpd:2.4-alpine

Steps to reproduce the issue:

1. follow Buildah tutorial 1
2. mycontainer=$(buildah from alpine)
3. buildah run $mycontainer -- apk --no-cache add bash
4. buildah config --created-by "uwe" $mycontainer
5. buildah config --author "also uwe" $mycontainer
6. buildah config --label name="alpine-basic" $mycontainer
7. buildah commit $mycontainer silent1:8082/alpine-basic
8. buildah push --tls-verify=false silent1:8082/alpine-basic docker://silent1:8083/uwe/alpine-basic:latest
9. podman pull silent1:8082/uwe/alpine-basic:latest

Describe the results you received:
podman pull command result is indicating a unsupported docker v2s2 media type: ""

Describe the results you expected:
podman would pull the existing image from local Nexus artifactory

Output of rpm -q buildah or apt list buildah:

buildah-1.14.3-1.fc31.x86_64

Output of buildah version:

Version:         1.14.3
Go Version:      go1.13.6
Image Spec:      1.0.1-dev
Runtime Spec:    1.0.1-dev
CNI Spec:        0.4.0
libcni Version:  
image Version:   5.2.1
Git Commit:      
Built:           Thu Jan  1 01:00:00 1970
OS/Arch:         linux/amd64

Output of podman version if reporting a podman build issue:

Version:            1.8.2
RemoteAPI Version:  1
Go Version:         go1.13.6
OS/Arch:            linux/amd64

Output of cat /etc/*release:

Fedora release 31 (Thirty One)

Output of uname -a:

Linux fedbox 5.5.11-200.fc31.x86_64 #1 SMP Mon Mar 23 17:32:43 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Output of cat /etc/containers/storage.conf:

[storage]

driver = "overlay"
runroot = "/var/run/containers/storage"
graphroot = "/var/lib/containers/storage"

[storage.options]
additionalimagestores = [
]
size = ""
override_kernel_check = "true"
mountopt = "nodev,metacopy=on"

[storage.options.thinpool]
ostree_repo = ""
skip_mount_home = "false"

[TomSweeneyRedHat]

I'm not familiar with the repository that you're using, but I'm guessing it's an image formating problem and that it doesn't like OCI based images. Can you try changing your commit command to:

buildah commit --format=docker $mycontainer silent1:8082/alpine-basic

and see if it flows from there?

[uwegeercken]

Thank you Tom! It worked.
I also tried to specify the same format parameter when pushing to Nexus and that worked as well. So that is even more convenient.
buildah push --tls-verify=false --format=docker silent1:8082/alpine-basic-new docker://silent1:8083/alpine-basic-new:latest

And also inspecting with skopeo now works - previously I got the same error message as during pull.

As I am beginning with podman and buildah, this has been of great help to me. Thanks again.

Uwe

[rhatdan]

You should probably report an error with whoever provides your container registry to support OCI Images.

[j-s-3]

Docker OCI support released in Nexus Repository v3.23.0: https://issues.sonatype.org/browse/NEXUS-21087

[rhatdan]

Awesome, @uwegeercken Does it work for you now?

[mattjohnson]

I think there might be a bug here in buildah. In this case, buildah sends Accept headers with application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.docker.distribution.manifest.v1+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.index.v1+json as the different values accepted. Nexus Repository does return a proper manifest with Content-Type of application/vnd.docker.distribution.manifest.v2+json. The OCI distribution-spec doesn't get overly prescriptive but does state The client SHOULD include an Accept header indicating which manifest content types it supports. In a successful response, the Content-Type header will indicate which manifest type is being returned.

Reference: https://github.com/opencontainers/distribution-spec/blob/master/spec.md#pulling-an-image-manifest

[TomSweeneyRedHat]

@mattjohnson I'm going to reopen so we can investigate further.

[gasinvein]

Hello. I believe I've stumbled on this issue, too. Images uploaded buy skopeo to the registry (github packages), can't be pulled buy skopeo or podman with the same error.
Workaround with builah --format=docker works for me, too.

[tsailiming]

Having the same issue with Nexus 3.24 OSS. Workaround is --format=docker.

[rhatdan]

@mtrmac @vrothberg Is this issue we have with Artifactory not properly supporting OCI Images?

[mtrmac]

@rhatdan The original issue was linked above. If there is still a problem / a different problem, it’s not one that we have investigated.

---

I think there might be a bug here in buildah. In this case, buildah sends Accept headers with … as the different values accepted. Nexus Repository does return a proper manifest with Content-Type of application/vnd.docker.distribution.manifest.v2+json. The OCI distribution-spec doesn't get overly prescriptive but does state The client SHOULD include an Accept header indicating which manifest content types it supports. In a successful response, the Content-Type header will indicate which manifest type is being returned.

From this description it’s not clear to me what is going wrong / what you think the behavior should be instead. (Historically, the Nexus bug was that it 1) when asked for a schema2 or OCI manifest, it returned the OCI manifest but a Content-Type header that claimed it was a schema2 manifest, resulting in validation errors, and 2) when uploading a manifest, accepted the OCI image even if it was not able to correctly return it back per 1) ).

[mattjohnson]

There were still some issues around the Accept/Content-Type headers in Nexus 3.24 that should be resolved in 3.25/3.25.1+. It now returns the correct content type based on the manifest that was uploaded.

[sycured]

I've this problem with GitHub Container Registry:

++ buildah from docker.pkg.github.com/sycured/centos-haskell-builder/centos-haskell-builder:latest
Error initializing image from source docker://docker.pkg.github.com/sycured/centos-haskell-builder/centos-haskell-builder:latest: unsupported docker v2s2 media type: ""

This image is built using buildah (https://github.com/sycured/centos-haskell-builder/blob/main/run.sh) and works with Docker

[rhatdan]

Can you report this issue to them? They need to support OCI Images properly.

[sycured]

Can you report this issue to them? They need to support OCI Images properly.

I did it, wait and see

[jcansdale]

Hi @sycured,

Last week GitHub released a new container registry that was developed with ICO compliance in mind. You can find it at https://ghcr.io.

Here is how to migrate your images:
https://docs.github.com/en/packages/getting-started-with-github-container-registry/migrating-to-github-container-registry-for-docker-images

Could you try pushing your images there and see if it works any better?

[sycured]

Hi @jcansdale
Thank you, it works after migrating to ghcr.io

[TomSweeneyRedHat]

@sycured thanks for the issue and the updates. As this is working, I'm going to close this issue.

[daveman1010221]

Getting this same error, pulling an image from ghcr.io:

podman pull ghcr.io/talos-systems/install-cni:v0.1.0-6-gdc32cc8

Trying to pull ghcr.io/talos-systems/install-cni:v0.1.0-6-gdc32cc8...
  unsupported docker v2s2 media type: "application/vnd.oci.image.layer.v1.tar+gzip"
Error: Error initializing image from source docker://ghcr.io/talos-systems/install-cni:v0.1.0-6-gdc32cc8: unsupported docker v2s2 media type: "application/vnd.oci.image.layer.v1.tar+gzip"

This isn't an image I have control over. Other info: podman version 2.2.2-dev.

Thanks.

[rhatdan]

Docker got further, but it did not work either.

$ podman pull ghcr.io/talos-systems/install-cni:v0.1.0-6-gdc32cc8
Trying to pull ghcr.io/talos-systems/install-cni:v0.1.0-6-gdc32cc8...
  unsupported docker v2s2 media type: "application/vnd.oci.image.layer.v1.tar+gzip"
Error: Error initializing image from source docker://ghcr.io/talos-systems/install-cni:v0.1.0-6-gdc32cc8: unsupported docker v2s2 media type: "application/vnd.oci.image.layer.v1.tar+gzip"
$ sudo docker pull ghcr.io/talos-systems/install-cni:v0.1.0-6-gdc32cc8
v0.1.0-6-gdc32cc8: Pulling from talos-systems/install-cni
8e0fc8e49c15: Pull complete 
                     error pulling image configuration: Get https://ghcr.io/v2/talos-systems/install-cni/blobs/sha256:ee4f3da7dd84aec927e52a610e9d1b39d1d1f28ae94a09faab80325b69b8793b: dial tcp 140.82.112.34:443: i/o timeout

[rhatdan]

@mtrmac @vrothberg PTAL ^^

[vrothberg]

Please open a new issue along with some more information. It's very hard to manage issues when they are already closed and potentially different.

The image in question is definitely wrong (see below):

~ $ skopeo inspect --raw docker://ghcr.io/talos-systems/install-cni@sha256:f4141445647db7b51b644f900080ef319a84da9ae9bbbf0ea105410e6853da0d
{
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "schemaVersion": 2,
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "digest": "sha256:ee4f3da7dd84aec927e52a610e9d1b39d1d1f28ae94a09faab80325b69b8793b",
      "size": 432
   },
   "layers": [
      {
         "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
         "digest": "sha256:8e0fc8e49c15c42d23ff08e1a8d0ea99bebb5455babd61567a0ea810ad9f7880",
         "size": 43622567
      }
   ]

It claims to be a Docker image with a Docker config but an OCI layer, which is invalid. So I think that Podman and Docker rightfully reject pulling it.