Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OCI images pushed to dockerhub don't show up and aren't available #1871

Open
johnmcollier opened this issue Aug 29, 2019 · 8 comments

Comments

@johnmcollier
Copy link

@johnmcollier johnmcollier commented Aug 29, 2019

Problem description

When I build and push OCI images using the buildah CLI to dockerhub, they get pushed fine (and can be pulled with buildah pull ...), but they won't show up in the Dockerhub UI, nor can they be pulled by the docker CLI or Kubernetes (using Docker). But Docker does appear to see that the image exists

docker pull:

johns-mbp-3:~ johncollier$ docker pull johncollier/spring-push-test
Using default tag: latest
Error response from daemon: mediaType in manifest should be 'application/vnd.docker.distribution.manifest.v2+json' not ''

buildah pull

sh-4.2# buildah pull johncollier/spring-push-test
Getting image source signatures
Copying blob edf72af6d627 skipped: already exists
Copying blob 8d3eac894db4 skipped: already exists
Copying blob 3e4f86211d23 skipped: already exists
Copying blob d72258b97fd0 skipped: already exists
Copying blob efaa572fa153 skipped: already exists
Copying blob f7277927d38a skipped: already exists
Copying blob 4b9886dd7157 skipped: already exists
Copying blob f81d814c0c2c skipped: already exists
Copying blob 3dcc376f5183 skipped: already exists
Copying config 7765f9ec00 done
Writing manifest to image destination
Storing signatures
7765f9ec00b0adc6cc1469535241b886deaf0b69e16617837d7608627bf6f3da

This seems to be a recent change, as up until yesterday, using the exact same version of buildah, everything worked fine and OCI images could be pushed to DockerHub using buildan and they could be pulled using Kubernetes/Docker (and seen in the DockerHub UI).

I should also note that other Docker registries work perfectly fine without issue (such as Artifactory, OKD's internal registry, etc). I can push images to them using buildah and Kubernetes can pull them perfectly fine.

@johnmcollier johnmcollier changed the title OCI images pushed to dockerhub don't show up. OCI images pushed to dockerhub don't show up and aren't available Aug 29, 2019
@manishtomar

This comment has been minimized.

Copy link
Member

@manishtomar manishtomar commented Aug 29, 2019

@johnmcollier Thanks for the report. Yesterday hub registry was upgraded to latest version that supports OCI images. In your case it looks like until yesterday buildah was pushing docker images and NOT OCI images probably because it detected that OCI support was not there. Yesterday after the upgrade it pushed an OCI image which docker does not support pulling as of now. Hence the following error:

Error response from daemon: mediaType in manifest should be 'application/vnd.docker.distribution.manifest.v2+json' not ''

However we are aware of hub UI not showing tags behind OCI images. We are working on getting this fixed and should release it (hopefully) sometime next week. LMK if you still have questions.

@kedmison

This comment has been minimized.

Copy link

@kedmison kedmison commented Sep 16, 2019

I am getting the same docker pull issue with the recent docker-ce and docker-registry.

# docker pull localhost:5000/kelvin/centos-bashecho
Using default tag: latest
Error response from daemon: mediaType in manifest should be 'application/vnd.docker.distribution.manifest.v2+json' not ''
# docker version
Client: Docker Engine - Community
 Version:           19.03.2
 API version:       1.40
 Go version:        go1.12.8
 Git commit:        6a30dfc
 Built:             Thu Aug 29 05:28:55 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.2
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.8
  Git commit:       6a30dfc
  Built:            Thu Aug 29 05:27:34 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683
@manishtomar

This comment has been minimized.

Copy link
Member

@manishtomar manishtomar commented Sep 16, 2019

@kedmison As I mentioned earlier docker-ce does not support pulling OCI images as of now. It only supports pulling docker images.

@johanbrandhorst

This comment has been minimized.

Copy link

@johanbrandhorst johanbrandhorst commented Oct 20, 2019

This affects any images built by podman since it uses buildah under the hood.

@thaJeztah

This comment has been minimized.

Copy link
Member

@thaJeztah thaJeztah commented Oct 20, 2019

Error response from daemon: mediaType in manifest should be 'application/vnd.docker.distribution.manifest.v2+json' not ''

That problem looks the same as moby/moby#39727, which is fixed by moby/moby#40070, and will be back ported to docker 19.03 for the next (19.03.5) patch release (backport is here: docker/engine#405)

thaJeztah added a commit to thaJeztah/cli that referenced this issue Oct 25, 2019
full diff: moby/moby@b6684a4...a30990b

relevant changes:

- moby/moby#39995 Update containerd binary to v1.2.10
- moby/moby#40001 Update runc to v1.0.0-rc8-92-g84373aaa (CVE-2019-16884)
- moby/moby#39999 bump golang 1.13.1 (CVE-2019-16276)
- moby/moby#40102 bump golang 1.13.3 (CVE-2019-17596)
- moby/moby#39994 homedir: add cgo or osusergo buildtag constraints for unix.
  This is to ensure that users of the homedir package cannot compile statically
  (`CGO_ENABLED=0`) without also setting the `osusergo` build tag.
- moby/moby#39983 builder: remove legacy build's session handling
  This feature was used by docker build --stream and it was kept experimental.
  Users of this endpoint should enable BuildKit anyway by setting Version to BuilderBuildKit.
    - Related: docker#2105 build: remove --stream (was experimental)
- moby/moby #40045 Bump logrus 1.4.2, go-shellwords, mergo, flock, creack/pty,
  golang/gddo, gorilla/mux
- moby/moby#39713 bump containerd and dependencies to v1.3.0
- moby/moby#39987 Add ability to handle index acknowledgment with splunk log driver
- moby/moby#40070 Use ocischema package instead of custom handler
    - relates to moby/moby#39727 Docker 19.03 doesn't support OCI image
    - relates to docker/hub-feedback#1871
    - relates to docker/distribution#3024
- moby/moby#39231 Add support for sending down service Running and Desired task counts
- moby/moby#39822 daemon: Use short libnetwork ID in exec-root

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
thaJeztah added a commit to thaJeztah/cli that referenced this issue Oct 26, 2019
full diff: moby/moby@b6684a4...a09e6e3

relevant changes:

- moby/moby#39995 Update containerd binary to v1.2.10
- moby/moby#40001 Update runc to v1.0.0-rc8-92-g84373aaa (CVE-2019-16884)
- moby/moby#39999 bump golang 1.13.1 (CVE-2019-16276)
- moby/moby#40102 bump golang 1.13.3 (CVE-2019-17596)
- moby/moby#40134 Revert "homedir: add cgo or osusergo buildtag constraints for unix"
    - reverts moby/moby#39994 homedir: add cgo or osusergo buildtag constraints for unix,
      in favor of documenting when to set the `osusergo` build tag. The `osusergo`
      build-flag must be used when compiling a static binary with `cgo` enabled,
      and linking against `glibc`.
- moby/moby#39983 builder: remove legacy build's session handling
  This feature was used by docker build --stream and it was kept experimental.
  Users of this endpoint should enable BuildKit anyway by setting Version to BuilderBuildKit.
    - Related: docker#2105 build: remove --stream (was experimental)
- moby/moby #40045 Bump logrus 1.4.2, go-shellwords, mergo, flock, creack/pty,
  golang/gddo, gorilla/mux
- moby/moby#39713 bump containerd and dependencies to v1.3.0
- moby/moby#39987 Add ability to handle index acknowledgment with splunk log driver
- moby/moby#40070 Use ocischema package instead of custom handler
    - relates to moby/moby#39727 Docker 19.03 doesn't support OCI image
    - relates to docker/hub-feedback#1871
    - relates to docker/distribution#3024
- moby/moby#39231 Add support for sending down service Running and Desired task counts
- moby/moby#39822 daemon: Use short libnetwork ID in exec-root
- moby/moby#39100 Use Microsoft/hcsshim constants and deprecate pkg/system.GetOsVersion()
    - updates/requires Microsoft/hscshim@2226e083fc390003ae5aa8325c3c92789afa0e7a

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
thaJeztah added a commit to thaJeztah/cli that referenced this issue Oct 26, 2019
full diff: moby/moby@b6684a4...a09e6e3

relevant changes:

- moby/moby#39995 Update containerd binary to v1.2.10
- moby/moby#40001 Update runc to v1.0.0-rc8-92-g84373aaa (CVE-2019-16884)
- moby/moby#39999 bump golang 1.13.1 (CVE-2019-16276)
- moby/moby#40102 bump golang 1.13.3 (CVE-2019-17596)
- moby/moby#40134 Revert "homedir: add cgo or osusergo buildtag constraints for unix"
    - reverts moby/moby#39994 homedir: add cgo or osusergo buildtag constraints for unix,
      in favor of documenting when to set the `osusergo` build tag. The `osusergo`
      build-flag must be used when compiling a static binary with `cgo` enabled,
      and linking against `glibc`.
- moby/moby#39983 builder: remove legacy build's session handling
  This feature was used by docker build --stream and it was kept experimental.
  Users of this endpoint should enable BuildKit anyway by setting Version to BuilderBuildKit.
    - Related: docker#2105 build: remove --stream (was experimental)
- moby/moby #40045 Bump logrus 1.4.2, go-shellwords, mergo, flock, creack/pty,
  golang/gddo, gorilla/mux
- moby/moby#39713 bump containerd and dependencies to v1.3.0
- moby/moby#39987 Add ability to handle index acknowledgment with splunk log driver
- moby/moby#40070 Use ocischema package instead of custom handler
    - relates to moby/moby#39727 Docker 19.03 doesn't support OCI image
    - relates to docker/hub-feedback#1871
    - relates to docker/distribution#3024
- moby/moby#39231 Add support for sending down service Running and Desired task counts
- moby/moby#39822 daemon: Use short libnetwork ID in exec-root
- moby/moby#39100 Use Microsoft/hcsshim constants and deprecate pkg/system.GetOsVersion()
    - updates/requires Microsoft/hscshim@2226e083fc390003ae5aa8325c3c92789afa0e7a

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
docker-jenkins pushed a commit to docker/docker-ce that referenced this issue Oct 28, 2019
full diff: moby/moby@b6684a4...a09e6e3

relevant changes:

- moby/moby#39995 Update containerd binary to v1.2.10
- moby/moby#40001 Update runc to v1.0.0-rc8-92-g84373aaa (CVE-2019-16884)
- moby/moby#39999 bump golang 1.13.1 (CVE-2019-16276)
- moby/moby#40102 bump golang 1.13.3 (CVE-2019-17596)
- moby/moby#40134 Revert "homedir: add cgo or osusergo buildtag constraints for unix"
    - reverts moby/moby#39994 homedir: add cgo or osusergo buildtag constraints for unix,
      in favor of documenting when to set the `osusergo` build tag. The `osusergo`
      build-flag must be used when compiling a static binary with `cgo` enabled,
      and linking against `glibc`.
- moby/moby#39983 builder: remove legacy build's session handling
  This feature was used by docker build --stream and it was kept experimental.
  Users of this endpoint should enable BuildKit anyway by setting Version to BuilderBuildKit.
    - Related: #2105 build: remove --stream (was experimental)
- moby/moby #40045 Bump logrus 1.4.2, go-shellwords, mergo, flock, creack/pty,
  golang/gddo, gorilla/mux
- moby/moby#39713 bump containerd and dependencies to v1.3.0
- moby/moby#39987 Add ability to handle index acknowledgment with splunk log driver
- moby/moby#40070 Use ocischema package instead of custom handler
    - relates to moby/moby#39727 Docker 19.03 doesn't support OCI image
    - relates to docker/hub-feedback#1871
    - relates to docker/distribution#3024
- moby/moby#39231 Add support for sending down service Running and Desired task counts
- moby/moby#39822 daemon: Use short libnetwork ID in exec-root
- moby/moby#39100 Use Microsoft/hcsshim constants and deprecate pkg/system.GetOsVersion()
    - updates/requires Microsoft/hscshim@2226e083fc390003ae5aa8325c3c92789afa0e7a

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: 7f6cd64335dc631efaa8204c01f92aa40939073a
Component: cli
@CameronNemo

This comment has been minimized.

Copy link

@CameronNemo CameronNemo commented Dec 4, 2019

Just spent a lot of time trying to deal with this... would really be nice if the docker client could handle OCI images, or even if there were a workaround available.

@hSATAC

This comment has been minimized.

Copy link

@hSATAC hSATAC commented Dec 4, 2019

@CameronNemo I feel you bro

@wipatrick

This comment has been minimized.

Copy link

@wipatrick wipatrick commented Dec 6, 2019

just upgraded to Docker 19.03.5 on my Jetson Nano as pointed out by @thaJeztah and it seems to work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
8 participants
You can’t perform that action at this time.