Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Images on DockerHub fail to import size because they are in schema1 #7706

Closed
liggitt opened this issue Mar 1, 2016 · 25 comments

Comments

Projects
None yet
9 participants
@liggitt
Copy link
Contributor

commented Mar 1, 2016

https://ci.openshift.redhat.com/jenkins/job/merge_pull_requests_origin/5142/consoleFull

--- FAIL: TestImageStreamImport-2 (6.29s)
    imageimporter_test.go:105: unexpected image output: (*api.Image)(nil)
FAIL
I0229 17:38:57.818710   25915 importer.go:308] importing remote Docker repository registry=https://registry-1.docker.io repository=library/mongo insecure=false
I0229 17:38:57.820419   25915 credentials.go:24] asked to provide Basic credentials for https://auth.docker.io/token
I0229 17:38:58.056945   25915 importer.go:352] unable to access tags for repository &importer.importRepository{Ref:api.DockerImageReference{Registry:"", Namespace:"", Name:"mongo", Tag:"", ID:""}, Registry:(*url.URL)(0xc20807a580), Name:"library/mongo", Insecure:false, Tags:[]importer.importTag(nil), Digests:[]importer.importDigest(nil), MaximumTags:3, AdditionalTags:[]string(nil), Err:error(nil)}: &url.Error{Op:"Get", URL:"https://registry-1.docker.io/v2/library/mongo/tags/list", Err:(*errors.errorString)(0xc2087ad360)}
--- FAIL: TestImageStreamImportDockerHub-2 (1.80s)
    imageimporter_test.go:523: unexpected repository: &api.RepositoryImportStatus{Status:unversioned.Status{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, ListMeta:unversioned.ListMeta{SelfLink:"", ResourceVersion:""}, Status:"Failure", Message:"Internal error occurred: Get https://registry-1.docker.io/v2/library/mongo/tags/list: token auth attempt for registry: https://auth.docker.io/token?scope=repository%3Alibrary%2Fmongo%3Apull&service=registry.docker.io request failed with status: 500 Internal Server Error", Reason:"InternalError", Details:(*unversioned.StatusDetails)(0xc2087d7c20), Code:500}, Images:[]api.ImageImportStatus(nil), AdditionalTags:[]string(nil)}
    imageimporter_test.go:530: unexpected object: (*api.Image)(nil)
    imageimporter_test.go:534: unexpected object: (*api.Image)(nil)
    imageimporter_test.go:538: unexpected object: (*api.Image)(nil)
    imageimporter_test.go:542: unexpected object: api.ImageImportStatus{Tag:"", Status:unversioned.Status{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, ListMeta:unversioned.ListMeta{SelfLink:"", ResourceVersion:""}, Status:"Failure", Message:"Internal error occurred: Get https://registry-1.docker.io/v2/mysql/doesnotexistinanyform/manifests/latest: token auth attempt for registry: https://auth.docker.io/token?scope=repository%3Amysql%2Fdoesnotexistinanyform%3Apull&service=registry.docker.io request failed with status: 500 Internal Server Error", Reason:"InternalError", Details:(*unversioned.StatusDetails)(0xc20897f3b0), Code:500}, Image:(*api.Image)(nil)}
FAIL
@smarterclayton

This comment has been minimized.

Copy link
Member

commented Apr 22, 2016

Not related to #8558 - will need a separate fix.

@kargakis

This comment has been minimized.

Copy link
Contributor

commented Jun 1, 2016

https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin_integration/1357/consoleFull

--- FAIL: TestImageStreamImport (4.95s)
    imageimporter_test.go:107: unexpected image output: &api.Image{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"sha256:54f2667d0acfc237e16957d3475d769baddceb79fd2a5d467e101cd8c4dde845", GenerateName:"", Namespace:"", SelfLink:"", UID:"ad86f2a6-27e9-11e6-9672-0e53f1960197", ResourceVersion:"224", Generation:0, CreationTimestamp:unversioned.Time{Time:time.Time{sec:63600376327, nsec:0, loc:(*time.Location)(0x5660680)}}, DeletionTimestamp:(*unversioned.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil)}, DockerImageReference:"redis@sha256:54f2667d0acfc237e16957d3475d769baddceb79fd2a5d467e101cd8c4dde845", DockerImageMetadata:api.DockerImage{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, ID:"0d40ae064933f3dbf69b212d5d840232558254d7df883febbb8b5434eb2ba5a5", Parent:"891a493d9ac30e888d39bb68024737bc55d953a917b280ba983384144fb50b7f", Comment:"", Created:unversioned.Time{Time:time.Time{sec:63599671923, nsec:0, loc:(*time.Location)(0x5660680)}}, Container:"6f515d06d10a2a3ed7682d7634f9764a1e43e1e05d4ea24d65cefce01561ce5a", ContainerConfig:api.DockerConfig{Hostname:"b0cf605c7757", Domainname:"", User:"", Memory:0, MemorySwap:0, CPUShares:0, CPUSet:"", AttachStdin:false, AttachStdout:false, AttachStderr:false, PortSpecs:[]string(nil), ExposedPorts:map[string]struct {}{"6379/tcp":struct {}{}}, Tty:false, OpenStdin:false, StdinOnce:false, Env:[]string{"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "GOSU_VERSION=1.7", "REDIS_VERSION=3.2.0", "REDIS_DOWNLOAD_URL=http://download.redis.io/releases/redis-3.2.0.tar.gz", "REDIS_DOWNLOAD_SHA1=0c1820931094369c8cc19fc1be62f598bc5961ca"}, Cmd:[]string{"/bin/sh", "-c", "#(nop) CMD [\"redis-server\"]"}, DNS:[]string(nil), Image:"c5def41d09ea6e43610485683115fded28d9cbca2b36218d3ce86dc76373b6fa", Volumes:map[string]struct {}{"/data":struct {}{}}, VolumesFrom:"", WorkingDir:"/data", Entrypoint:[]string{"docker-entrypoint.sh"}, NetworkDisabled:false, SecurityOpts:[]string(nil), OnBuild:[]string(nil), Labels:map[string]string(nil)}, DockerVersion:"1.9.1", Author:"", Config:(*api.DockerConfig)(0xc824de7e40), Architecture:"amd64", Size:0}, DockerImageMetadataVersion:"1.0", DockerImageManifest:"", DockerImageLayers:[]api.ImageLayer{api.ImageLayer{Name:"sha256:51f5c6a04d83efd2d45c5fd59537218924bc46705e3de6ffc8bc07b51481610b", Size:0}, api.ImageLayer{Name:"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4", Size:0}, api.ImageLayer{Name:"sha256:647d1f6838b59bca027c7bb93da09ed499c5d085a300b4cddf82c8c3372578d6", Size:0}, api.ImageLayer{Name:"sha256:8152112cb8fbc3a56a6f6dfe4ae17c50112494ef1b1c3db12abeae2a99083304", Size:0}, api.ImageLayer{Name:"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4", Size:0}, api.ImageLayer{Name:"sha256:eba38d612e23ffe3e6e8fedfad525cca3c1795a9f6548bc3f2cc45b68fe23d82", Size:0}, api.ImageLayer{Name:"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4", Size:0}, api.ImageLayer{Name:"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4", Size:0}, api.ImageLayer{Name:"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4", Size:0}, api.ImageLayer{Name:"sha256:dbe778655086b0a3be0972d08395c7b3f4ff66e0730d7964f467b6fea241803e", Size:0}, api.ImageLayer{Name:"sha256:fec3d49dbc45340f713e49dfc9fdb020f8ed6d08c6ec4b6c0590da2c75897ba2", Size:0}, api.ImageLayer{Name:"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4", Size:0}, api.ImageLayer{Name:"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4", Size:0}, api.ImageLayer{Name:"sha256:78fc0c16bb0ce4476aed667f8c01d8f09feca620b4039ac308bf69423e1500bf", Size:0}, api.ImageLayer{Name:"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4", Size:0}, api.ImageLayer{Name:"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4", Size:0}, api.ImageLayer{Name:"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4", Size:0}}}
FAIL

Been hitting this all day today

@kargakis

This comment has been minimized.

Copy link
Contributor

commented Jun 1, 2016

We should stop using DockerHub in our tests.

@stevekuznetsov

This comment has been minimized.

Copy link
Contributor

commented Jun 1, 2016

Seeing similar (??) failures on recent runs ... moving to P0 as the last 20 test_integration runs have failed on it.

--- FAIL: TestImageStreamImport (4.94s)
    imageimporter_test.go:107: unexpected image output: &api.Image{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"sha256:54f2667d0acfc237e16957d3475d769baddceb79fd2a5d467e101cd8c4dde845", GenerateName:"", Namespace:"", SelfLink:"", UID:"e9d6aa2c-27f3-11e6-b00d-0e95339e1311", ResourceVersion:"226", Generation:0, CreationTimestamp:unversioned.Time{Time:time.Time{sec:63600380723, nsec:0, loc:(*time.Location)(0x565c680)}}, DeletionTimestamp:(*unversioned.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil)}, DockerImageReference:"redis@sha256:54f2667d0acfc237e16957d3475d769baddceb79fd2a5d467e101cd8c4dde845", DockerImageMetadata:api.DockerImage{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, ID:"0d40ae064933f3dbf69b212d5d840232558254d7df883febbb8b5434eb2ba5a5", Parent:"891a493d9ac30e888d39bb68024737bc55d953a917b280ba983384144fb50b7f", Comment:"", Created:unversioned.Time{Time:time.Time{sec:63599671923, nsec:0, loc:(*time.Location)(0x565c680)}}, Container:"6f515d06d10a2a3ed7682d7634f9764a1e43e1e05d4ea24d65cefce01561ce5a", ContainerConfig:api.DockerConfig{Hostname:"b0cf605c7757", Domainname:"", User:"", Memory:0, MemorySwap:0, CPUShares:0, CPUSet:"", AttachStdin:false, AttachStdout:false, AttachStderr:false, PortSpecs:[]string(nil), ExposedPorts:map[string]struct {}{"6379/tcp":struct {}{}}, Tty:false, OpenStdin:false, StdinOnce:false, Env:[]string{"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "GOSU_VERSION=1.7", "REDIS_VERSION=3.2.0", "REDIS_DOWNLOAD_URL=http://download.redis.io/releases/redis-3.2.0.tar.gz", "REDIS_DOWNLOAD_SHA1=0c1820931094369c8cc19fc1be62f598bc5961ca"}, Cmd:[]string{"/bin/sh", "-c", "#(nop) CMD [\"redis-server\"]"}, DNS:[]string(nil), Image:"c5def41d09ea6e43610485683115fded28d9cbca2b36218d3ce86dc76373b6fa", Volumes:map[string]struct {}{"/data":struct {}{}}, VolumesFrom:"", WorkingDir:"/data", Entrypoint:[]string{"docker-entrypoint.sh"}, NetworkDisabled:false, SecurityOpts:[]string(nil), OnBuild:[]string(nil), Labels:map[string]string(nil)}, DockerVersion:"1.9.1", Author:"", Config:(*api.DockerConfig)(0xc8239411e0), Architecture:"amd64", Size:0}, DockerImageMetadataVersion:"1.0", DockerImageManifest:"", DockerImageLayers:[]api.ImageLayer{api.ImageLayer{Name:"sha256:51f5c6a04d83efd2d45c5fd59537218924bc46705e3de6ffc8bc07b51481610b", Size:0}, api.ImageLayer{Name:"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4", Size:0}, api.ImageLayer{Name:"sha256:647d1f6838b59bca027c7bb93da09ed499c5d085a300b4cddf82c8c3372578d6", Size:0}, api.ImageLayer{Name:"sha256:8152112cb8fbc3a56a6f6dfe4ae17c50112494ef1b1c3db12abeae2a99083304", Size:0}, api.ImageLayer{Name:"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4", Size:0}, api.ImageLayer{Name:"sha256:eba38d612e23ffe3e6e8fedfad525cca3c1795a9f6548bc3f2cc45b68fe23d82", Size:0}, api.ImageLayer{Name:"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4", Size:0}, api.ImageLayer{Name:"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4", Size:0}, api.ImageLayer{Name:"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4", Size:0}, api.ImageLayer{Name:"sha256:dbe778655086b0a3be0972d08395c7b3f4ff66e0730d7964f467b6fea241803e", Size:0}, api.ImageLayer{Name:"sha256:fec3d49dbc45340f713e49dfc9fdb020f8ed6d08c6ec4b6c0590da2c75897ba2", Size:0}, api.ImageLayer{Name:"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4", Size:0}, api.ImageLayer{Name:"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4", Size:0}, api.ImageLayer{Name:"sha256:78fc0c16bb0ce4476aed667f8c01d8f09feca620b4039ac308bf69423e1500bf", Size:0}, api.ImageLayer{Name:"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4", Size:0}, api.ImageLayer{Name:"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4", Size:0}, api.ImageLayer{Name:"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4", Size:0}}}
@deads2k

This comment has been minimized.

Copy link
Contributor

commented Jun 1, 2016

We should stop using DockerHub in our tests.

Not all tests can run disconnected from DockerHub. Being able to inter-operate with DockerHub is one of our core use-cases, so we will always end up having some level of test dependency on them and given how critical the path is, I think its reasonable to block merge on them.

I'm not against trying to make it more resilient to transient failures, but I'm far less inclined to remove the dependency entirely. Imagine if we had a new-app bug that prevented it from working against the docker hub.

@deads2k

This comment has been minimized.

Copy link
Contributor

commented Jun 1, 2016

Looks like the reported image size is wrong. Check to see if dockerhub stopped reporting the field.

@kargakis

This comment has been minimized.

Copy link
Contributor

commented Jun 1, 2016

@deads2k we can consolidate the tests we need to run against DockerHub. Also, it would be nice if we wouldn't run those tests, when pulls don't affect things we need to work against DockerHub AND DockerHub is down at the same time.

@stevekuznetsov

This comment has been minimized.

Copy link
Contributor

commented Jun 1, 2016

Kind of the whole point of running the full suite always is that the program is incredibly complex and we can't reason about how any one change could fan out to the rest of the software, so I don't think it's ideologically sound to say that we can, for any pull, determine if it has no chance of affecting image import.

@smarterclayton

This comment has been minimized.

Copy link
Member

commented Jun 1, 2016

DockerHub switched from schema 1 to schema 2 today, @miminar

@legionus

This comment has been minimized.

Copy link
Contributor

commented Jun 1, 2016

We use v1Compatibility to calculate size of each layer:

https://github.com/openshift/origin/blob/master/pkg/image/api/helper.go#L366-L373

but I think Size is no longer there:

https://github.com/docker/distribution/blob/master/manifest/schema1/config_builder.go#L105-L115

That's why we have layers, but their size is zero as the size of the metadata.

@smarterclayton

This comment has been minimized.

Copy link
Member

commented Jun 1, 2016

Can we patch quickly to work with v2 for Size, or do we need to wait for the 2.4 registry update?

@legionus

This comment has been minimized.

Copy link
Contributor

commented Jun 1, 2016

Can we patch quickly to work with v2 for Size

We can't change the manifest, because it will change the digest of manifest.

do we need to wait for the 2.4 registry

This update doesn't help because schema v1 is affected.

We need to calculate the Size in different way. In same way as dockerregistry does:

https://github.com/openshift/origin/blob/master/pkg/dockerregistry/server/repositorymiddleware.go#L427-L443

@legionus

This comment has been minimized.

Copy link
Contributor

commented Jun 1, 2016

I guess I can fix it.

@smarterclayton

This comment has been minimized.

Copy link
Member

commented Jun 1, 2016

That's what I'm referring to - calculating size differently on import by
adding the schema2 fields to the schema1 object (which we already partially
do)

On Wed, Jun 1, 2016 at 11:21 AM, Alexey Gladkov notifications@github.com
wrote:

I guess I can fix it.


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#7706 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/ABG_py3c5zWOQuZOgz2MTw3kJI0cVwfbks5qHaNogaJpZM4HmE1K
.

@smarterclayton

This comment has been minimized.

Copy link
Member

commented Jun 1, 2016

Can I get a commit from someone to either put a fix in or work around this by 2pm EDT? I'm ok with disabling the test condition that's failing as long as someone is committed to a fix.

This is something we may have to backport to 1.2.0 and earlier so that people can actually use us.

@kargakis

This comment has been minimized.

Copy link
Contributor

commented Jun 1, 2016

@legionus has a fix locally

@smarterclayton

This comment has been minimized.

Copy link
Member

commented Jun 1, 2016

Thanks, please keep this issue updated and be sure to notify people on the
mailing lists that the queue is broken.

On Wed, Jun 1, 2016 at 12:31 PM, Michail Kargakis notifications@github.com
wrote:

@legionus https://github.com/legionus has a fix locally


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#7706 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/ABG_pzBD4lO1zX6kmdqTVhwSxPbsDJbgks5qHbPigaJpZM4HmE1K
.

@legionus legionus assigned smarterclayton and unassigned legionus Jun 1, 2016

@smarterclayton

This comment has been minimized.

Copy link
Member

commented Jun 1, 2016

For 1.3 we need to make the transition to schema2 (well under way) and then be sure we import the new size.

@legionus

This comment has been minimized.

Copy link
Contributor

commented Jun 1, 2016

@smarterclayton I'm not sure I understand.
Cc: @miminar

@smarterclayton

This comment has been minimized.

Copy link
Member

commented Jun 1, 2016

We need to be sure that we import via schema2 by default, and fall back to schema1 only if necessary.

@smarterclayton smarterclayton changed the title TestImageStreamImportDockerHub / TestImageStreamImport Images on DockerHub fail to import size because they are in schema1 Jun 1, 2016

@smarterclayton

This comment has been minimized.

Copy link
Member

commented Jun 2, 2016

Summary:

  1. When the new DockerHub (and docker/distribution 2.4+) receive a manifest pushed from docker 1.10 (which will be schema2), a client that requests the manifest without specifying wanting schema2 will get back a schema1 manifest that has no v1Compatibility data containing size
  2. Clients who only push and pull from Docker 1.9 will still be sending schema1 manifests, which will contain the size data
  3. When we upgrade to using the newer registry client in 2.4, we'll start asking for schema2 entries from the Hub and get back the layer size, which we'll need to copy into our internal DockerImageMetadata section when the image is imported
  4. For now, we'll return zero size layers and images via our API when fetching such objects
@deads2k

This comment has been minimized.

Copy link
Contributor

commented Aug 25, 2016

I've seen this in 5 of the last 6 test runs, the sixth failed on yum. Bumping to P1.

@smarterclayton @mfojtik @miminar

@deads2k deads2k added priority/P1 and removed priority/P2 labels Aug 25, 2016

@deads2k

This comment has been minimized.

@smarterclayton

This comment has been minimized.

Copy link
Member

commented Aug 25, 2016

@mfojtik

This comment has been minimized.

Copy link
Member

commented Aug 25, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.