Images on DockerHub fail to import size because they are in schema1

[liggitt]

https://ci.openshift.redhat.com/jenkins/job/merge_pull_requests_origin/5142/consoleFull

--- FAIL: TestImageStreamImport-2 (6.29s)
    imageimporter_test.go:105: unexpected image output: (*api.Image)(nil)
FAIL
I0229 17:38:57.818710   25915 importer.go:308] importing remote Docker repository registry=https://registry-1.docker.io repository=library/mongo insecure=false
I0229 17:38:57.820419   25915 credentials.go:24] asked to provide Basic credentials for https://auth.docker.io/token
I0229 17:38:58.056945   25915 importer.go:352] unable to access tags for repository &importer.importRepository{Ref:api.DockerImageReference{Registry:"", Namespace:"", Name:"mongo", Tag:"", ID:""}, Registry:(*url.URL)(0xc20807a580), Name:"library/mongo", Insecure:false, Tags:[]importer.importTag(nil), Digests:[]importer.importDigest(nil), MaximumTags:3, AdditionalTags:[]string(nil), Err:error(nil)}: &url.Error{Op:"Get", URL:"https://registry-1.docker.io/v2/library/mongo/tags/list", Err:(*errors.errorString)(0xc2087ad360)}
--- FAIL: TestImageStreamImportDockerHub-2 (1.80s)
    imageimporter_test.go:523: unexpected repository: &api.RepositoryImportStatus{Status:unversioned.Status{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, ListMeta:unversioned.ListMeta{SelfLink:"", ResourceVersion:""}, Status:"Failure", Message:"Internal error occurred: Get https://registry-1.docker.io/v2/library/mongo/tags/list: token auth attempt for registry: https://auth.docker.io/token?scope=repository%3Alibrary%2Fmongo%3Apull&service=registry.docker.io request failed with status: 500 Internal Server Error", Reason:"InternalError", Details:(*unversioned.StatusDetails)(0xc2087d7c20), Code:500}, Images:[]api.ImageImportStatus(nil), AdditionalTags:[]string(nil)}
    imageimporter_test.go:530: unexpected object: (*api.Image)(nil)
    imageimporter_test.go:534: unexpected object: (*api.Image)(nil)
    imageimporter_test.go:538: unexpected object: (*api.Image)(nil)
    imageimporter_test.go:542: unexpected object: api.ImageImportStatus{Tag:"", Status:unversioned.Status{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, ListMeta:unversioned.ListMeta{SelfLink:"", ResourceVersion:""}, Status:"Failure", Message:"Internal error occurred: Get https://registry-1.docker.io/v2/mysql/doesnotexistinanyform/manifests/latest: token auth attempt for registry: https://auth.docker.io/token?scope=repository%3Amysql%2Fdoesnotexistinanyform%3Apull&service=registry.docker.io request failed with status: 500 Internal Server Error", Reason:"InternalError", Details:(*unversioned.StatusDetails)(0xc20897f3b0), Code:500}, Image:(*api.Image)(nil)}
FAIL

[smarterclayton]

Not related to #8558 - will need a separate fix.

[0xmichalis]

https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin_integration/1357/consoleFull

--- FAIL: TestImageStreamImport (4.95s)
    imageimporter_test.go:107: unexpected image output: &api.Image{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"sha256:54f2667d0acfc237e16957d3475d769baddceb79fd2a5d467e101cd8c4dde845", GenerateName:"", Namespace:"", SelfLink:"", UID:"ad86f2a6-27e9-11e6-9672-0e53f1960197", ResourceVersion:"224", Generation:0, CreationTimestamp:unversioned.Time{Time:time.Time{sec:63600376327, nsec:0, loc:(*time.Location)(0x5660680)}}, DeletionTimestamp:(*unversioned.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil)}, DockerImageReference:"redis@sha256:54f2667d0acfc237e16957d3475d769baddceb79fd2a5d467e101cd8c4dde845", DockerImageMetadata:api.DockerImage{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, ID:"0d40ae064933f3dbf69b212d5d840232558254d7df883febbb8b5434eb2ba5a5", Parent:"891a493d9ac30e888d39bb68024737bc55d953a917b280ba983384144fb50b7f", Comment:"", Created:unversioned.Time{Time:time.Time{sec:63599671923, nsec:0, loc:(*time.Location)(0x5660680)}}, Container:"6f515d06d10a2a3ed7682d7634f9764a1e43e1e05d4ea24d65cefce01561ce5a", ContainerConfig:api.DockerConfig{Hostname:"b0cf605c7757", Domainname:"", User:"", Memory:0, MemorySwap:0, CPUShares:0, CPUSet:"", AttachStdin:false, AttachStdout:false, AttachStderr:false, PortSpecs:[]string(nil), ExposedPorts:map[string]struct {}{"6379/tcp":struct {}{}}, Tty:false, OpenStdin:false, StdinOnce:false, Env:[]string{"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "GOSU_VERSION=1.7", "REDIS_VERSION=3.2.0", "REDIS_DOWNLOAD_URL=http://download.redis.io/releases/redis-3.2.0.tar.gz", "REDIS_DOWNLOAD_SHA1=0c1820931094369c8cc19fc1be62f598bc5961ca"}, Cmd:[]string{"/bin/sh", "-c", "#(nop) CMD [\"redis-server\"]"}, DNS:[]string(nil), Image:"c5def41d09ea6e43610485683115fded28d9cbca2b36218d3ce86dc76373b6fa", Volumes:map[string]struct {}{"/data":struct {}{}}, VolumesFrom:"", WorkingDir:"/data", Entrypoint:[]string{"docker-entrypoint.sh"}, NetworkDisabled:false, SecurityOpts:[]string(nil), OnBuild:[]string(nil), Labels:map[string]string(nil)}, DockerVersion:"1.9.1", Author:"", Config:(*api.DockerConfig)(0xc824de7e40), Architecture:"amd64", Size:0}, DockerImageMetadataVersion:"1.0", DockerImageManifest:"", DockerImageLayers:[]api.ImageLayer{api.ImageLayer{Name:"sha256:51f5c6a04d83efd2d45c5fd59537218924bc46705e3de6ffc8bc07b51481610b", Size:0}, api.ImageLayer{Name:"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4", Size:0}, api.ImageLayer{Name:"sha256:647d1f6838b59bca027c7bb93da09ed499c5d085a300b4cddf82c8c3372578d6", Size:0}, api.ImageLayer{Name:"sha256:8152112cb8fbc3a56a6f6dfe4ae17c50112494ef1b1c3db12abeae2a99083304", Size:0}, api.ImageLayer{Name:"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4", Size:0}, api.ImageLayer{Name:"sha256:eba38d612e23ffe3e6e8fedfad525cca3c1795a9f6548bc3f2cc45b68fe23d82", Size:0}, api.ImageLayer{Name:"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4", Size:0}, api.ImageLayer{Name:"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4", Size:0}, api.ImageLayer{Name:"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4", Size:0}, api.ImageLayer{Name:"sha256:dbe778655086b0a3be0972d08395c7b3f4ff66e0730d7964f467b6fea241803e", Size:0}, api.ImageLayer{Name:"sha256:fec3d49dbc45340f713e49dfc9fdb020f8ed6d08c6ec4b6c0590da2c75897ba2", Size:0}, api.ImageLayer{Name:"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4", Size:0}, api.ImageLayer{Name:"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4", Size:0}, api.ImageLayer{Name:"sha256:78fc0c16bb0ce4476aed667f8c01d8f09feca620b4039ac308bf69423e1500bf", Size:0}, api.ImageLayer{Name:"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4", Size:0}, api.ImageLayer{Name:"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4", Size:0}, api.ImageLayer{Name:"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4", Size:0}}}
FAIL

Been hitting this all day today

[0xmichalis]

We should stop using DockerHub in our tests.

[stevekuznetsov]

Seeing similar (??) failures on recent runs ... moving to P0 as the last 20 test_integration runs have failed on it.

--- FAIL: TestImageStreamImport (4.94s)
    imageimporter_test.go:107: unexpected image output: &api.Image{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"sha256:54f2667d0acfc237e16957d3475d769baddceb79fd2a5d467e101cd8c4dde845", GenerateName:"", Namespace:"", SelfLink:"", UID:"e9d6aa2c-27f3-11e6-b00d-0e95339e1311", ResourceVersion:"226", Generation:0, CreationTimestamp:unversioned.Time{Time:time.Time{sec:63600380723, nsec:0, loc:(*time.Location)(0x565c680)}}, DeletionTimestamp:(*unversioned.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil)}, DockerImageReference:"redis@sha256:54f2667d0acfc237e16957d3475d769baddceb79fd2a5d467e101cd8c4dde845", DockerImageMetadata:api.DockerImage{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, ID:"0d40ae064933f3dbf69b212d5d840232558254d7df883febbb8b5434eb2ba5a5", Parent:"891a493d9ac30e888d39bb68024737bc55d953a917b280ba983384144fb50b7f", Comment:"", Created:unversioned.Time{Time:time.Time{sec:63599671923, nsec:0, loc:(*time.Location)(0x565c680)}}, Container:"6f515d06d10a2a3ed7682d7634f9764a1e43e1e05d4ea24d65cefce01561ce5a", ContainerConfig:api.DockerConfig{Hostname:"b0cf605c7757", Domainname:"", User:"", Memory:0, MemorySwap:0, CPUShares:0, CPUSet:"", AttachStdin:false, AttachStdout:false, AttachStderr:false, PortSpecs:[]string(nil), ExposedPorts:map[string]struct {}{"6379/tcp":struct {}{}}, Tty:false, OpenStdin:false, StdinOnce:false, Env:[]string{"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "GOSU_VERSION=1.7", "REDIS_VERSION=3.2.0", "REDIS_DOWNLOAD_URL=http://download.redis.io/releases/redis-3.2.0.tar.gz", "REDIS_DOWNLOAD_SHA1=0c1820931094369c8cc19fc1be62f598bc5961ca"}, Cmd:[]string{"/bin/sh", "-c", "#(nop) CMD [\"redis-server\"]"}, DNS:[]string(nil), Image:"c5def41d09ea6e43610485683115fded28d9cbca2b36218d3ce86dc76373b6fa", Volumes:map[string]struct {}{"/data":struct {}{}}, VolumesFrom:"", WorkingDir:"/data", Entrypoint:[]string{"docker-entrypoint.sh"}, NetworkDisabled:false, SecurityOpts:[]string(nil), OnBuild:[]string(nil), Labels:map[string]string(nil)}, DockerVersion:"1.9.1", Author:"", Config:(*api.DockerConfig)(0xc8239411e0), Architecture:"amd64", Size:0}, DockerImageMetadataVersion:"1.0", DockerImageManifest:"", DockerImageLayers:[]api.ImageLayer{api.ImageLayer{Name:"sha256:51f5c6a04d83efd2d45c5fd59537218924bc46705e3de6ffc8bc07b51481610b", Size:0}, api.ImageLayer{Name:"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4", Size:0}, api.ImageLayer{Name:"sha256:647d1f6838b59bca027c7bb93da09ed499c5d085a300b4cddf82c8c3372578d6", Size:0}, api.ImageLayer{Name:"sha256:8152112cb8fbc3a56a6f6dfe4ae17c50112494ef1b1c3db12abeae2a99083304", Size:0}, api.ImageLayer{Name:"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4", Size:0}, api.ImageLayer{Name:"sha256:eba38d612e23ffe3e6e8fedfad525cca3c1795a9f6548bc3f2cc45b68fe23d82", Size:0}, api.ImageLayer{Name:"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4", Size:0}, api.ImageLayer{Name:"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4", Size:0}, api.ImageLayer{Name:"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4", Size:0}, api.ImageLayer{Name:"sha256:dbe778655086b0a3be0972d08395c7b3f4ff66e0730d7964f467b6fea241803e", Size:0}, api.ImageLayer{Name:"sha256:fec3d49dbc45340f713e49dfc9fdb020f8ed6d08c6ec4b6c0590da2c75897ba2", Size:0}, api.ImageLayer{Name:"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4", Size:0}, api.ImageLayer{Name:"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4", Size:0}, api.ImageLayer{Name:"sha256:78fc0c16bb0ce4476aed667f8c01d8f09feca620b4039ac308bf69423e1500bf", Size:0}, api.ImageLayer{Name:"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4", Size:0}, api.ImageLayer{Name:"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4", Size:0}, api.ImageLayer{Name:"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4", Size:0}}}

[deads2k]

We should stop using DockerHub in our tests.

Not all tests can run disconnected from DockerHub. Being able to inter-operate with DockerHub is one of our core use-cases, so we will always end up having some level of test dependency on them and given how critical the path is, I think its reasonable to block merge on them.

I'm not against trying to make it more resilient to transient failures, but I'm far less inclined to remove the dependency entirely. Imagine if we had a new-app bug that prevented it from working against the docker hub.

[deads2k]

Looks like the reported image size is wrong. Check to see if dockerhub stopped reporting the field.

[0xmichalis]

@deads2k we can consolidate the tests we need to run against DockerHub. Also, it would be nice if we wouldn't run those tests, when pulls don't affect things we need to work against DockerHub AND DockerHub is down at the same time.

[stevekuznetsov]

Kind of the whole point of running the full suite always is that the program is incredibly complex and we can't reason about how any one change could fan out to the rest of the software, so I don't think it's ideologically sound to say that we can, for any pull, determine if it has no chance of affecting image import.

[smarterclayton]

DockerHub switched from schema 1 to schema 2 today, @miminar

[legionus]

We use v1Compatibility to calculate size of each layer:

https://github.com/openshift/origin/blob/master/pkg/image/api/helper.go#L366-L373

but I think Size is no longer there:

https://github.com/docker/distribution/blob/master/manifest/schema1/config_builder.go#L105-L115

That's why we have layers, but their size is zero as the size of the metadata.

[smarterclayton]

Can we patch quickly to work with v2 for Size, or do we need to wait for the 2.4 registry update?

[legionus]

Can we patch quickly to work with v2 for Size

We can't change the manifest, because it will change the digest of manifest.

do we need to wait for the 2.4 registry

This update doesn't help because schema v1 is affected.

We need to calculate the Size in different way. In same way as dockerregistry does:

https://github.com/openshift/origin/blob/master/pkg/dockerregistry/server/repositorymiddleware.go#L427-L443

[legionus]

I guess I can fix it.

[smarterclayton]

That's what I'm referring to - calculating size differently on import by
adding the schema2 fields to the schema1 object (which we already partially
do)

On Wed, Jun 1, 2016 at 11:21 AM, Alexey Gladkov notifications@github.com
wrote:

I guess I can fix it.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#7706 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/ABG_py3c5zWOQuZOgz2MTw3kJI0cVwfbks5qHaNogaJpZM4HmE1K
.

[smarterclayton]

Can I get a commit from someone to either put a fix in or work around this by 2pm EDT? I'm ok with disabling the test condition that's failing as long as someone is committed to a fix.

This is something we may have to backport to 1.2.0 and earlier so that people can actually use us.

[0xmichalis]

@legionus has a fix locally

[smarterclayton]

Thanks, please keep this issue updated and be sure to notify people on the
mailing lists that the queue is broken.

On Wed, Jun 1, 2016 at 12:31 PM, Michail Kargakis notifications@github.com
wrote:

@legionus https://github.com/legionus has a fix locally

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#7706 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/ABG_pzBD4lO1zX6kmdqTVhwSxPbsDJbgks5qHbPigaJpZM4HmE1K
.

[smarterclayton]

For 1.3 we need to make the transition to schema2 (well under way) and then be sure we import the new size.

[legionus]

@smarterclayton I'm not sure I understand.
Cc: @miminar

[smarterclayton]

We need to be sure that we import via schema2 by default, and fall back to schema1 only if necessary.

[smarterclayton]

Summary:

1. When the new DockerHub (and docker/distribution 2.4+) receive a manifest pushed from docker 1.10 (which will be schema2), a client that requests the manifest without specifying wanting schema2 will get back a schema1 manifest that has no v1Compatibility data containing size
2. Clients who only push and pull from Docker 1.9 will still be sending schema1 manifests, which will contain the size data
3. When we upgrade to using the newer registry client in 2.4, we'll start asking for schema2 entries from the Hub and get back the layer size, which we'll need to copy into our internal DockerImageMetadata section when the image is imported
4. For now, we'll return zero size layers and images via our API when fetching such objects

[deads2k]

I've seen this in 5 of the last 6 test runs, the sixth failed on yum. Bumping to P1.

@smarterclayton @mfojtik @miminar

[deads2k]

See https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin_integration/5503/ as an example.

[smarterclayton]

They updated pulp to v2 today, fixing the bug. We need to fix the test

[mfojtik]

@smarterclayton see #10639