Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug 1761921: Manifest lists can fail to mirror because dependent manifests are not uploaded #124

Merged

Conversation

smarterclayton
Copy link
Contributor

@smarterclayton smarterclayton commented Oct 9, 2019

We mirror the manifest list and blobs, but not the sub manifests. Correct that by registering those with the plan.

Without this change mirroring a manifest list and --filter-by-os=.* fails with 'manifest invalid' due to the sub manifest not being copied (you can't push a manifest list unless all manifests are found).

@openshift-ci-robot openshift-ci-robot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Oct 9, 2019
@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 9, 2019
@jupierce
Copy link
Contributor

/retest

@jupierce
Copy link
Contributor

# Verify that source image is a manifest list
[jupierce@localhost tmp]$ oc image info $PULL
error: the image is a manifest list and contains multiple images - use --filter-by-os to select from:

  OS            DIGEST
  linux/amd64   sha256:4e08611a1e91857415193c19f9e1cab76ee770c99fb278c6de8a532ca22ee846
  linux/ppc64le sha256:1d453e11abab173ce475886076ab79f2ce82807d8d09acd84b6e34fd493d8a0d
  linux/s390x   sha256:da71db0a07446e584fb58225a82d3600c5827a632a3be8f22707a00ffe615f9e

# mirror the manifest with all os
[jupierce@localhost tmp]$ ./oc image mirror --filter-by-os='.*' $PULL=docker.io/jupierce/jmp-mco-test:13
docker.io/
  jupierce/jmp-mco-test
    blobs:
      registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-machine-config-operator sha256:bff3b73cbcc496de1de4ea51df88b7249169d0b6eb7d677169eaf90b8a92240e 1.575KiB
      registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-machine-config-operator sha256:c01a8db16457d5f2a2b608e58ea55af853f06d8fae9b6703885eb95713c489b7 1.589KiB
      registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-machine-config-operator sha256:6ab3594990f7d846cbb7db9d9f91681b06de1676c6aa0aef30b7e1ecb97eda20 1.601KiB
      registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-machine-config-operator sha256:d7dbec4a1a0d644d03c0f67dbac2eaf58bf24b832adc9b5fd1547991c231a252 5.02KiB
      registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-machine-config-operator sha256:d5f91c6c381615c6e759c0efebf49cc1ff49d5efe213afe1adf5be9c018d6283 5.021KiB
      registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-machine-config-operator sha256:5cf79a25ac49372eb532db884f79736a5b61a63f010f0a84aa38e5159e1beee7 5.034KiB
      registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-machine-config-operator sha256:15231c89c2718e48d76137164c992cc62f08de68ba38024621c7f3dc58ca93c1 4.045MiB
      registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-machine-config-operator sha256:011f8a055611753cfa575b3d7fcf87e0f3c75b122e150b34f8607676c2da0578 4.245MiB
      registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-machine-config-operator sha256:0896ce2aa386eeec9568a366da509d9459d73cd856867b0a569ed55814b1e6c5 4.78MiB
      registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-machine-config-operator sha256:ec1631bd8e793af2f01874681fc038169b2256c639fa839375c05990e02a3a52 7.838MiB
      registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-machine-config-operator sha256:fcdf092e5283999e73df866bfd0305469b601f4ee949786a13fceb738207c05d 8.148MiB
      registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-machine-config-operator sha256:46a8cf4dc25598d8b737dd47ca052d40726471a725fe195209f8e325fc7dd8dd 8.573MiB
      registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-machine-config-operator sha256:cce31979f10e4547de1a9a6cb41aef6b2280b65644427aa02f7bc6e0d0774ae3 42.75MiB
      registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-machine-config-operator sha256:4dbc243c988badba0a961462eae38bfdd4898f699b9451c55c89f33e125dde22 46.96MiB
      registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-machine-config-operator sha256:fbe2eceb904fb481d9648a6924cc63b5f009afc47828580165fbf664eed1c345 47.45MiB
      registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-machine-config-operator sha256:7b1c937e0f6794db2535be6e4cb6d60a0b668ef78c2576611a3fb9c97a95ccdf 72.71MiB
      registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-machine-config-operator sha256:4a75061d90808a8be6e5704398222e188888adf806548bd7ee3b53e425ac797f 72.99MiB
      registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-machine-config-operator sha256:7a3219ff9fc9133e6639a340b5fbf26fe968ed65d78a7df30862591a938852a9 76.6MiB
    manifests:
      {}
      sha256:01a06ebcb96f0f9b4fc903b4dd6489cdf4fa197211cdd041015ab403b471a5e1 -> 13
  stats: shared=0 unique=18 size=397.1MiB ratio=1.00

phase 0:
  docker.io jupierce/jmp-mco-test blobs=18 mounts=0 manifests=2 shared=0

info: Planning completed in 1.53s
uploading: docker.io/library/jupierce/jmp-mco-test sha256:7b1c937e0f6794db2535be6e4cb6d60a0b668ef78c2576611a3fb9c97a95ccdf 72.71MiB
uploading: docker.io/library/jupierce/jmp-mco-test sha256:ec1631bd8e793af2f01874681fc038169b2256c639fa839375c05990e02a3a52 7.838MiB
uploading: docker.io/library/jupierce/jmp-mco-test sha256:4dbc243c988badba0a961462eae38bfdd4898f699b9451c55c89f33e125dde22 46.96MiB
uploading: docker.io/library/jupierce/jmp-mco-test sha256:fbe2eceb904fb481d9648a6924cc63b5f009afc47828580165fbf664eed1c345 47.45MiB
uploading: docker.io/library/jupierce/jmp-mco-test sha256:46a8cf4dc25598d8b737dd47ca052d40726471a725fe195209f8e325fc7dd8dd 8.573MiB
uploading: docker.io/library/jupierce/jmp-mco-test sha256:15231c89c2718e48d76137164c992cc62f08de68ba38024621c7f3dc58ca93c1 4.045MiB
uploading: docker.io/library/jupierce/jmp-mco-test sha256:cce31979f10e4547de1a9a6cb41aef6b2280b65644427aa02f7bc6e0d0774ae3 42.75MiB
uploading: docker.io/library/jupierce/jmp-mco-test sha256:7a3219ff9fc9133e6639a340b5fbf26fe968ed65d78a7df30862591a938852a9 76.6MiB
uploading: docker.io/library/jupierce/jmp-mco-test sha256:fcdf092e5283999e73df866bfd0305469b601f4ee949786a13fceb738207c05d 8.148MiB
uploading: docker.io/library/jupierce/jmp-mco-test sha256:011f8a055611753cfa575b3d7fcf87e0f3c75b122e150b34f8607676c2da0578 4.245MiB
uploading: docker.io/library/jupierce/jmp-mco-test sha256:4a75061d90808a8be6e5704398222e188888adf806548bd7ee3b53e425ac797f 72.99MiB
uploading: docker.io/library/jupierce/jmp-mco-test sha256:0896ce2aa386eeec9568a366da509d9459d73cd856867b0a569ed55814b1e6c5 4.78MiB
sha256:4e08611a1e91857415193c19f9e1cab76ee770c99fb278c6de8a532ca22ee846 docker.io/library/jupierce/jmp-mco-test
sha256:4e08611a1e91857415193c19f9e1cab76ee770c99fb278c6de8a532ca22ee846 docker.io/library/jupierce/jmp-mco-test:13
info: Mirroring completed in 12.02s (34.62MB/s)

# Check the result and find that the destination is not a manifestlist
[jupierce@localhost tmp]$ oc image info docker.io/jupierce/jmp-mco-test:13
Name:        docker.io/jupierce/jmp-mco-test:13
Digest:      sha256:4e08611a1e91857415193c19f9e1cab76ee770c99fb278c6de8a532ca22ee846
Media Type:  application/vnd.docker.distribution.manifest.v2+json
Created:     8d ago
Image Size:  138.5MB in 5 layers
Layers:      76.24MB sha256:7b1c937e0f6794db2535be6e4cb6d60a0b668ef78c2576611a3fb9c97a95ccdf
             1.613kB sha256:bff3b73cbcc496de1de4ea51df88b7249169d0b6eb7d677169eaf90b8a92240e
             4.241MB sha256:15231c89c2718e48d76137164c992cc62f08de68ba38024621c7f3dc58ca93c1
             8.219MB sha256:ec1631bd8e793af2f01874681fc038169b2256c639fa839375c05990e02a3a52
             49.75MB sha256:fbe2eceb904fb481d9648a6924cc63b5f009afc47828580165fbf664eed1c345
OS:          linux
Arch:        amd64
Entrypoint:  /usr/bin/machine-config-operator
....

We mirror the manifest list and blobs, but not the sub manifests.
Correct that by registering those with the plan. We must then
upload dependent manifests first, since the registry will not allow
partial manifest lists.
@openshift-ci-robot openshift-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Oct 11, 2019
@smarterclayton
Copy link
Contributor Author

Updated with ordering fixes

@jupierce
Copy link
Contributor

/retest

@jupierce
Copy link
Contributor

/test e2e-cmd

@jupierce
Copy link
Contributor

Success!

[jupierce@localhost tmp]$ ./oc image mirror '--filter-by-os=.*' $PULL=docker.io/jupierce/jmp-mco-test:16
docker.io/
  jupierce/jmp-mco-test
    blobs:
      registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-machine-config-operator sha256:bff3b73cbcc496de1de4ea51df88b7249169d0b6eb7d677169eaf90b8a92240e 1.575KiB
      registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-machine-config-operator sha256:c01a8db16457d5f2a2b608e58ea55af853f06d8fae9b6703885eb95713c489b7 1.589KiB
      registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-machine-config-operator sha256:6ab3594990f7d846cbb7db9d9f91681b06de1676c6aa0aef30b7e1ecb97eda20 1.601KiB
      registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-machine-config-operator sha256:d7dbec4a1a0d644d03c0f67dbac2eaf58bf24b832adc9b5fd1547991c231a252 5.02KiB
      registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-machine-config-operator sha256:d5f91c6c381615c6e759c0efebf49cc1ff49d5efe213afe1adf5be9c018d6283 5.021KiB
      registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-machine-config-operator sha256:5cf79a25ac49372eb532db884f79736a5b61a63f010f0a84aa38e5159e1beee7 5.034KiB
      registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-machine-config-operator sha256:15231c89c2718e48d76137164c992cc62f08de68ba38024621c7f3dc58ca93c1 4.045MiB
      registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-machine-config-operator sha256:011f8a055611753cfa575b3d7fcf87e0f3c75b122e150b34f8607676c2da0578 4.245MiB
      registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-machine-config-operator sha256:0896ce2aa386eeec9568a366da509d9459d73cd856867b0a569ed55814b1e6c5 4.78MiB
      registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-machine-config-operator sha256:ec1631bd8e793af2f01874681fc038169b2256c639fa839375c05990e02a3a52 7.838MiB
      registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-machine-config-operator sha256:fcdf092e5283999e73df866bfd0305469b601f4ee949786a13fceb738207c05d 8.148MiB
      registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-machine-config-operator sha256:46a8cf4dc25598d8b737dd47ca052d40726471a725fe195209f8e325fc7dd8dd 8.573MiB
      registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-machine-config-operator sha256:cce31979f10e4547de1a9a6cb41aef6b2280b65644427aa02f7bc6e0d0774ae3 42.75MiB
      registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-machine-config-operator sha256:4dbc243c988badba0a961462eae38bfdd4898f699b9451c55c89f33e125dde22 46.96MiB
      registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-machine-config-operator sha256:fbe2eceb904fb481d9648a6924cc63b5f009afc47828580165fbf664eed1c345 47.45MiB
      registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-machine-config-operator sha256:7b1c937e0f6794db2535be6e4cb6d60a0b668ef78c2576611a3fb9c97a95ccdf 72.71MiB
      registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-machine-config-operator sha256:4a75061d90808a8be6e5704398222e188888adf806548bd7ee3b53e425ac797f 72.99MiB
      registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-machine-config-operator sha256:7a3219ff9fc9133e6639a340b5fbf26fe968ed65d78a7df30862591a938852a9 76.6MiB
    manifests:
      sha256:01a06ebcb96f0f9b4fc903b4dd6489cdf4fa197211cdd041015ab403b471a5e1
      sha256:1d453e11abab173ce475886076ab79f2ce82807d8d09acd84b6e34fd493d8a0d
      sha256:4e08611a1e91857415193c19f9e1cab76ee770c99fb278c6de8a532ca22ee846
      sha256:da71db0a07446e584fb58225a82d3600c5827a632a3be8f22707a00ffe615f9e
      sha256:01a06ebcb96f0f9b4fc903b4dd6489cdf4fa197211cdd041015ab403b471a5e1 -> 16
  stats: shared=0 unique=18 size=397.1MiB ratio=1.00

phase 0:
  docker.io jupierce/jmp-mco-test blobs=18 mounts=0 manifests=5 shared=0

info: Planning completed in 2.17s
uploading: docker.io/library/jupierce/jmp-mco-test sha256:15231c89c2718e48d76137164c992cc62f08de68ba38024621c7f3dc58ca93c1 4.045MiB
uploading: docker.io/library/jupierce/jmp-mco-test sha256:7a3219ff9fc9133e6639a340b5fbf26fe968ed65d78a7df30862591a938852a9 76.6MiB
uploading: docker.io/library/jupierce/jmp-mco-test sha256:fbe2eceb904fb481d9648a6924cc63b5f009afc47828580165fbf664eed1c345 47.45MiB
uploading: docker.io/library/jupierce/jmp-mco-test sha256:4dbc243c988badba0a961462eae38bfdd4898f699b9451c55c89f33e125dde22 46.96MiB
uploading: docker.io/library/jupierce/jmp-mco-test sha256:fcdf092e5283999e73df866bfd0305469b601f4ee949786a13fceb738207c05d 8.148MiB
uploading: docker.io/library/jupierce/jmp-mco-test sha256:0896ce2aa386eeec9568a366da509d9459d73cd856867b0a569ed55814b1e6c5 4.78MiB
uploading: docker.io/library/jupierce/jmp-mco-test sha256:46a8cf4dc25598d8b737dd47ca052d40726471a725fe195209f8e325fc7dd8dd 8.573MiB
uploading: docker.io/library/jupierce/jmp-mco-test sha256:cce31979f10e4547de1a9a6cb41aef6b2280b65644427aa02f7bc6e0d0774ae3 42.75MiB
uploading: docker.io/library/jupierce/jmp-mco-test sha256:4a75061d90808a8be6e5704398222e188888adf806548bd7ee3b53e425ac797f 72.99MiB
uploading: docker.io/library/jupierce/jmp-mco-test sha256:7b1c937e0f6794db2535be6e4cb6d60a0b668ef78c2576611a3fb9c97a95ccdf 72.71MiB
uploading: docker.io/library/jupierce/jmp-mco-test sha256:ec1631bd8e793af2f01874681fc038169b2256c639fa839375c05990e02a3a52 7.838MiB
uploading: docker.io/library/jupierce/jmp-mco-test sha256:011f8a055611753cfa575b3d7fcf87e0f3c75b122e150b34f8607676c2da0578 4.245MiB
sha256:da71db0a07446e584fb58225a82d3600c5827a632a3be8f22707a00ffe615f9e docker.io/library/jupierce/jmp-mco-test
sha256:4e08611a1e91857415193c19f9e1cab76ee770c99fb278c6de8a532ca22ee846 docker.io/library/jupierce/jmp-mco-test
sha256:1d453e11abab173ce475886076ab79f2ce82807d8d09acd84b6e34fd493d8a0d docker.io/library/jupierce/jmp-mco-test
sha256:01a06ebcb96f0f9b4fc903b4dd6489cdf4fa197211cdd041015ab403b471a5e1 docker.io/library/jupierce/jmp-mco-test:16
info: Mirroring completed in 3m49.21s (1.817MB/s)
[jupierce@localhost tmp]$ oc image info docker.io/jupierce/jmp-mco-test:16
error: the image is a manifest list and contains multiple images - use --filter-by-os to select from:

  OS            DIGEST
  linux/amd64   sha256:4e08611a1e91857415193c19f9e1cab76ee770c99fb278c6de8a532ca22ee846
  linux/ppc64le sha256:1d453e11abab173ce475886076ab79f2ce82807d8d09acd84b6e34fd493d8a0d
  linux/s390x   sha256:da71db0a07446e584fb58225a82d3600c5827a632a3be8f22707a00ffe615f9e

Copy link
Contributor

@soltysh soltysh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tagging based on @jupierce tests
/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Oct 14, 2019
@openshift-ci-robot
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: smarterclayton, soltysh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [smarterclayton,soltysh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@smarterclayton
Copy link
Contributor Author

I will add origin mirroring e2e tests for this after (we have a few new tests to be created)

@openshift-merge-robot openshift-merge-robot merged commit 6a5ac77 into openshift:master Oct 14, 2019
@smarterclayton smarterclayton changed the title image-mirror: Manifest list manifests have to be copied Bug 1761921: Manifest lists can fail to mirror because dependent manifests are not uploaded Oct 15, 2019
@openshift-ci-robot
Copy link

@smarterclayton: All pull requests linked via external trackers have merged. Bugzilla bug 1761921 has been moved to the MODIFIED state.

In response to this:

Bug 1761921: Manifest lists can fail to mirror because dependent manifests are not uploaded

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@smarterclayton
Copy link
Contributor Author

/cherrypick release-4.2

@openshift-cherrypick-robot

@smarterclayton: new pull request created: #128

In response to this:

/cherrypick release-4.2

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants