Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bug 1902299: catalog mirror improvements #673

Merged
merged 4 commits into from Jan 16, 2021
Merged

Bug 1902299: catalog mirror improvements #673

merged 4 commits into from Jan 16, 2021

Conversation

ecordell
Copy link
Contributor

@ecordell ecordell commented Dec 16, 2020 •
edited

This fixes some issues with oc adm catalog mirror, post-appregistry:

  1. Images without namespaces were not generating proper mappings. Example: registry.access.redhat.com/ubi8-minimal
  2. --filter-by-os was previously allowed to filter arches of mirrored content. This would break references to those images in the catalog, which point the manifestlist and not the manifest.
  3. --filter-by-os now only filters the index image that is pulled and unpacked. To clarify this, a new flag --index-filter-by-os was added and should be used instead.
  4. Because of the way String() is calculated on parsed references, some valid targets were failing validation. Minimal validation for a mirroring target now only checks that a hostname has been parsed successfully.

These need to be backported to 4.6 as the address issues with mirroring index images (vs. previous workflows that assumed mirroring started with oc adm catalog build)

@openshift-ci-robot openshift-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 16, 2020
@openshift-merge-robot
Copy link
Contributor

@ecordell: The following test failed, say /retest to rerun all failed tests:

Test name Commit Details Rerun command
ci/prow/e2e-agnostic-cmd 2ce382f link /test e2e-agnostic-cmd

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@ecordell ecordell changed the title WIP: catalog mirror improvements Bug 1902299: catalog mirror improvements Dec 17, 2020
@openshift-ci-robot openshift-ci-robot added bugzilla/severity-high Referenced Bugzilla bug's severity is high for the branch this PR is targeting. and removed do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Dec 17, 2020
@openshift-ci-robot
Copy link

@ecordell: This pull request references Bugzilla bug 1902299, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.7.0) matches configured target release for branch (4.7.0)
  • bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

Bug 1902299: catalog mirror improvements

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot openshift-ci-robot added the bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. label Dec 17, 2020
@jianzhangbjz
Copy link
Contributor

/retest

@kevinrizza kevinrizza mentioned this pull request Jan 5, 2021
Closed
@jianzhangbjz
Copy link
Contributor

Depends on openshift/library-go#973

@ecordell ecordell force-pushed the catalog-mirror-namespaceless-fix branch 3 times, most recently from 69b2381 to 264bbf3 Compare January 7, 2021 15:53
@openshift-ci-robot openshift-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 10, 2021
@ecordell
properly handle mirroring namespaceless images
38e14d6
@ecordell
mirroring catalog contents forces mirroring the entire manifestlist
128660e 
this ensures that the mirrored manifests can still reference the
manifestlist by digest
@ecordell ecordell force-pushed the catalog-mirror-namespaceless-fix branch from 264bbf3 to dbef35a Compare January 14, 2021 16:30
@openshift-ci-robot openshift-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Jan 14, 2021
@openshift-ci-robot
Copy link

@ecordell: This pull request references Bugzilla bug 1902299, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.7.0) matches configured target release for branch (4.7.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

Bug 1902299: catalog mirror improvements

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@ecordell ecordell force-pushed the catalog-mirror-namespaceless-fix branch from dbef35a to 5f2fd3e Compare January 14, 2021 18:09
@openshift-ci-robot openshift-ci-robot removed the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 14, 2021
@ecordell
Copy link
Contributor Author

/retest

@kevinrizza
Copy link
Member

/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Jan 14, 2021
@ecordell
Copy link
Contributor Author

ecordell commented Jan 14, 2021 via email

@ecordell
Copy link
Contributor Author

/retest

Quota issues

@ecordell
Copy link
Contributor Author

/approve

@ecordell
Copy link
Contributor Author

/retest

2 similar comments
@ecordell
Copy link
Contributor Author

/retest

@ecordell
Copy link
Contributor Author

/retest

@ecordell ecordell force-pushed the catalog-mirror-namespaceless-fix branch from 5f2fd3e to 836f56f Compare January 15, 2021 14:03
@openshift-ci-robot openshift-ci-robot removed the lgtm Indicates that a PR is ready to be merged. label Jan 15, 2021
@ecordell ecordell force-pushed the catalog-mirror-namespaceless-fix branch from 836f56f to c615889 Compare January 15, 2021 14:04
soltysh
soltysh approved these changes Jan 15, 2021
View reviewed changes
Copy link
Member

@soltysh soltysh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few nits that can be addressed in a follow-up
/approve

pkg/cli/admin/catalog/mirror.go

// the old flag name is kept for backwards-compatibility.
// if both old and new are specified, the value of the flag coming later will be used.
flags.StringVar(&o.FilterOptions.FilterByOS, "filter-by-os", o.FilterOptions.FilterByOS, "Use --index-filter-by-os instead. A regular expression to control which index image is picked when multiple variants are available. Images will be passed as '<platform>/<architecture>[/<variant>]'. This does not apply to images referenced by the index.")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd suggest deprecating this flag, not to confuse users. Can be done in a follow-up.

pkg/cli/admin/catalog/mirror.go
@@ -144,8 +154,10 @@ func (o *MirrorCatalogOptions) Complete(cmd *cobra.Command, args []string) error
src := args[0]
dest := args[1]

if err := o.FilterOptions.Complete(cmd.Flags()); err != nil {
return err
// default to linux/arm64 for index image, which we generally expect to exist
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: comment says default to linux/arm64 but you default to linux/amd64

pkg/cli/admin/catalog/mirror.go
// default to linux/arm64 for index image, which we generally expect to exist
pattern := o.FilterOptions.FilterByOS
if len(pattern) == 0 {
o.FilterOptions.FilterByOS = regexp.QuoteMeta(fmt.Sprintf("%s/%s", "linux", "amd64"))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is excessive, just pass linux/amd64.

@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 15, 2021
@ecordell
Copy link
Contributor Author

/retest

1 similar comment
@ecordell
Copy link
Contributor Author

/retest

@kevinrizza
Copy link
Member

/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Jan 15, 2021
@openshift-ci-robot
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ecordell, kevinrizza, soltysh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

6 similar comments
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-merge-robot openshift-merge-robot merged commit 4d52be6 into openshift:master Jan 16, 2021
@openshift-ci-robot
Copy link

@ecordell: All pull requests linked via external trackers have merged:

Bugzilla bug 1902299 has been moved to the MODIFIED state.

In response to this:

Bug 1902299: catalog mirror improvements

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@ecordell
Copy link
Contributor Author

/cherry-pick release-4.6

@openshift-cherrypick-robot
Copy link

@ecordell: #673 failed to apply on top of branch "release-4.6":

Applying: properly handle mirroring namespaceless images
Using index info to reconstruct a base tree...
M	pkg/cli/admin/catalog/mirrorer.go
M	pkg/cli/admin/catalog/mirrorer_test.go
Falling back to patching base and 3-way merge...
Auto-merging pkg/cli/admin/catalog/mirrorer_test.go
CONFLICT (content): Merge conflict in pkg/cli/admin/catalog/mirrorer_test.go
Auto-merging pkg/cli/admin/catalog/mirrorer.go
CONFLICT (content): Merge conflict in pkg/cli/admin/catalog/mirrorer.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 properly handle mirroring namespaceless images
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick release-4.6

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

This was referenced Jan 18, 2021
Merged
Merged
@stevsmit stevsmit mentioned this pull request Mar 18, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@soltysh soltysh soltysh approved these changes

@smarterclayton smarterclayton Awaiting requested review from smarterclayton

Assignees

@kevinrizza kevinrizza

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. bugzilla/severity-high Referenced Bugzilla bug's severity is high for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. lgtm Indicates that a PR is ready to be merged.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
8 participants
@ecordell @openshift-merge-robot @openshift-ci-robot @jianzhangbjz @kevinrizza @openshift-bot @openshift-cherrypick-robot @soltysh