Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CLID-10: Generate CatalogSource files #784

Merged
merged 1 commit into from Jan 31, 2024

Conversation

sherine-k
Copy link
Contributor

@sherine-k sherine-k commented Jan 19, 2024

Description

This PR contains implementation for CLID-10 :

  • CatalogSource for operator catalogs
  • It introduces targetCatalogSourceTemplate to the imageSetConfig, in order to allow the user customization of the catalogSource
    • if not provided, the catalogSource will not contain any updateStrategy (polling interval)
    • when provided, oc-mirror will attempt to use the template as a base for catalogSource generation, and in case of failure, it will failover to generation without template with a warning in the logs
  • It removes deprecated fields targetName and originalRef from the imageSetConfig

Fixes # CLID-10.

PS: catalogSource resource name and filename use a different naming convention than v1: "cs-" + catalogRepository + "-" + date . We might need to reconsider this strategy based on how customers use these resources in the cluster. ie. Do they deploy the new catalogSource and delete the old? do they expect an update of the existing resource?

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

Prior to testing, create the file clid10.yaml with content:

kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v1alpha2
archiveSize: 1
mirror:
  operators:
  - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.14
    packages:
    - name: aws-load-balancer-operator
    targetCatalogSourceTemplate: "/home/skhoury/go/src/github.com/openshift/oc-mirror/v2/tests/catalog-source_template.yaml"

And the catalog-source_template.yaml

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: totalRubbish
  namespace: openshift-marketplace
spec:
  image: totalRubbish
  sourceType: grpc
  updateStrategy:
    registryPoll: 
      interval: 30m0s

Next, perform a mirrorToDisk + DiskToMirror

./bin/oc-mirror --v2 -c clid10.yaml file:///home/skhoury/clid-10
./bin/oc-mirror --v2 -c clid10.yaml --from file:///home/skhoury/clid-10 docker://localhost:5000 --dest-tls-verify=false

Expected Outcome

The catalogSource file is generated under the working directory, under subfolder cluster-resources. The catalog source file name is derived from the catalog name, with cs- as prefix, and the date as suffix.

Ex: cs-redhat-operator-index-2024-01-22t17-37-25z.yaml

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  creationTimestamp: null
  name: cs-redhat-operator-index-2024-01-22t17-37-25z
  namespace: openshift-marketplace
spec:
  icon: {}
  image: localhost:5000/clid10/redhat/redhat-operator-index:v4.14
  sourceType: grpc
  updateStrategy:
    registryPoll:
      interval: 30m0s
status: {}

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jan 19, 2024
@openshift-ci-robot
Copy link

openshift-ci-robot commented Jan 19, 2024

@sherine-k: This pull request references CLID-10 which is a valid jira issue.

In response to this:

Description

This PR implements generation of catalog source custom resource files.

Fixes CLID-10

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

Additional unit tests.
With the following imagesetconfig, do mirror to disk followed by disk to mirror(./bin/oc-mirror --v2 -c [clid-10](https://issues.redhat.com//browse/clid-10).yaml --from file:///home/skhoury/clid-10 docker://localhost:5000/cs --dest-tls-verify=false), you should get the catalog source in the working-dir/cluster-resources folder

kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v1alpha2
archiveSize: 1
mirror:
 operators:
 - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.14
   packages:
   - name: aws-load-balancer-operator
 additionalImages:
 - name: "gcr.io/kubebuilder/kube-rbac-proxy:v0.5.0"
 - name: "quay.io/cockroachdb/cockroach-helm-operator:6.0.0"
 - name: "quay.io/helmoperators/cockroachdb:v5.0.3"
 - name: "quay.io/helmoperators/cockroachdb:v5.0.4"
 - name: "quay.io/openshift-community-operators/cockroachdb@sha256:a5d4f4467250074216eb1ba1c36e06a3ab797d81c431427fc2aca97ecaf4e9d8"
 - name: "quay.io/openshift-community-operators/cockroachdb@sha256:d3016b1507515fc7712f9c47fd9082baf9ccb070aaab58ed0ef6e5abdedde8ba"
 - name: "quay.io/openshift-community-operators/cockroachdb@sha256:f42337e7b85a46d83c94694638e2312e10ca16a03542399a65ba783c94a32b63"

Expected Outcome

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
 name: cs-redhat-operator-index-2024-01-19t17-17-35z
 namespace: openshift-marketplace
spec:
 image: localhost:5000/cs/redhat/redhat-operator-index:v4.14
 sourceType: grpc

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 19, 2024
@openshift-ci-robot
Copy link

openshift-ci-robot commented Jan 22, 2024

@sherine-k: This pull request references CLID-10 which is a valid jira issue.

In response to this:

Description

This PR implements generation of catalog source custom resource files.

Fixes CLID-10.

In this implementation, the custom resource name, as well as the file name are constructed as:
"cs-"+catalogImage+"-date".

This is different than the logic adopted in V1.

❓ should the custom resource name be built based on the date? still rely on the old logic? What is the impact in a disconnected cluster.

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

Additional unit tests.
With the following imagesetconfig, do mirror to disk followed by disk to mirror(./bin/oc-mirror --v2 -c [clid-10](https://issues.redhat.com//browse/clid-10).yaml --from file:///home/skhoury/clid-10 docker://localhost:5000/cs --dest-tls-verify=false), you should get the catalog source in the working-dir/cluster-resources folder

kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v1alpha2
archiveSize: 1
mirror:
 operators:
 - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.14
   packages:
   - name: aws-load-balancer-operator
 additionalImages:
 - name: "gcr.io/kubebuilder/kube-rbac-proxy:v0.5.0"
 - name: "quay.io/cockroachdb/cockroach-helm-operator:6.0.0"
 - name: "quay.io/helmoperators/cockroachdb:v5.0.3"
 - name: "quay.io/helmoperators/cockroachdb:v5.0.4"
 - name: "quay.io/openshift-community-operators/cockroachdb@sha256:a5d4f4467250074216eb1ba1c36e06a3ab797d81c431427fc2aca97ecaf4e9d8"
 - name: "quay.io/openshift-community-operators/cockroachdb@sha256:d3016b1507515fc7712f9c47fd9082baf9ccb070aaab58ed0ef6e5abdedde8ba"
 - name: "quay.io/openshift-community-operators/cockroachdb@sha256:f42337e7b85a46d83c94694638e2312e10ca16a03542399a65ba783c94a32b63"

Expected Outcome

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
 name: cs-redhat-operator-index-2024-01-19t17-17-35z
 namespace: openshift-marketplace
spec:
 image: localhost:5000/cs/redhat/redhat-operator-index:v4.14
 sourceType: grpc

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link

openshift-ci-robot commented Jan 22, 2024

@sherine-k: This pull request references CLID-10 which is a valid jira issue.

In response to this:

Description

This PR implements generation of catalog source custom resource files.

Fixes CLID-10.

In this implementation, the custom resource name, as well as the file name are constructed as:
"cs-"+catalogImage+"-date".

This is different than the logic adopted in V1.

❓ should the custom resource name be built based on the date? still rely on the old logic? What is the impact in a disconnected cluster.

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

Additional unit tests.
With the following imagesetconfig, do mirror to disk followed by disk to mirror(./bin/oc-mirror --v2 -c clid10.yaml --from file:///home/skhoury/clid-10 docker://localhost:5000/cs --dest-tls-verify=false), you should get the catalog source in the working-dir/cluster-resources folder

kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v1alpha2
archiveSize: 1
mirror:
 operators:
 - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.14
   packages:
   - name: aws-load-balancer-operator
 additionalImages:
 - name: "gcr.io/kubebuilder/kube-rbac-proxy:v0.5.0"
 - name: "quay.io/cockroachdb/cockroach-helm-operator:6.0.0"
 - name: "quay.io/helmoperators/cockroachdb:v5.0.3"
 - name: "quay.io/helmoperators/cockroachdb:v5.0.4"
 - name: "quay.io/openshift-community-operators/cockroachdb@sha256:a5d4f4467250074216eb1ba1c36e06a3ab797d81c431427fc2aca97ecaf4e9d8"
 - name: "quay.io/openshift-community-operators/cockroachdb@sha256:d3016b1507515fc7712f9c47fd9082baf9ccb070aaab58ed0ef6e5abdedde8ba"
 - name: "quay.io/openshift-community-operators/cockroachdb@sha256:f42337e7b85a46d83c94694638e2312e10ca16a03542399a65ba783c94a32b63"

Expected Outcome

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
 name: cs-redhat-operator-index-2024-01-19t17-17-35z
 namespace: openshift-marketplace
spec:
 image: localhost:5000/cs/redhat/redhat-operator-index:v4.14
 sourceType: grpc

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link

openshift-ci-robot commented Jan 22, 2024

@sherine-k: This pull request references CLID-10 which is a valid jira issue.

In response to this:

Description

This PR implements generation of catalog source custom resource files.

Fixes CLID-10.

In this implementation, the custom resource name, as well as the file name are constructed as:
"cs-"+catalogImage+"-date".

This is different than the logic adopted in V1.

❓ should the custom resource name be built based on the date? still rely on the old logic? What is the impact in a disconnected cluster.

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

Additional unit tests.
With the following imagesetconfig, do mirror to disk followed by disk to mirror(./bin/oc-mirror --v2 -c clid10.yaml --from file:///home/skhoury/clid-10 docker://localhost:5000/cs --dest-tls-verify=false), you should get the catalog source in the working-dir/cluster-resources folder

kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v1alpha2
archiveSize: 1
mirror:
 operators:
 - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.14
   packages:
   - name: aws-load-balancer-operator
 additionalImages:
 - name: "gcr.io/kubebuilder/kube-rbac-proxy:v0.5.0"
 - name: "quay.io/cockroachdb/cockroach-helm-operator:6.0.0"
 - name: "quay.io/helmoperators/cockroachdb:v5.0.3"
 - name: "quay.io/helmoperators/cockroachdb:v5.0.4"
 - name: "quay.io/openshift-community-operators/cockroachdb@sha256:a5d4f4467250074216eb1ba1c36e06a3ab797d81c431427fc2aca97ecaf4e9d8"
 - name: "quay.io/openshift-community-operators/cockroachdb@sha256:d3016b1507515fc7712f9c47fd9082baf9ccb070aaab58ed0ef6e5abdedde8ba"
 - name: "quay.io/openshift-community-operators/cockroachdb@sha256:f42337e7b85a46d83c94694638e2312e10ca16a03542399a65ba783c94a32b63"

Expected Outcome

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
 name: cs-redhat-operator-index-2024-01-19t17-17-35z
 namespace: openshift-marketplace
spec:
 image: localhost:5000/cs/redhat/redhat-operator-index:v4.14
 sourceType: grpc

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@sherine-k sherine-k mentioned this pull request Jan 22, 2024
4 tasks
@openshift-ci-robot
Copy link

openshift-ci-robot commented Jan 22, 2024

@sherine-k: This pull request references CLID-10 which is a valid jira issue.

In response to this:

Description

This PR contains implementation for CLID-10 (see jira for detailed design) :

  • ImageTagMirrorSet : enabled by flag --generate-itms
  • ImageDigestMirrorSet: with the following behavior
  • Generated by default (when no --generate-itms flag): Contains source and mirrors for all mirrored images.
  • Contains source and mirror paths namespace scoped, unless --max-nested-paths has been used
  • Contains source and mirrors only for images by digest if --generate-itms is used
  • Excluded: CatalogSource for operator catalogs will be in a different PR CLID-10: Generate CatalogSource files #784

Fixes # CLID-10.

Regarding IDMS/ITMS generation, a few questions are still pending in CLID-10 (see comments):

  • By default, oc-mirror will generate IDMS only. Images referenced by tag will be added to that IDMS. If a cluster deploys such an IDMS, and references those images by tag, the mirror will not be used. Therefore, should we:
  • keep the specified behavior, but and display warnings for images by tag (Implemented)
  • By default generate IDMS and ITMS (images by tag can therefore benefit from the mirrors)
  • Keep the default to generate IDMS only, exclude images by tag and display warnings for them
  • To be confirmed: In this PR, we stop resolving tags to digest prior to mirroring. In V1, oc-mirror was removing replacing tags by digests before handing mappings.txt to the oc code, which in turn, created a fake tag (derived from the digest value). This V1 behavior should not be used for V2, because we have ITMS, because we don't call oc, and because we don't rebuild catalogs (the reference for the image is not modified in the declarative config).

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

Prior to testing, create the file clid10.yaml with content:

kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v1alpha2
archiveSize: 1
mirror:
 additionalImages:
 - name: "gcr.io/kubebuilder/kube-rbac-proxy:v0.5.0"
 - name: "quay.io/cockroachdb/cockroach-helm-operator:6.0.0"
 - name: "quay.io/helmoperators/cockroachdb:v5.0.3"
 - name: "quay.io/helmoperators/cockroachdb:v5.0.4"
 - name: "quay.io/openshift-community-operators/cockroachdb@sha256:a5d4f4467250074216eb1ba1c36e06a3ab797d81c431427fc2aca97ecaf4e9d8"
 - name: "quay.io/openshift-community-operators/cockroachdb@sha256:d3016b1507515fc7712f9c47fd9082baf9ccb070aaab58ed0ef6e5abdedde8ba"
 - name: "quay.io/openshift-community-operators/cockroachdb@sha256:f42337e7b85a46d83c94694638e2312e10ca16a03542399a65ba783c94a32b63"

Next, perform a mirrorToDisk

./bin/oc-mirror --v2 -c clid10.yaml file:///home/skhoury/clid-10
Test Description Outputs
./bin/oc-mirror --v2 -c clid10.yaml --from file:///home/skhoury/clid-10/ docker://localhost:5000/clid10/noflag --dest-tls-verify=false Nominal use case - IDMS generation only - All tags resolved to digest - IDMS with namespace scope idms-noflags.yaml (see below)
images pushed: x,y,z
./bin/oc-mirror --v2 -c clid10.yaml --generate-itms --from file:///home/skhoury/clid-10/ docker://localhost:5000/clid10/itms --dest-tls-verify=false with --generate-itms - generates ITMS and IDMS files, with namespace scope idms-tagflag.yaml + itms-tagflag.yaml
./bin/oc-mirror --v2 -c clid10.yaml --max-nested-paths=2 --from file:///home/skhoury/clid-10/ docker://localhost:5000/clid10/max --dest-tls-verify=false with --max-nested-paths - generates IDMS only, with repository scope idms-max.yaml
./bin/oc-mirror --v2 -c clid10.yaml --max-nested-paths=2 --generate-itms --from file:///home/skhoury/clid-10/ docker://localhost:5000/clid10/maxitms --dest-tls-verify=false with --max-nested-paths and --generate-itms - generates IDMS and ITMS, with repository scope idms-maxitms.yaml and itms-maxitms.yaml

Expected Outcome

idms-noflags.yaml

apiVersion: config.openshift.io/v1
kind: ImageDigestMirrorSet
metadata:
 creationTimestamp: null
 name: idms-2024-01-18t15-33-28z
spec:
 imageDigestMirrors:
 - mirrors:
   - localhost:5000/clid10/noflag/kubebuilder
   source: gcr.io/kubebuilder
 - mirrors:
   - localhost:5000/clid10/noflag/cockroachdb
   source: quay.io/cockroachdb
 - mirrors:
   - localhost:5000/clid10/noflag/helmoperators
   source: quay.io/helmoperators
 - mirrors:
   - localhost:5000/clid10/noflag/openshift-community-operators
   source: quay.io/openshift-community-operators
status: {}

idms-tagflag.yaml

apiVersion: config.openshift.io/v1
kind: ImageDigestMirrorSet
metadata:
 creationTimestamp: null
 name: idms-2024-01-18t15-50-26z
spec:
 imageDigestMirrors:
 - mirrors:
   - localhost:5000/clid10/itms/openshift-community-operators
   source: quay.io/openshift-community-operators
status: {}

itms-tagflag.yaml

apiVersion: config.openshift.io/v1
kind: ImageTagMirrorSet
metadata:
 creationTimestamp: null
 name: itms-2024-01-18t15-50-26z
spec:
 imageTagMirrors:
 - mirrors:
   - localhost:5000/clid10/itms/helmoperators
   source: quay.io/helmoperators
 - mirrors:
   - localhost:5000/clid10/itms/kubebuilder
   source: gcr.io/kubebuilder
 - mirrors:
   - localhost:5000/clid10/itms/cockroachdb
   source: quay.io/cockroachdb
status: {}

idms-max.yaml

apiVersion: config.openshift.io/v1
kind: ImageDigestMirrorSet
metadata:
 creationTimestamp: null
 name: idms-2024-01-18t16-19-16z
spec:
 imageDigestMirrors:
 - mirrors:
   - localhost:5000/clid10/max-kubebuilder-kube-rbac-proxy
   source: gcr.io/kubebuilder/kube-rbac-proxy
 - mirrors:
   - localhost:5000/clid10/max-cockroachdb-cockroach-helm-operator
   source: quay.io/cockroachdb/cockroach-helm-operator
 - mirrors:
   - localhost:5000/clid10/max-helmoperators-cockroachdb
   source: quay.io/helmoperators/cockroachdb
 - mirrors:
   - localhost:5000/clid10/max-openshift-community-operators-cockroachdb
   source: quay.io/openshift-community-operators/cockroachdb
status: {}

itms-maxitms.yaml

apiVersion: config.openshift.io/v1
kind: ImageTagMirrorSet
metadata:
 creationTimestamp: null
 name: itms-2024-01-18t16-23-27z
spec:
 imageTagMirrors:
 - mirrors:
   - localhost:5000/clid10/maxitms-kubebuilder-kube-rbac-proxy
   source: gcr.io/kubebuilder/kube-rbac-proxy
 - mirrors:
   - localhost:5000/clid10/maxitms-cockroachdb-cockroach-helm-operator
   source: quay.io/cockroachdb/cockroach-helm-operator
 - mirrors:
   - localhost:5000/clid10/maxitms-helmoperators-cockroachdb
   source: quay.io/helmoperators/cockroachdb
status: {}

idms-maxitms.yaml

apiVersion: config.openshift.io/v1
kind: ImageDigestMirrorSet
metadata:
 creationTimestamp: null
 name: idms-2024-01-18t16-23-27z
spec:
 imageDigestMirrors:
 - mirrors:
   - localhost:5000/clid10/maxitms-openshift-community-operators-cockroachdb
   source: quay.io/openshift-community-operators/cockroachdb
status: {}

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

1 similar comment
@openshift-ci-robot
Copy link

openshift-ci-robot commented Jan 22, 2024

@sherine-k: This pull request references CLID-10 which is a valid jira issue.

In response to this:

Description

This PR contains implementation for CLID-10 (see jira for detailed design) :

  • ImageTagMirrorSet : enabled by flag --generate-itms
  • ImageDigestMirrorSet: with the following behavior
  • Generated by default (when no --generate-itms flag): Contains source and mirrors for all mirrored images.
  • Contains source and mirror paths namespace scoped, unless --max-nested-paths has been used
  • Contains source and mirrors only for images by digest if --generate-itms is used
  • Excluded: CatalogSource for operator catalogs will be in a different PR CLID-10: Generate CatalogSource files #784

Fixes # CLID-10.

Regarding IDMS/ITMS generation, a few questions are still pending in CLID-10 (see comments):

  • By default, oc-mirror will generate IDMS only. Images referenced by tag will be added to that IDMS. If a cluster deploys such an IDMS, and references those images by tag, the mirror will not be used. Therefore, should we:
  • keep the specified behavior, but and display warnings for images by tag (Implemented)
  • By default generate IDMS and ITMS (images by tag can therefore benefit from the mirrors)
  • Keep the default to generate IDMS only, exclude images by tag and display warnings for them
  • To be confirmed: In this PR, we stop resolving tags to digest prior to mirroring. In V1, oc-mirror was removing replacing tags by digests before handing mappings.txt to the oc code, which in turn, created a fake tag (derived from the digest value). This V1 behavior should not be used for V2, because we have ITMS, because we don't call oc, and because we don't rebuild catalogs (the reference for the image is not modified in the declarative config).

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

Prior to testing, create the file clid10.yaml with content:

kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v1alpha2
archiveSize: 1
mirror:
 additionalImages:
 - name: "gcr.io/kubebuilder/kube-rbac-proxy:v0.5.0"
 - name: "quay.io/cockroachdb/cockroach-helm-operator:6.0.0"
 - name: "quay.io/helmoperators/cockroachdb:v5.0.3"
 - name: "quay.io/helmoperators/cockroachdb:v5.0.4"
 - name: "quay.io/openshift-community-operators/cockroachdb@sha256:a5d4f4467250074216eb1ba1c36e06a3ab797d81c431427fc2aca97ecaf4e9d8"
 - name: "quay.io/openshift-community-operators/cockroachdb@sha256:d3016b1507515fc7712f9c47fd9082baf9ccb070aaab58ed0ef6e5abdedde8ba"
 - name: "quay.io/openshift-community-operators/cockroachdb@sha256:f42337e7b85a46d83c94694638e2312e10ca16a03542399a65ba783c94a32b63"

Next, perform a mirrorToDisk

./bin/oc-mirror --v2 -c clid10.yaml file:///home/skhoury/clid-10
Test Description Outputs
./bin/oc-mirror --v2 -c clid10.yaml --from file:///home/skhoury/clid-10/ docker://localhost:5000/clid10/noflag --dest-tls-verify=false Nominal use case - IDMS generation only - All tags resolved to digest - IDMS with namespace scope idms-noflags.yaml (see below)
images pushed: x,y,z
./bin/oc-mirror --v2 -c clid10.yaml --generate-itms --from file:///home/skhoury/clid-10/ docker://localhost:5000/clid10/itms --dest-tls-verify=false with --generate-itms - generates ITMS and IDMS files, with namespace scope idms-tagflag.yaml + itms-tagflag.yaml
./bin/oc-mirror --v2 -c clid10.yaml --max-nested-paths=2 --from file:///home/skhoury/clid-10/ docker://localhost:5000/clid10/max --dest-tls-verify=false with --max-nested-paths - generates IDMS only, with repository scope idms-max.yaml
./bin/oc-mirror --v2 -c clid10.yaml --max-nested-paths=2 --generate-itms --from file:///home/skhoury/clid-10/ docker://localhost:5000/clid10/maxitms --dest-tls-verify=false with --max-nested-paths and --generate-itms - generates IDMS and ITMS, with repository scope idms-maxitms.yaml and itms-maxitms.yaml

Expected Outcome

idms-noflags.yaml

apiVersion: config.openshift.io/v1
kind: ImageDigestMirrorSet
metadata:
 creationTimestamp: null
 name: idms-2024-01-18t15-33-28z
spec:
 imageDigestMirrors:
 - mirrors:
   - localhost:5000/clid10/noflag/kubebuilder
   source: gcr.io/kubebuilder
 - mirrors:
   - localhost:5000/clid10/noflag/cockroachdb
   source: quay.io/cockroachdb
 - mirrors:
   - localhost:5000/clid10/noflag/helmoperators
   source: quay.io/helmoperators
 - mirrors:
   - localhost:5000/clid10/noflag/openshift-community-operators
   source: quay.io/openshift-community-operators
status: {}

idms-tagflag.yaml

apiVersion: config.openshift.io/v1
kind: ImageDigestMirrorSet
metadata:
 creationTimestamp: null
 name: idms-2024-01-18t15-50-26z
spec:
 imageDigestMirrors:
 - mirrors:
   - localhost:5000/clid10/itms/openshift-community-operators
   source: quay.io/openshift-community-operators
status: {}

itms-tagflag.yaml

apiVersion: config.openshift.io/v1
kind: ImageTagMirrorSet
metadata:
 creationTimestamp: null
 name: itms-2024-01-18t15-50-26z
spec:
 imageTagMirrors:
 - mirrors:
   - localhost:5000/clid10/itms/helmoperators
   source: quay.io/helmoperators
 - mirrors:
   - localhost:5000/clid10/itms/kubebuilder
   source: gcr.io/kubebuilder
 - mirrors:
   - localhost:5000/clid10/itms/cockroachdb
   source: quay.io/cockroachdb
status: {}

idms-max.yaml

apiVersion: config.openshift.io/v1
kind: ImageDigestMirrorSet
metadata:
 creationTimestamp: null
 name: idms-2024-01-18t16-19-16z
spec:
 imageDigestMirrors:
 - mirrors:
   - localhost:5000/clid10/max-kubebuilder-kube-rbac-proxy
   source: gcr.io/kubebuilder/kube-rbac-proxy
 - mirrors:
   - localhost:5000/clid10/max-cockroachdb-cockroach-helm-operator
   source: quay.io/cockroachdb/cockroach-helm-operator
 - mirrors:
   - localhost:5000/clid10/max-helmoperators-cockroachdb
   source: quay.io/helmoperators/cockroachdb
 - mirrors:
   - localhost:5000/clid10/max-openshift-community-operators-cockroachdb
   source: quay.io/openshift-community-operators/cockroachdb
status: {}

itms-maxitms.yaml

apiVersion: config.openshift.io/v1
kind: ImageTagMirrorSet
metadata:
 creationTimestamp: null
 name: itms-2024-01-18t16-23-27z
spec:
 imageTagMirrors:
 - mirrors:
   - localhost:5000/clid10/maxitms-kubebuilder-kube-rbac-proxy
   source: gcr.io/kubebuilder/kube-rbac-proxy
 - mirrors:
   - localhost:5000/clid10/maxitms-cockroachdb-cockroach-helm-operator
   source: quay.io/cockroachdb/cockroach-helm-operator
 - mirrors:
   - localhost:5000/clid10/maxitms-helmoperators-cockroachdb
   source: quay.io/helmoperators/cockroachdb
status: {}

idms-maxitms.yaml

apiVersion: config.openshift.io/v1
kind: ImageDigestMirrorSet
metadata:
 creationTimestamp: null
 name: idms-2024-01-18t16-23-27z
spec:
 imageDigestMirrors:
 - mirrors:
   - localhost:5000/clid10/maxitms-openshift-community-operators-cockroachdb
   source: quay.io/openshift-community-operators/cockroachdb
status: {}

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@sherine-k sherine-k force-pushed the CLID-10-2 branch 3 times, most recently from dd13288 to a3e0b53 Compare January 22, 2024 17:26
@openshift-ci-robot
Copy link

openshift-ci-robot commented Jan 22, 2024

@sherine-k: This pull request references CLID-10 which is a valid jira issue.

In response to this:

Description

This PR contains implementation for CLID-10 :

  • CatalogSource for operator catalogs
  • It introduces targetCatalogSourceTemplate to the imageSetConfig, in order to allow the user customization of the catalogSource
  • if not provided, the catalogSource will not contain any updateStrategy (polling interval)
  • It removes deprecated fields targetName and originalRef from the imageSetConfig

Fixes # CLID-10.

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

Prior to testing, create the file clid10.yaml with content:

kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v1alpha2
archiveSize: 1
mirror:
 operators:
 - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.14
   packages:
   - name: aws-load-balancer-operator
   targetCatalogSourceTemplate: "/home/skhoury/go/src/github.com/openshift/oc-mirror/v2/tests/catalog-source_template.yaml"

And the catalog-source_template.yaml

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
 name: totalRubbish
 namespace: openshift-marketplace
spec:
 image: totalRubbish
 sourceType: grpc
 updateStrategy:
   registryPoll: 
     interval: 30m0s

Next, perform a mirrorToDisk + DiskToMirror

./bin/oc-mirror --v2 -c clid10.yaml file:///home/skhoury/clid-10
./bin/oc-mirror --v2 -c clid10.yaml --from file:///home/skhoury/clid-10 docker://localhost:5000 --dest-tls-verify=false

Expected Outcome

The catalogSource file is generated under the working directory, under subfolder cluster-resources. The catalog source file name is derived from the catalog name, with cs- as prefix, and the date as suffix.

Ex: cs-redhat-operator-index-2024-01-22t17-37-25z.yaml

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
 creationTimestamp: null
 name: cs-redhat-operator-index-2024-01-22t17-37-25z
 namespace: openshift-marketplace
spec:
 icon: {}
 image: localhost:5000/clid10/redhat/redhat-operator-index:v4.14
 sourceType: grpc
 updateStrategy:
   registryPoll:
     interval: 30m0s
status: {}

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link

openshift-ci-robot commented Jan 23, 2024

@sherine-k: This pull request references CLID-10 which is a valid jira issue.

In response to this:

Description

This PR contains implementation for CLID-10 :

  • CatalogSource for operator catalogs
  • It introduces targetCatalogSourceTemplate to the imageSetConfig, in order to allow the user customization of the catalogSource
  • if not provided, the catalogSource will not contain any updateStrategy (polling interval)
  • when provided, oc-mirror will attempt to use the template as a base for catalogSource generation, and in case of failure, it will failover to generation without template with a warning in the logs
  • It removes deprecated fields targetName and originalRef from the imageSetConfig

Fixes # CLID-10.

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

Prior to testing, create the file clid10.yaml with content:

kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v1alpha2
archiveSize: 1
mirror:
 operators:
 - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.14
   packages:
   - name: aws-load-balancer-operator
   targetCatalogSourceTemplate: "/home/skhoury/go/src/github.com/openshift/oc-mirror/v2/tests/catalog-source_template.yaml"

And the catalog-source_template.yaml

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
 name: totalRubbish
 namespace: openshift-marketplace
spec:
 image: totalRubbish
 sourceType: grpc
 updateStrategy:
   registryPoll: 
     interval: 30m0s

Next, perform a mirrorToDisk + DiskToMirror

./bin/oc-mirror --v2 -c clid10.yaml file:///home/skhoury/clid-10
./bin/oc-mirror --v2 -c clid10.yaml --from file:///home/skhoury/clid-10 docker://localhost:5000 --dest-tls-verify=false

Expected Outcome

The catalogSource file is generated under the working directory, under subfolder cluster-resources. The catalog source file name is derived from the catalog name, with cs- as prefix, and the date as suffix.

Ex: cs-redhat-operator-index-2024-01-22t17-37-25z.yaml

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
 creationTimestamp: null
 name: cs-redhat-operator-index-2024-01-22t17-37-25z
 namespace: openshift-marketplace
spec:
 icon: {}
 image: localhost:5000/clid10/redhat/redhat-operator-index:v4.14
 sourceType: grpc
 updateStrategy:
   registryPoll:
     interval: 30m0s
status: {}

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link

openshift-ci-robot commented Jan 24, 2024

@sherine-k: This pull request references CLID-10 which is a valid jira issue.

In response to this:

Description

This PR contains implementation for CLID-10 :

  • CatalogSource for operator catalogs
  • It introduces targetCatalogSourceTemplate to the imageSetConfig, in order to allow the user customization of the catalogSource
  • if not provided, the catalogSource will not contain any updateStrategy (polling interval)
  • when provided, oc-mirror will attempt to use the template as a base for catalogSource generation, and in case of failure, it will failover to generation without template with a warning in the logs
  • It removes deprecated fields targetName and originalRef from the imageSetConfig

Fixes # CLID-10.

PS: catalogSource resource name and filename use a different naming convention than v1: "cs-" + catalogRepository + "-" + date . We might need to reconsider this strategy based on how customers use these resources in the cluster. ie. Do they deploy the new catalogSource and delete the old? do they expect an update of the existing resource?

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

Prior to testing, create the file clid10.yaml with content:

kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v1alpha2
archiveSize: 1
mirror:
 operators:
 - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.14
   packages:
   - name: aws-load-balancer-operator
   targetCatalogSourceTemplate: "/home/skhoury/go/src/github.com/openshift/oc-mirror/v2/tests/catalog-source_template.yaml"

And the catalog-source_template.yaml

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
 name: totalRubbish
 namespace: openshift-marketplace
spec:
 image: totalRubbish
 sourceType: grpc
 updateStrategy:
   registryPoll: 
     interval: 30m0s

Next, perform a mirrorToDisk + DiskToMirror

./bin/oc-mirror --v2 -c clid10.yaml file:///home/skhoury/clid-10
./bin/oc-mirror --v2 -c clid10.yaml --from file:///home/skhoury/clid-10 docker://localhost:5000 --dest-tls-verify=false

Expected Outcome

The catalogSource file is generated under the working directory, under subfolder cluster-resources. The catalog source file name is derived from the catalog name, with cs- as prefix, and the date as suffix.

Ex: cs-redhat-operator-index-2024-01-22t17-37-25z.yaml

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
 creationTimestamp: null
 name: cs-redhat-operator-index-2024-01-22t17-37-25z
 namespace: openshift-marketplace
spec:
 icon: {}
 image: localhost:5000/clid10/redhat/redhat-operator-index:v4.14
 sourceType: grpc
 updateStrategy:
   registryPoll:
     interval: 30m0s
status: {}

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@sherine-k
Copy link
Contributor Author

/jira refresh

@openshift-ci-robot
Copy link

openshift-ci-robot commented Jan 24, 2024

@sherine-k: This pull request references CLID-10 which is a valid jira issue.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@sherine-k
Copy link
Contributor Author

/hold
wait for PR #783

@openshift-ci openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 29, 2024
pathComponents := strings.Split(catalogSpec.PathComponent, "/")
catalogRepository := pathComponents[len(pathComponents)-1]
catalogSourceName := "cs-" + catalogRepository + "-" + csSuffix
errs := validation.IsDNS1035Label(catalogSourceName)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catching finding this validation

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed !!!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed !!!

return ""
}

func (o *ClusterResourcesGenerator) generateCatalogSource(catalogRef string, catalogSourceTemplateFile string) error {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really like the validations happening on RFC here.

@@ -276,7 +276,19 @@ func getRelatedImageByDefaultChannel(log clog.PluggableLoggerInterface, olm []v1
if bundles[obj.Name] {
log.Debug("config bundle: %d %v", i, obj.Name)
log.Trace("config relatedImages: %d %v", i, obj.RelatedImages)
relatedImages[obj.Name] = obj.RelatedImages
riList := relatedImages[obj.Name]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change needs to be on CLID-14 so I would prefer to merge CLID-10 first.

Comment on lines +331 to +351
toBeAdded := false
if bundles[obj.Name] && !pkg.Full {
log.Debug("config bundle: %d %v", i, obj.Name)
log.Trace("config relatedImages: %d %v", i, obj.RelatedImages)
relatedImages[obj.Name] = obj.RelatedImages
toBeAdded = true
}
// add all bundles
if pkg.Full {
relatedImages[obj.Name] = obj.RelatedImages
if pkg.Full || toBeAdded {
riList := relatedImages[obj.Name]
if riList == nil {
riList = []v1alpha3.RelatedImage{}
}
for _, ri := range obj.RelatedImages {
if ri.Image == obj.Image {
ri.Type = v1alpha2.TypeOperatorBundle
} else {
ri.Type = v1alpha2.TypeOperatorRelatedImage
}
riList = append(riList, ri)
}
relatedImages[obj.Name] = riList
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change needs to be on CLID-14 so I would prefer to merge CLID-10 first.

Copy link
Contributor

@aguidirh aguidirh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jan 29, 2024
Copy link
Contributor

@lmzuccarelli lmzuccarelli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sherine-k - as usual great work

I left some nits about the messaging

v2/pkg/clusterresources/clusterresources.go Outdated Show resolved Hide resolved
pathComponents := strings.Split(catalogSpec.PathComponent, "/")
catalogRepository := pathComponents[len(pathComponents)-1]
catalogSourceName := "cs-" + catalogRepository + "-" + csSuffix
errs := validation.IsDNS1035Label(catalogSourceName)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed !!!

pathComponents := strings.Split(catalogSpec.PathComponent, "/")
catalogRepository := pathComponents[len(pathComponents)-1]
catalogSourceName := "cs-" + catalogRepository + "-" + csSuffix
errs := validation.IsDNS1035Label(catalogSourceName)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed !!!

v2/pkg/clusterresources/clusterresources.go Outdated Show resolved Hide resolved
v2/pkg/clusterresources/clusterresources.go Outdated Show resolved Hide resolved
v2/pkg/clusterresources/clusterresources.go Outdated Show resolved Hide resolved
v2/pkg/clusterresources/clusterresources.go Outdated Show resolved Hide resolved
Copy link

openshift-ci bot commented Jan 31, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lmzuccarelli, sherine-k

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [lmzuccarelli,sherine-k]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

CLID-10: targetCatalog replaces deprecated field targetName(removed) for GetUniqueName.

CLID-10,CFE-824: Add updateStrategy to generated catalog source

Failover to regular catalogSource generation if template generation fails

V2 - Modify naming convention for catalogSource
@openshift-ci openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Jan 31, 2024
@sherine-k
Copy link
Contributor Author

/unhold

@openshift-ci openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 31, 2024
@sherine-k
Copy link
Contributor Author

/unhold

Copy link

openshift-ci bot commented Jan 31, 2024

@sherine-k: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@aguidirh
Copy link
Contributor

/lgtm

@aguidirh
Copy link
Contributor

/label acknowledge-critical-fixes-only

@openshift-ci openshift-ci bot added acknowledge-critical-fixes-only Indicates if the issuer of the label is OK with the policy. lgtm Indicates that a PR is ready to be merged. labels Jan 31, 2024
@openshift-merge-bot openshift-merge-bot bot merged commit 3b94f36 into openshift:main Jan 31, 2024
5 checks passed
aguidirh pushed a commit to aguidirh/oc-mirror that referenced this pull request Jan 31, 2024
CLID-10: targetCatalog replaces deprecated field targetName(removed) for GetUniqueName.

CLID-10,CFE-824: Add updateStrategy to generated catalog source

Failover to regular catalogSource generation if template generation fails

V2 - Modify naming convention for catalogSource
@openshift-bot
Copy link
Contributor

[ART PR BUILD NOTIFIER]

This PR has been included in build oc-mirror-plugin-container-v4.16.0-202401311812.p0.g3b94f36.assembly.stream for distgit oc-mirror-plugin.
All builds following this will include this PR.

@cdjohnson
Copy link

By looking at the PR, I can see that the CatalogSourceTemplate that pretty much anything in the CatlogSource metadata and spec can be updated except for 4 fields:

	obj.Name = catalogSourceName
	obj.Namespace = "openshift-marketplace"
	obj.Spec.SourceType = "grpc"
	obj.Spec.Image = image

It would be helpful if the contract for "what" in the template is designed to be templatable vs. not is clearly documented. Especially hard-coding the namespace and format of the auto-generated name.

I personally think those options should be overridable, but in my use cases, I would treat the generated CatalogSource as itself a template anyway, using Kustomize to override the namespace and name to my own purposes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
acknowledge-critical-fixes-only Indicates if the issuer of the label is OK with the policy. approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants