From 79074476a1c4ac91ecbdbc18fa49f3c4e9d3d4eb Mon Sep 17 00:00:00 2001 From: jmccormick2001 Date: Wed, 29 Apr 2020 12:04:18 -0500 Subject: [PATCH 01/14] initial version --- enhancements/scorecard-metadata.md | 100 +++++++++++++++++++++++++++++ 1 file changed, 100 insertions(+) create mode 100644 enhancements/scorecard-metadata.md diff --git a/enhancements/scorecard-metadata.md b/enhancements/scorecard-metadata.md new file mode 100644 index 000000000..0c34267ea --- /dev/null +++ b/enhancements/scorecard-metadata.md @@ -0,0 +1,100 @@ +--- +title: scorecard-metadata +authors: + - "@jemccorm" + - "@jlanford" +reviewers: + - TBD +approvers: + - TBD +creation-date: 2020-04-29 +last-updated: 2020-04-29 +status: provisional +--- + +# scorecard-metadata + +## Release Signoff Checklist + +- [ ] Enhancement is `implementable` +- [ ] Design details are appropriately documented from clear requirements +- [ ] Test plan is defined +- [ ] Graduation criteria for dev preview, tech preview, GA + +## Summary + +This enhancement is to define a place within the bundle format whereby +scorecard metadata could be added. This will allow tests to be co-located +with the normal operator content found in a bundle. This enhancement +suggests naming and paths for the scorecard metadata within the bundle. + +## Motivation + +The purpose of this enhancement is to define a location for scorecard +and other test assets to be included into the bundle. This allows +test frameworks and tools a known place to search for test assets +within a bundle and reduces confusion to test developers as to +where test assets will reside. + +### Goals + +This enhancement will be used by the scorecard feature within the operator-sdk +initially as part of a major redesign of scorecard functionality. Other +test frameworks apart from operator-sdk or scorecard could leverage this +enhancement as well. + +### Non-Goals + +This enhancement is scoped to define the location within the bundle for +this sort of metadata and get consensus on names and paths. Follow on +work would be required to implement tooling that lets a user inject +their test assets into a bundle image as well as extract this metadata. + +## Design Details + +The bundle format would allow for test metadata to be +included in the following locations within a bundle: + +``` +bundle/ +├── manifests +│ ├── cache.example.com_memcacheds_crd.yaml +│ └── memcached-operator.v0.0.1.clusterserviceversion.yaml +├── metadata +│ └── annotations.yaml +└── tests + └── scorecard + ├── config.yaml + └── kuttl + ├── example1 + └── example2 +``` + +In the above proposed layout, the test metadata resides under a new +directory called `tests`. Below that directory is free form but +in the case of operator-sdk scorecard, it would populate this +directory with a subdirectory named `scorecard` as depicted. + +## Proposal + +### User Stories + +#### Story 1 + +As a test developer I would like to be able to include and replace test metadata into +operator bundles. + +#### Story 2 + +As a test developer I would like to extract test metadata from +operator bundles so as to execute tests against the operator and its bundle contents. + +### Risks and Mitigations + +You would not want to define locations in the bundle that might +conflict with other metadata or usage of the bundle. + +Some sort of check would need to be created when inserting test +metadata within a bundle to allow a maximum size of metadata +that needs to be determined. + From 31f7c4cc1deb48fc3e56ad4a4aee3c6f025207f8 Mon Sep 17 00:00:00 2001 From: Jeff McCormick Date: Tue, 12 May 2020 16:31:34 -0500 Subject: [PATCH 02/14] Update enhancements/scorecard-metadata.md Co-authored-by: Eric Stroczynski --- enhancements/scorecard-metadata.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/enhancements/scorecard-metadata.md b/enhancements/scorecard-metadata.md index 0c34267ea..bb409929c 100644 --- a/enhancements/scorecard-metadata.md +++ b/enhancements/scorecard-metadata.md @@ -31,7 +31,7 @@ suggests naming and paths for the scorecard metadata within the bundle. ## Motivation The purpose of this enhancement is to define a location for scorecard -and other test assets to be included into the bundle. This allows +and other test assets to be included into the bundle. This definition test frameworks and tools a known place to search for test assets within a bundle and reduces confusion to test developers as to where test assets will reside. @@ -97,4 +97,3 @@ conflict with other metadata or usage of the bundle. Some sort of check would need to be created when inserting test metadata within a bundle to allow a maximum size of metadata that needs to be determined. - From 880f90f4b8380f1bab25784fb68edd27d57b6f46 Mon Sep 17 00:00:00 2001 From: Jeff McCormick Date: Tue, 12 May 2020 16:31:45 -0500 Subject: [PATCH 03/14] Update enhancements/scorecard-metadata.md Co-authored-by: Eric Stroczynski --- enhancements/scorecard-metadata.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/enhancements/scorecard-metadata.md b/enhancements/scorecard-metadata.md index bb409929c..ef4722d8d 100644 --- a/enhancements/scorecard-metadata.md +++ b/enhancements/scorecard-metadata.md @@ -59,7 +59,7 @@ included in the following locations within a bundle: bundle/ ├── manifests │ ├── cache.example.com_memcacheds_crd.yaml -│ └── memcached-operator.v0.0.1.clusterserviceversion.yaml +│ └── memcached-operator.clusterserviceversion.yaml ├── metadata │ └── annotations.yaml └── tests From 32a7e7a685ceac330f5926dffdcae36003804de2 Mon Sep 17 00:00:00 2001 From: Jeff McCormick Date: Tue, 12 May 2020 16:31:57 -0500 Subject: [PATCH 04/14] Update enhancements/scorecard-metadata.md Co-authored-by: Eric Stroczynski --- enhancements/scorecard-metadata.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/enhancements/scorecard-metadata.md b/enhancements/scorecard-metadata.md index ef4722d8d..8c85a0b8f 100644 --- a/enhancements/scorecard-metadata.md +++ b/enhancements/scorecard-metadata.md @@ -32,7 +32,7 @@ suggests naming and paths for the scorecard metadata within the bundle. The purpose of this enhancement is to define a location for scorecard and other test assets to be included into the bundle. This definition -test frameworks and tools a known place to search for test assets +informs test frameworks and tools of a well-known location to search for test assets within a bundle and reduces confusion to test developers as to where test assets will reside. From bed85202fd713e43d265d5f497674fd01b2641ef Mon Sep 17 00:00:00 2001 From: Jeff McCormick Date: Thu, 14 May 2020 13:14:26 -0500 Subject: [PATCH 05/14] add updates based on review comments and discussions --- enhancements/scorecard-metadata.md | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/enhancements/scorecard-metadata.md b/enhancements/scorecard-metadata.md index 8c85a0b8f..f8de2ad45 100644 --- a/enhancements/scorecard-metadata.md +++ b/enhancements/scorecard-metadata.md @@ -8,7 +8,7 @@ reviewers: approvers: - TBD creation-date: 2020-04-29 -last-updated: 2020-04-29 +last-updated: 2020-05-14 status: provisional --- @@ -75,6 +75,22 @@ directory called `tests`. Below that directory is free form but in the case of operator-sdk scorecard, it would populate this directory with a subdirectory named `scorecard` as depicted. +The scorecard test label annotations for the bundle would be as follows: +``` +annotations: + operators.operatorframework.io.test.mediatype.v1: "scorecard+v1" + operators.operatorframework.io.test.config.v1: "/tests/scorecard/config.yaml" + # optional + operators.operatorframework.io.test.type.v1: "kuttl" +``` + +For the purpose of this enhancement, scorecard would assume that its test +metadata would be in the same bundle image as the operator image. + +The operator-registry API would need to be updated to support an ability +to add custom metadata such as what scorecard would need into a bundle +image. + ## Proposal ### User Stories From f2a892b083c56e03a179cf80f762b247a5182123 Mon Sep 17 00:00:00 2001 From: Jeff McCormick Date: Tue, 19 May 2020 14:47:05 -0500 Subject: [PATCH 06/14] update to show how other test frameworks could be represented in labels --- enhancements/scorecard-metadata.md | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/enhancements/scorecard-metadata.md b/enhancements/scorecard-metadata.md index f8de2ad45..5ade57c2c 100644 --- a/enhancements/scorecard-metadata.md +++ b/enhancements/scorecard-metadata.md @@ -8,7 +8,7 @@ reviewers: approvers: - TBD creation-date: 2020-04-29 -last-updated: 2020-05-14 +last-updated: 2020-05-19 status: provisional --- @@ -63,6 +63,7 @@ bundle/ ├── metadata │ └── annotations.yaml └── tests + └── other-tests └── scorecard ├── config.yaml └── kuttl @@ -78,10 +79,14 @@ directory with a subdirectory named `scorecard` as depicted. The scorecard test label annotations for the bundle would be as follows: ``` annotations: - operators.operatorframework.io.test.mediatype.v1: "scorecard+v1" - operators.operatorframework.io.test.config.v1: "/tests/scorecard/config.yaml" + operators.operatorframework.io.test.scorecard.mediatype.v1: "scorecard+v1" + operators.operatorframework.io.test.scorecard.config.v1: "/tests/scorecard/config.yaml" # optional - operators.operatorframework.io.test.type.v1: "kuttl" + operators.operatorframework.io.test.scorecard.type.v1: "kuttl" + operators.operatorframework.io.test.other-tests.mediatype.v1: "other-tests+v1" + operators.operatorframework.io.test.other-tests.config.v1: "/tests/other-tests/setup.yaml" + # optional + operators.operatorframework.io.test.other-tests.type.v1: "other-tests" ``` For the purpose of this enhancement, scorecard would assume that its test @@ -97,13 +102,14 @@ image. #### Story 1 -As a test developer I would like to be able to include and replace test metadata into -operator bundles. +As a test developer I would like to be able to include and replace test +metadata into operator bundles. #### Story 2 As a test developer I would like to extract test metadata from -operator bundles so as to execute tests against the operator and its bundle contents. +operator bundles so as to execute tests against the operator and its i +bundle contents. ### Risks and Mitigations From 16e039cdd818a5fa54f78d522dd6bdb066ad5f03 Mon Sep 17 00:00:00 2001 From: jmccormick2001 Date: Thu, 21 May 2020 07:59:51 -0500 Subject: [PATCH 07/14] remove kuttl annotations --- enhancements/scorecard-metadata.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/enhancements/scorecard-metadata.md b/enhancements/scorecard-metadata.md index 5ade57c2c..b5b310fc4 100644 --- a/enhancements/scorecard-metadata.md +++ b/enhancements/scorecard-metadata.md @@ -81,12 +81,8 @@ The scorecard test label annotations for the bundle would be as follows: annotations: operators.operatorframework.io.test.scorecard.mediatype.v1: "scorecard+v1" operators.operatorframework.io.test.scorecard.config.v1: "/tests/scorecard/config.yaml" - # optional - operators.operatorframework.io.test.scorecard.type.v1: "kuttl" operators.operatorframework.io.test.other-tests.mediatype.v1: "other-tests+v1" operators.operatorframework.io.test.other-tests.config.v1: "/tests/other-tests/setup.yaml" - # optional - operators.operatorframework.io.test.other-tests.type.v1: "other-tests" ``` For the purpose of this enhancement, scorecard would assume that its test From ec58de0ba602fe11402bf7717dda5304b9fdc685 Mon Sep 17 00:00:00 2001 From: Jeff McCormick Date: Thu, 28 May 2020 12:36:42 -0500 Subject: [PATCH 08/14] change key to be operators.operatorframework.io.scorecard --- enhancements/scorecard-metadata.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/enhancements/scorecard-metadata.md b/enhancements/scorecard-metadata.md index b5b310fc4..2556cf508 100644 --- a/enhancements/scorecard-metadata.md +++ b/enhancements/scorecard-metadata.md @@ -79,10 +79,10 @@ directory with a subdirectory named `scorecard` as depicted. The scorecard test label annotations for the bundle would be as follows: ``` annotations: - operators.operatorframework.io.test.scorecard.mediatype.v1: "scorecard+v1" - operators.operatorframework.io.test.scorecard.config.v1: "/tests/scorecard/config.yaml" - operators.operatorframework.io.test.other-tests.mediatype.v1: "other-tests+v1" - operators.operatorframework.io.test.other-tests.config.v1: "/tests/other-tests/setup.yaml" + operators.operatorframework.io.scorecard.mediatype.v1: "scorecard+v1" + operators.operatorframework.io.scorecard.config.v1: "/tests/scorecard/config.yaml" + operators.operatorframework.io.other-tests.mediatype.v1: "other-tests+v1" + operators.operatorframework.io.other-tests.config.v1: "/tests/other-tests/setup.yaml" ``` For the purpose of this enhancement, scorecard would assume that its test From cba334342c77881f2e1ada793d1d495b818c3be0 Mon Sep 17 00:00:00 2001 From: Jeff McCormick Date: Fri, 29 May 2020 11:32:48 -0500 Subject: [PATCH 09/14] Update enhancements/scorecard-metadata.md Co-authored-by: Eric Stroczynski --- enhancements/scorecard-metadata.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/enhancements/scorecard-metadata.md b/enhancements/scorecard-metadata.md index 2556cf508..0aa2d23ed 100644 --- a/enhancements/scorecard-metadata.md +++ b/enhancements/scorecard-metadata.md @@ -65,7 +65,7 @@ bundle/ └── tests └── other-tests └── scorecard - ├── config.yaml + └── config.yaml └── kuttl ├── example1 └── example2 From f631f5f3ea448cc381ea3f5c5fd924853e024869 Mon Sep 17 00:00:00 2001 From: Jeff McCormick Date: Fri, 29 May 2020 11:36:19 -0500 Subject: [PATCH 10/14] Update scorecard-metadata.md removed the kuttl files from the example, here we assume that we control what is under tests/scorecard so an example of that is not useful to call out. --- enhancements/scorecard-metadata.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/enhancements/scorecard-metadata.md b/enhancements/scorecard-metadata.md index 0aa2d23ed..2d8059a5f 100644 --- a/enhancements/scorecard-metadata.md +++ b/enhancements/scorecard-metadata.md @@ -66,9 +66,7 @@ bundle/ └── other-tests └── scorecard └── config.yaml - └── kuttl - ├── example1 - └── example2 + ``` In the above proposed layout, the test metadata resides under a new From 70904edeaa492d799a414d323559288570a875e7 Mon Sep 17 00:00:00 2001 From: Jeff McCormick Date: Thu, 4 Jun 2020 10:40:02 -0500 Subject: [PATCH 11/14] add note on the API needing an ability to remove custom metadata to support disconnected environments --- enhancements/scorecard-metadata.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/enhancements/scorecard-metadata.md b/enhancements/scorecard-metadata.md index 2d8059a5f..1b3d5089a 100644 --- a/enhancements/scorecard-metadata.md +++ b/enhancements/scorecard-metadata.md @@ -88,7 +88,8 @@ metadata would be in the same bundle image as the operator image. The operator-registry API would need to be updated to support an ability to add custom metadata such as what scorecard would need into a bundle -image. +image. The API would also need to support updating or removing custom +metadata from a bundle image. ## Proposal @@ -96,8 +97,10 @@ image. #### Story 1 -As a test developer I would like to be able to include and replace test -metadata into operator bundles. +As a test developer I would like to be able to include, replace, and +remove test metadata within operator bundles. The ability to remove +test metadata would support disconnected environments where running +tests is not applicable. #### Story 2 From 7d9f3bee22ddf983a780d80477f716cb9b39c1b1 Mon Sep 17 00:00:00 2001 From: Jeff McCormick Date: Thu, 11 Jun 2020 14:27:26 -0500 Subject: [PATCH 12/14] Update enhancements/scorecard-metadata.md Co-authored-by: Evan Cordell --- enhancements/scorecard-metadata.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/enhancements/scorecard-metadata.md b/enhancements/scorecard-metadata.md index 1b3d5089a..5d11931e0 100644 --- a/enhancements/scorecard-metadata.md +++ b/enhancements/scorecard-metadata.md @@ -105,7 +105,7 @@ tests is not applicable. #### Story 2 As a test developer I would like to extract test metadata from -operator bundles so as to execute tests against the operator and its i +operator bundles so as to execute tests against the operator and its bundle contents. ### Risks and Mitigations From 32baea40a8997f9116d9974189de248ce964a02b Mon Sep 17 00:00:00 2001 From: jmccormick2001 Date: Thu, 11 Jun 2020 14:44:37 -0500 Subject: [PATCH 13/14] address PR comments, add user story 3, be more specific as to operator-registry API requirements --- enhancements/scorecard-metadata.md | 34 ++++++++++++++++++++---------- 1 file changed, 23 insertions(+), 11 deletions(-) diff --git a/enhancements/scorecard-metadata.md b/enhancements/scorecard-metadata.md index 5d11931e0..c0d069643 100644 --- a/enhancements/scorecard-metadata.md +++ b/enhancements/scorecard-metadata.md @@ -86,10 +86,16 @@ annotations: For the purpose of this enhancement, scorecard would assume that its test metadata would be in the same bundle image as the operator image. -The operator-registry API would need to be updated to support an ability -to add custom metadata such as what scorecard would need into a bundle -image. The API would also need to support updating or removing custom -metadata from a bundle image. +This proposal would require the operator-registry github repo to +contain a golang API that can be used by the SDK to add scorecard +metadata into a bundle image. + +The SDK scorecard is just one example of the type of metadata +that might be added into the bundle image using the proposed +operator-registry golang API. + +The proposed operator-registry golang API would also need to +support updating or removing custom metadata from a bundle image. ## Proposal @@ -97,16 +103,22 @@ metadata from a bundle image. #### Story 1 -As a test developer I would like to be able to include, replace, and -remove test metadata within operator bundles. The ability to remove -test metadata would support disconnected environments where running -tests is not applicable. +As a developer, I would like to use the operator-registry github repo +golang API to include, replace, and remove test metadata within operator +bundles. The ability to remove test metadata would support +disconnected environments where running tests is not applicable. #### Story 2 -As a test developer I would like to extract test metadata from -operator bundles so as to execute tests against the operator and its -bundle contents. +As a developer I would like to use the operator-registry github repo +golang API to extract test metadata from operator bundles so as to +execute tests against the operator and its bundle contents. + +#### Story 3 + +As a developer I would like to use the operator-registry github repo +golang API to extract test metadata from operator bundles but also +have the ability to control what bundle contents are extracted. ### Risks and Mitigations From 0fbf9e72fc13fc9b3fc3d8e1e7c9b4bc09f26423 Mon Sep 17 00:00:00 2001 From: Jeff McCormick Date: Tue, 16 Jun 2020 15:18:11 -0500 Subject: [PATCH 14/14] add user story 4 about providing the option of mirroring test images like other catalog images --- enhancements/scorecard-metadata.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/enhancements/scorecard-metadata.md b/enhancements/scorecard-metadata.md index c0d069643..ecb81e4eb 100644 --- a/enhancements/scorecard-metadata.md +++ b/enhancements/scorecard-metadata.md @@ -120,6 +120,13 @@ As a developer I would like to use the operator-registry github repo golang API to extract test metadata from operator bundles but also have the ability to control what bundle contents are extracted. +#### Story 4 + +As a openshift user I would like to use the 'oc adm catalog build' +command to optionally include the scorecard test images into the catalog. +This would enable openshift users to mirror the test images like other +operator images using the 'oc adm catalog mirror' command. + ### Risks and Mitigations You would not want to define locations in the bundle that might