Filter out Internal APIs in Installed Operators #3877

bipuladh · 2020-01-07T07:19:33Z

Filter Internal Cards
Filter Internal Tabs in Installed Operators
Filter Create New Button Items
Filter row filters and items shown in the All Instances page.

Example annotation:
operators.operatorframework.io/internal-objects: '["cephblockpools.ceph.rook.io", "cephobjectstores.ceph.rook.io"]'

umangachapagain · 2020-01-08T07:00:14Z

Can you provide an example of CRD/CSV annotations that is used by this filter?

umangachapagain · 2020-01-08T07:01:26Z

Did you also filter it from the Operator Install page (one where you choose channels and namespace)?

cloudbehl · 2020-01-10T06:47:14Z

frontend/packages/operator-lifecycle-manager/src/components/clusterserviceversion.tsx

+  const INTERNAL = 'operators.operatorframework.io/internal-objects';
+  let internals: string = _.get(annotations, INTERNAL);
+  if (internals) {
+    internals = internals.replace(/'/g, '"');


are we doing it because annotations are written this way?

yes. you're right. and JSON doesn't work with '.

Cant this be changed from BE itself or its a convention to do so?
^^ @aruniiird

@spadgett Is this the general convention? Why we are not using the YAML way of creating a list and then using it directly.

@bipuladh can you paste the operators.operatorframework.io/internal-objects value, so we can better understand.

@cloudbehl I have put it in the description.

Cant this be changed from BE itself or its a convention to do so?
^^ @aruniiird

@afreen23 , I have made (and submitted) the changes in the backend PR, red-hat-storage/ocs-operator#385, and it worked (without the special handling of internals string).
But isn't it a good idea (or programming practice) to have a check, thus you don't have to be dependent on BE to provide a correct string? Regardless may have to handle parsing errors.

afreen23 · 2020-01-10T07:00:31Z

frontend/packages/operator-lifecycle-manager/src/components/clusterserviceversion.tsx

@@ -795,24 +814,27 @@ export const ClusterServiceVersionsDetailsPage: React.FC<ClusterServiceVersionsD
  props,
 ) => {
  const instancePagesFor = (obj: ClusterServiceVersionKind) => {
+    const internalObjects = getInternalNames(_.get(obj, 'metadata.annotations'));


Suggested change

const internalObjects = getInternalNames(_.get(obj, 'metadata.annotations'));

const internalObjects = getInternalObjects(_.get(obj, 'metadata.annotations'));

afreen23 · 2020-01-10T07:16:51Z

frontend/packages/operator-lifecycle-manager/src/components/clusterserviceversion.tsx

+  [key: string]: string;
+};
+
+const getInternalNames = (annotations: Annotations) => {


Suggested change

const getInternalNames = (annotations: Annotations) => {

const getInternalObjects = (annotations: Annotations) => {

spadgett · 2020-01-13T19:51:05Z

frontend/packages/operator-lifecycle-manager/src/utils.ts

+  const INTERNAL = 'operators.operatorframework.io/internal-objects';
+  let internals: string = _.get(annotations, INTERNAL);
+  if (internals) {
+    internals = internals.replace(/'/g, '"');


Wait, why is this needed? It should parse as JSON with no changes.

Ohh sorry, I missed you comment. I have the same concern.

#3877 (comment)

ack. I had once gotten a malformed string. Hence put it. But I will remove it.

spadgett · 2020-01-13T19:51:24Z

frontend/packages/operator-lifecycle-manager/src/utils.ts

+
+export const getInternalObjects = (annotations: Annotation): string[] => {
+  const INTERNAL = 'operators.operatorframework.io/internal-objects';
+  let internals: string = _.get(annotations, INTERNAL);


Suggested change

let internals: string = _.get(annotations, INTERNAL);

let internals: string = _.get(annotations, ['operators.operatorframework.io/internal-objects']);

frontend/packages/operator-lifecycle-manager/src/utils.ts

spadgett · 2020-01-13T19:55:14Z

frontend/packages/operator-lifecycle-manager/src/utils.ts

+};
+
+export const isInternalObject = (internalObjects: string[], objectName: string): boolean =>
+  internalObjects.some((obj) => obj.includes(objectName));


Why is this a substring match?

ack. Will make it a strict match.

spadgett · 2020-01-13T19:58:50Z

frontend/packages/operator-lifecycle-manager/src/utils.ts

+  csv: ClusterServiceVersionKind,
+): string[] => {
+  const { owned } = csv.spec.customresourcedefinitions;
+  return owned.filter((obj) => isInternalObject(internalObjects, obj.name)).map((obj) => obj.kind);


kind is not unique since it doesn't contain the API group, so we can't use this helper to hide items by kind

ack. will use referenceForProvidedAPI.

TheRealJon

Just a couple of minor suggestions on the iterative methods we are using to filter and map items.

TheRealJon · 2020-01-13T20:18:24Z

frontend/packages/operator-lifecycle-manager/src/components/operand.tsx

+            .filter((item) => !isInternalObject(internalObjects, item.name))
+            .reduce((acc, crd) => ({ ...acc, [crd.name]: crd.displayName }), {}),


Reduce can be used to filter. No need to filter and then reduce. We can expand the logic in the reduce callback to exclude items that should be filtered.

TheRealJon · 2020-01-13T20:56:25Z

frontend/packages/operator-lifecycle-manager/src/components/clusterserviceversion.tsx

+      ) : (
+        props.crdDescs
+          .filter((crd) => !isInternalObject(internals, crd.name))
+          .map((desc) => (


Suggest using reduce instead of filter -> map.

spadgett · 2020-01-14T13:18:14Z

frontend/packages/operator-lifecycle-manager/src/components/operand.tsx

+            (acc, crd) =>
+              !isInternalObject(internalObjects, crd.name)
+                ? { ...acc, [crd.name]: crd.displayName }
+                : { ...acc },


No reason to make a copy here

Suggested change

: { ...acc },

: acc,

spadgett · 2020-01-14T13:20:43Z

frontend/packages/operator-lifecycle-manager/src/types.ts

@@ -246,3 +246,7 @@ export type OperatorGroupKind = {
    lastUpdated: string;
  };
 } & K8sResourceKind;
+
+export type Annotation = {


I'd use ObjectMetadata['annotations'] or export the type here:

https://github.com/openshift/console/blob/master/frontend/public/module/k8s/index.ts#L39

spadgett · 2020-01-14T13:21:12Z

frontend/packages/operator-lifecycle-manager/src/utils.ts

+    (acc, obj) =>
+      isInternalObject(internalObjects, obj.name)
+        ? [referenceForProvidedAPI(obj), ...acc]
+        : [...acc],


Suggested change

: [...acc],

: acc,

spadgett · 2020-01-14T13:21:57Z

frontend/packages/operator-lifecycle-manager/src/utils.ts

+    try {
+      return JSON.parse(internals);
+    } catch {
+      return [];


We should add a console.error for debugging

spadgett · 2020-01-14T13:22:42Z

frontend/packages/operator-lifecycle-manager/src/utils.ts

+
+export const getInternalObjects = (annotations: Annotation): string[] => {
+  const internals: string = _.get(annotations, 'operators.operatorframework.io/internal-objects');
+  if (internals) {


Prefer early returns, which are easier to reason aboue.

if (!internals) { return []; } // parse json....

spadgett · 2020-01-14T13:24:42Z

frontend/packages/operator-lifecycle-manager/src/utils.ts

+  internalObjects.some((obj) => obj === objectName);
+
+export const internalAPIReferences = (
+  internalObjects: string[],


I'd just parse the internal objects here instead of taking it in as a param. That makes the API simpler.

spadgett · 2020-01-14T13:24:53Z

frontend/packages/operator-lifecycle-manager/src/utils.ts

+export const isInternalObject = (internalObjects: string[], objectName: string): boolean =>
+  internalObjects.some((obj) => obj === objectName);
+
+export const internalAPIReferences = (


Suggested change

export const internalAPIReferences = (

export const getInternalAPIReferences = (

spadgett · 2020-01-14T14:20:54Z

frontend/packages/operator-lifecycle-manager/src/utils.ts

+  internalObjects.some((obj) => obj === objectName);
+
+export const getInternalAPIReferences = (
+  annotations: ObjectMetadata['annotations'],


No need to pass in the annotations. They can be ready from csv

spadgett · 2020-01-14T14:22:09Z

frontend/packages/operator-lifecycle-manager/src/utils.ts

+import { ClusterServiceVersionKind } from './types';
+import { referenceForProvidedAPI } from './components';
+
+export const getInternalObjects = (annotations: ObjectMetadata['annotations']): string[] => {


This is a nit, but I think it's a better abstraction if this takes in the CSV obj instead of the annotations.

spadgett · 2020-01-22T14:43:40Z

This doesn't appear to be simply a timeout. There's an actual JS runtime error in the screenshot:

https://storage.googleapis.com/origin-ci-test/pr-logs/pull/openshift_console/3877/pull-ci-openshift-console-master-e2e-gcp-console/10157/artifacts/e2e-gcp-console/gui_test_screenshots/7b798fb81212e0a923d6c0f3d579bc8f.png

spadgett · 2020-01-22T14:44:29Z

[SEVERE] https://console-openshift-console.apps.ci-op-3834lipi-75d12.origin-ci-int-gce.dev.openshift.com/static/vendors~main-chunk-123dbe3b7bb38cae78e3.min.js 117:71742 TypeError: t.reduce is not a function
    at i (https://console-openshift-console.apps.ci-op-3834lipi-75d12.origin-ci-int-gce.dev.openshift.com/static/operand-chunk-53cd1a5aaac54e1ba847.min.js:1:21823)
    at https://console-openshift-console.apps.ci-op-3834lipi-75d12.origin-ci-int-gce.dev.openshift.com/static/operand-chunk-53cd1a5aaac54e1ba847.min.js:1:5613
    at Li (https://console-openshift-console.apps.ci-op-3834lipi-75d12.origin-ci-int-gce.dev.openshift.com/static/vendors~main-chunk-123dbe3b7bb38cae78e3.min.js:118:55337)
    at Ku (https://console-openshift-console.apps.ci-op-3834lipi-75d12.origin-ci-int-gce.dev.openshift.com/static/vendors~main-chunk-123dbe3b7bb38cae78e3.min.js:118:98261)
    at Bu (https://console-openshift-console.apps.ci-op-3834lipi-75d12.origin-ci-int-gce.dev.openshift.com/static/vendors~main-chunk-123dbe3b7bb38cae78e3.min.js:118:84008)
    at Nu (https://console-openshift-console.apps.ci-op-3834lipi-75d12.origin-ci-int-gce.dev.openshift.com/static/vendors~main-chunk-123dbe3b7bb38cae78e3.min.js:118:81035)
    at Iu (https://console-openshift-console.apps.ci-op-3834lipi-75d12.origin-ci-int-gce.dev.openshift.com/static/vendors~main-chunk-123dbe3b7bb38cae78e3.min.js:118:79608)
    at https://console-openshift-console.apps.ci-op-3834lipi-75d12.origin-ci-int-gce.dev.openshift.com/static/vendors~main-chunk-123dbe3b7bb38cae78e3.min.js:118:41759
    at t.unstable_runWithPriority (https://console-openshift-console.apps.ci-op-3834lipi-75d12.origin-ci-int-gce.dev.openshift.com/static/vendors~main-chunk-123dbe3b7bb38cae78e3.min.js:126:3878)
    at so (https://console-openshift-console.apps.ci-op-3834lipi-75d12.origin-ci-int-gce.dev.openshift.com/static/vendors~main-chunk-123dbe3b7bb38cae78e3.min.js:118:41488)

bipuladh · 2020-01-22T16:45:36Z

/retest

spadgett · 2020-01-22T16:57:08Z

/hold cancel
/retest

spadgett · 2020-01-22T16:57:22Z

Thanks for tracking the test failure down 👍

bipuladh · 2020-01-22T17:50:50Z

/retest

frontend/packages/operator-lifecycle-manager/src/components/clusterserviceversion.tsx

spadgett · 2020-01-22T18:10:33Z

frontend/packages/operator-lifecycle-manager/src/utils.ts

+  internalObjects.some((obj) => obj === objectName);
+
+export const getInternalAPIReferences = (csv: ClusterServiceVersionKind): string[] => {
+  const owned: CRDDescription[] = _.flatten(


Isn't spec.customresourcedefinitions.owned an array? I'd expect it to be simply

const owned: CRDDescription[] = csv?.spec?.customresourcedefinitions?.owned || [];

Yes, you're right. I was doing this because I was taking all CRDs both owned and required. #shithappens ;)

spadgett

/approve
/lgtm

openshift-ci-robot · 2020-01-22T19:56:22Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bipuladh, cloudbehl, jeff-phillips-18, spadgett, TheRealJon, vojtechszocs

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~frontend/packages/operator-lifecycle-manager/OWNERS~~ [jeff-phillips-18,spadgett,vojtechszocs]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

spadgett · 2020-01-22T19:59:17Z

/retest

openshift-bot · 2020-01-22T20:56:58Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-01-22T21:09:31Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-01-22T22:15:08Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-01-22T23:58:38Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-01-23T00:40:56Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-01-23T01:55:53Z

/retest

Please review the full test history for this PR and help us cut down flakes.

bipuladh · 2020-01-23T04:32:25Z

/retest

openshift-bot · 2020-01-23T04:44:36Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. component/olm Related to OLM labels Jan 7, 2020

openshift-ci-robot requested review from jeff-phillips-18 and spadgett January 7, 2020 07:20

bipuladh force-pushed the olm-internal-test branch from be4f834 to 71b98df Compare January 8, 2020 06:16

bipuladh changed the title ~~[WIP] Filter out Internal APIs in Installed Operators~~ Filter out Internal APIs in Installed Operators Jan 8, 2020

openshift-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 8, 2020

cloudbehl reviewed Jan 10, 2020

View reviewed changes

afreen23 suggested changes Jan 10, 2020

View reviewed changes

aruniiird mentioned this pull request Jan 13, 2020

Annotating the internal CRDs red-hat-storage/ocs-operator#385

Merged

bipuladh force-pushed the olm-internal-test branch from 71b98df to f9a2a7d Compare January 13, 2020 19:48

openshift-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jan 13, 2020

spadgett requested changes Jan 13, 2020

View reviewed changes

TheRealJon reviewed Jan 13, 2020

View reviewed changes

bipuladh force-pushed the olm-internal-test branch from f9a2a7d to efac119 Compare January 14, 2020 10:03

bipuladh requested review from spadgett, afreen23, TheRealJon and cloudbehl January 14, 2020 10:07

bipuladh force-pushed the olm-internal-test branch from efac119 to 4843614 Compare January 14, 2020 13:16

spadgett reviewed Jan 14, 2020

View reviewed changes

bipuladh force-pushed the olm-internal-test branch 2 times, most recently from 358e1a0 to 50c8811 Compare January 14, 2020 14:14

spadgett reviewed Jan 14, 2020

View reviewed changes

bipuladh force-pushed the olm-internal-test branch from 50c8811 to d131ec8 Compare January 14, 2020 15:14

bipuladh requested a review from spadgett January 14, 2020 15:21

openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 22, 2020

bipuladh force-pushed the olm-internal-test branch 2 times, most recently from 900e9f7 to a52c456 Compare January 22, 2020 16:03

openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 22, 2020

spadgett reviewed Jan 22, 2020

View reviewed changes

bipuladh force-pushed the olm-internal-test branch from a52c456 to f13372e Compare January 22, 2020 18:23

Added filter for internal objects in Installed Operators

4ee8510

bipuladh force-pushed the olm-internal-test branch from f13372e to 4ee8510 Compare January 22, 2020 18:51

spadgett approved these changes Jan 22, 2020

View reviewed changes

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Jan 22, 2020

openshift-merge-robot merged commit 9f1009b into openshift:master Jan 23, 2020

spadgett added this to the v4.4 milestone Jan 27, 2020

tiraboschi mentioned this pull request Mar 2, 2020

Hide internal CRDs kubevirt/hyperconverged-cluster-operator#479

Merged

andrewballantyne mentioned this pull request Apr 17, 2020

Installing OpenShift Pipelines Instructions openshift/openshift-docs#20746

Merged

	const internalObjects = getInternalNames(_.get(obj, 'metadata.annotations'));
	const internalObjects = getInternalObjects(_.get(obj, 'metadata.annotations'));

	const getInternalNames = (annotations: Annotations) => {
	const getInternalObjects = (annotations: Annotations) => {

	let internals: string = _.get(annotations, INTERNAL);
	let internals: string = _.get(annotations, ['operators.operatorframework.io/internal-objects']);

		.filter((item) => !isInternalObject(internalObjects, item.name))
		.reduce((acc, crd) => ({ ...acc, [crd.name]: crd.displayName }), {}),

	export const internalAPIReferences = (
	export const getInternalAPIReferences = (

Filter out Internal APIs in Installed Operators #3877

Filter out Internal APIs in Installed Operators #3877

Conversation

bipuladh commented Jan 7, 2020 • edited

umangachapagain commented Jan 8, 2020

umangachapagain commented Jan 8, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aruniiird Jan 14, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TheRealJon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

spadgett commented Jan 22, 2020

spadgett commented Jan 22, 2020

bipuladh commented Jan 22, 2020

spadgett commented Jan 22, 2020

spadgett commented Jan 22, 2020

bipuladh commented Jan 22, 2020

spadgett Jan 22, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

spadgett left a comment

Choose a reason for hiding this comment

openshift-ci-robot commented Jan 22, 2020

spadgett commented Jan 22, 2020

openshift-bot commented Jan 22, 2020

openshift-bot commented Jan 22, 2020

openshift-bot commented Jan 22, 2020

openshift-bot commented Jan 22, 2020

openshift-bot commented Jan 23, 2020

openshift-bot commented Jan 23, 2020

bipuladh commented Jan 23, 2020

openshift-bot commented Jan 23, 2020

bipuladh commented Jan 7, 2020 •

edited

aruniiird Jan 14, 2020 •

edited

spadgett Jan 22, 2020 •

edited