Bug 1904497: Add vsphere problem detector deployment #111

gnufied · 2020-12-04T06:06:47Z

Use deployment for managing vspher problem detector:

Create static assets for RBAC, deployment, service and prometheus
Sync static assets we can sync with library-go
Sync ServiceMonitor object
Sync Deployment object.
Link up the deployment with rest of the CSO

fixes https://bugzilla.redhat.com/show_bug.cgi?id=1904497

Add code to sync static assets Add generated binary assets Add code to create service monitor requests

openshift-ci-robot · 2020-12-08T20:41:26Z

@gnufied: This pull request references Bugzilla bug 1904497, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.7.0) matches configured target release for branch (4.7.0)
bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

Bug 1904497: Add vsphere problem detector deployment

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2020-12-08T20:41:40Z

@gnufied: This pull request references Bugzilla bug 1904497, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.7.0) matches configured target release for branch (4.7.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

Bug 1904497: Add vsphere problem detector deployment

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

gnufied · 2020-12-09T15:45:45Z

This is how conditions look like btw:

    conditions:
    - lastTransitionTime: "2020-11-30T04:03:30Z"
      reason: AsExpected
      status: "False"
      type: CSIDriverStarterDegraded
    - lastTransitionTime: "2020-11-30T04:03:30Z"
      status: "False"
      type: ManagementStateDegraded
    - lastTransitionTime: "2020-11-30T04:03:30Z"
      status: "True"
      type: DefaultStorageClassControllerAvailable
    - lastTransitionTime: "2020-11-30T04:03:30Z"
      status: "False"
      type: DefaultStorageClassControllerProgressing
    - lastTransitionTime: "2020-11-30T04:03:30Z"
      reason: AsExpected
      status: "False"
      type: DefaultStorageClassControllerDegraded
    - lastTransitionTime: "2020-11-30T04:03:30Z"
      status: "True"
      type: SnapshotCRDControllerUpgradeable
    - lastTransitionTime: "2020-11-30T04:03:30Z"
      reason: AsExpected
      status: "False"
      type: SnapshotCRDControllerDegraded
    - lastTransitionTime: "2020-12-04T02:41:07Z"
      status: "True"
      type: VSphereProblemDetectorControllerAvailable
    - lastTransitionTime: "2020-12-04T02:41:07Z"
      reason: AsExpected
      status: "False"
      type: VSphereProblemDetectorControllerDegraded
    - lastTransitionTime: "2020-12-07T20:24:12Z"
      reason: AsExpected
      status: "False"
      type: VSphereProblemDetectorStarterDegraded
    - lastTransitionTime: "2020-12-07T20:24:18Z"
      reason: AsExpected
      status: "False"
      type: VSphereProblemDetectorStarterStaticControllerDegraded
    - lastTransitionTime: "2020-12-09T15:39:49Z"
      status: "True"
      type: VSphereProblemDetectorDeploymentControllerAvailable
    - lastTransitionTime: "2020-12-08T19:36:23Z"
      status: "False"
      type: VSphereProblemDetectorDeploymentControllerProgressing
    - lastTransitionTime: "2020-12-07T20:34:57Z"
      reason: AsExpected
      status: "False"
      type: VSphereProblemDetectorDeploymentControllerDegraded
    - lastTransitionTime: "2020-12-08T19:24:41Z"
      status: "True"
      type: VSphereProblemDetectorMonitoringControllerAvailable
    - lastTransitionTime: "2020-12-08T19:24:41Z"
      reason: AsExpected
      status: "False"
      type: VSphereProblemDetectorMonitoringControllerDegraded

jsafrane · 2020-12-09T17:40:02Z

pkg/operator/vsphereproblemdetector/vsphere_problem_detector_starter.go

+		return nil
+	}
+
+	go c.controller.Start(ctx)


Please run the controller only once, this looks like it's going to be called repeatedly in each sync()

pkg/operator/vsphereproblemdetector/vsphere_problem_detector_starter.go

pkg/operator/vsphereproblemdetector/monitoring.go

jsafrane · 2020-12-09T17:43:40Z

pkg/operator/vsphereproblemdetector/monitoring.go

+		eventRecorder:  eventRecorder,
+	}
+	return factory.New().
+		WithInformers(c.operatorClient.Informer()).


Should it have informer of ServiceMonitors too?

We could but it will require importing all of - https://github.com/prometheus-operator/prometheus-operator/tree/master/pkg/apis/monitoring but I was being thwarted by dependency conflicts, so I have avoided doing this.

We are doing what we did for syncing credentials (before we moved them to CVO) and that is the reason we are rescyncing every 1 minute because we are not watching ServiceMonitor objects.

I did ended up importing those informer and APIs and hence marking this as resolved.

pkg/operator/vsphereproblemdetector/monitoring.go

pkg/operator/vsphereproblemdetector/vsphere_problem_detector_deployment.go

jsafrane · 2020-12-09T17:52:30Z

pkg/operator/vsphereproblemdetector/vsphere_problem_detector_deployment.go

+	return nil
+}
+
+func getLogLevel(logLevel operatorapi.LogLevel) int {


Please move it to a common package, it's used in deploymentcontroller.go too.

fixed. I extracted most of deployment controller creation code in a util function and used in both places.

gnufied · 2020-12-11T15:37:38Z

/retest

bertinatto · 2020-12-09T17:37:34Z

pkg/operator/vsphereproblemdetector/vsphere_problem_detector_deployment.go

+	opSpec, opStatus, _, err := c.operatorClient.GetOperatorState()
+	if err != nil {
+		return err
+	}


We need to check for IsNotFound error here

bertinatto · 2020-12-09T17:39:45Z

pkg/operator/vsphereproblemdetector/vsphere_problem_detector_deployment.go

+	resourcemerge.SetDeploymentGeneration(&opStatus.Generations, deployment)
+
+	updateGenerationFn := func(newStatus *operatorapi.OperatorStatus) error {
+		if deployment != nil {


If deployment were nil it would have crashed up above

bertinatto · 2020-12-11T14:47:01Z

assets/vsphere_problem_detector/04_clusterrole.yaml

+- apiGroups:
+  - security.openshift.io
+  resourceNames:
+  - privileged


Which functionality require it to be privileged?

bertinatto · 2020-12-11T14:51:49Z

assets/vsphere_problem_detector/08_prometheus_role.yaml

+  namespace: openshift-cluster-storage-operator
+  annotations:
+    include.release.openshift.io/self-managed-high-availability: "true"
+    include.release.openshift.io/single-node-developer: "true"


This is deployed by CSO, not CVO. Why do we need these annotations?

If we do need them, shouldn't the deployment have them as well?

Same comment for the other files in assets/vsphere_problem_detector

bertinatto · 2020-12-11T14:54:51Z

pkg/operator/vsphereproblemdetector/monitoring.go

+		operatorClient: clients.OperatorClient,
+		kubeClient:     clients.KubeClient,
+		dynamicClient:  clients.DynamicClient,
+		eventRecorder:  eventRecorder,


Please add a suffix to this recorder so that we know where events are coming from

shouldn't eventRecorder.WithComponentSuffix("vsphere-monitoring-controller") applied when creating controller be enough?

Not really because it's a different control loop

but we don't use it anywhere else.

oops I missed this. fixed.

pkg/operator/vsphereproblemdetector/vsphere_problem_detector_deployment.go

pkg/utils/deployment_controller.go

bertinatto · 2020-12-11T15:09:26Z

manifests/10_deployment.yaml

@@ -82,35 +82,17 @@ spec:
            value: quay.io/openshift/origin-csi-node-driver-registrar:latest
          - name: LIVENESS_PROBE_IMAGE
            value: quay.io/openshift/origin-csi-livenessprobe:latest
+          - name: VSPHERE_PROBLEM_DETECTOR_OPERATOR_IMAGE
+            value: quay.io/openshift/origin-vsphere-problem-detector:latest


nit: PR descriptions says it's adding CredentialsRequest, but it's already there

bertinatto · 2020-12-11T15:14:15Z

pkg/operator/vsphereproblemdetector/vsphere_problem_detector_starter.go

+	opSpec, _, _, err := c.operatorClient.GetOperatorState()
+	if err != nil {
+		return err
+	}


Missing check for not found error...

gnufied

placeholder

openshift-ci-robot · 2020-12-11T17:01:24Z

@gnufied: This pull request references Bugzilla bug 1904497, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.7.0) matches configured target release for branch (4.7.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

Bug 1904497: Add vsphere problem detector deployment

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

bertinatto · 2020-12-11T18:08:21Z

pkg/operator/vsphereproblemdetector/monitoring.go

+		operatorClient: clients.OperatorClient,
+		kubeClient:     clients.KubeClient,
+		dynamicClient:  clients.DynamicClient,
+		eventRecorder:  eventRecorder,


Not really because it's a different control loop

bertinatto · 2020-12-11T18:19:34Z

pkg/operator/vsphereproblemdetector/monitoring.go

+	_, err = resourceapply.ApplyServiceMonitor(c.dynamicClient, c.eventRecorder, serviceMonitorBytes)
+
+	if err != nil {
+		return err


IMO we shouldn't make CSO degraded if we fail to create a service monitor

why not? currently service monitor objects are created by CVO and if it can't create those, I think cluster will be degraded.

bertinatto · 2020-12-11T18:28:18Z

pkg/utils/deployment_controller.go

+		deploymentAvailable.Status = operatorapi.ConditionTrue
+	} else {
+		deploymentAvailable.Status = operatorapi.ConditionFalse
+		deploymentAvailable.Reason = "WaitDeployment"


I know it was like this before, but we should have a single Reason value to represent the same thing: openshift/library-go#901

bertinatto · 2020-12-11T18:51:11Z

pkg/operator/vsphereproblemdetector/vsphere_problem_detector_starter.go

+
+func (c *VSphereProblemDetectorStarter) sync(ctx context.Context, syncCtx factory.SyncContext) error {
+	klog.V(4).Infof("VSphereProblemDetectorStarter.Sync started")
+	defer klog.V(4).Infof("VSphereProblemDetectorStarter.Sync finished")


hm... this controller would still be running in all platforms, not just vSphere...

yes, this controller will still run but it won't do anything.

bertinatto · 2020-12-11T19:03:18Z

pkg/operator/vsphereproblemdetector/vsphere_problem_detector_starter.go

+	eventRecorder  events.Recorder
+}
+
+func NewVSphereProblemDetectorStarter(


I really don't like that we have 2 "starter" controllers whose main job is the same: start certain sub-controllers per platform.

I'd prefer to make the necessary adjustments to CSIDriverStarterController (to make it easily to start non-CSI controllers), rather than creating another starter controller.

not that I disagree to it but can we do this in a future PR(I am happy to open a BZ/github issue to remind myself), because I think that this PR is quite large already.

idea: when we have vSphere CSI driver, we can add detector deployment at an extra controller started with CSI driver, similarly as Manila starts certificate syncer:

cluster-storage-operator/pkg/operator/csidriveroperator/csioperatorclient/manila.go

Lines 46 to 48 in e7919b1

ExtraControllers: []factory.Controller{

newCertificateSyncerOrDie(clients, recorder),

},

It will be somewhat hidden in the CSI driver deployment, but still, it will run.

I like that. Can we add a TODO entry in this starter controller?

bertinatto · 2020-12-11T19:18:50Z

pkg/utils/deployment_controller.go

+		v1helpers.UpdateConditionFn(deploymentProgressing),
+		updateGenerationFn,
+	); err != nil {
+		return nil, err


I know the idea of this function was to avoid code duplication, but this will make CSO degraded if we can't deploy the problem detector operator. Is that what we want?

Why wouldn't we want that? vsphere-problem-detector is not an optional operator.

I think it's better to get degraded, it's really not an optional operator.

jsafrane · 2020-12-14T16:51:03Z

assets/vsphere_problem_detector/04_clusterrole.yaml

+- apiGroups:
+  - rbac.authorization.k8s.io
+  resources:
+  - clusterroles
+  - clusterrolebindings
+  - roles
+  - rolebindings
+  verbs:
+  - watch
+  - list
+  - get
+- apiGroups:
+  - ''
+  resources:
+  - serviceaccounts
+  verbs:
+  - get
+  - list
+  - watch
+- apiGroups:
+  - authentication.k8s.io
+  resources:
+  - tokenreviews
+  verbs:
+  - '*'
+- apiGroups:
+  - authorization.k8s.io
+  resources:
+  - subjectaccessreviews
+  verbs:
+  - '*'
+- apiGroups:
+  - apiextensions.k8s.io
+  resources:
+  - customresourcedefinitions
+  verbs:
+  - list
+  - watch
+- apiGroups:
+  - coordination.k8s.io
+  resources:
+  - leases
+  verbs:
+  - '*'


Does the problem detector need anything from this list?

I think this got copied from csi driver's clusterroles. We can drop it.

jsafrane · 2020-12-14T16:51:50Z

assets/vsphere_problem_detector/04_clusterrole.yaml

+- apiGroups:
+  - ''
+  resources:
+  - namespaces
+  verbs:
+  - get
+  - list
+  - watch


The operator should not need read namespaces.

jsafrane · 2020-12-14T16:52:21Z

assets/vsphere_problem_detector/04_clusterrole.yaml

+- apiGroups:
+  - '*'
+  resources:
+  - events
+  verbs:
+  - get
+  - patch
+  - create
+  - list
+  - watch
+  - update
+  - delete


The operator should not need read/write events in another namespaces (use Role for emitting events in the operator namespace.)

jsafrane · 2020-12-14T16:52:48Z

assets/vsphere_problem_detector/04_clusterrole.yaml

+- apiGroups:
+  - cloudcredential.openshift.io
+  resources:
+  - credentialsrequests
+  verbs:
+  - '*'


The operator should not need manipulate CredentialRequests.

removed those extra verbs. but still have CR permissions to get, list, watch.

jsafrane · 2020-12-14T16:54:49Z

go.mod

 	k8s.io/apimachinery v0.19.2
-	k8s.io/client-go v0.19.0
+	k8s.io/client-go v12.0.0+incompatible


This is suspicious - why downgrade to 1.12?

This does not actually result in downgrade of client-go(check go.sum), but used client-go is still v0.19.2 but what is happening is - one of the dependencies of prometheus-operator depends on v0.12.0 - https://github.com/prometheus-operator/prometheus-operator/blob/master/go.mod#L48 and because of golang-1.13 semver check it is almost impossible to use client-go v0.12.0 without using replace directive.

I have not manually added this line - but go mod vendor and go mod tidy will replace v.0.19.2 with this version.

jsafrane · 2020-12-14T17:01:02Z

pkg/utils/deployment_controller.go

+		}
+	}
+
+	depOpts.OpStatus.ReadyReplicas = deployment.Status.ReadyReplicas


CSI driver will replace nr. of replica of problem-detector deployment and a vice versa. IMO, one of the deployments (or even both) should ignore ReadyReplicas completely. The field is not very usable anyway.

I removed setting that field.

gnufied · 2020-12-15T02:20:41Z

/retest

gnufied · 2020-12-15T15:25:39Z

/retest

jsafrane · 2020-12-15T16:42:05Z

assets/vsphere_problem_detector/04_clusterrole.yaml

+- apiGroups:
+  - rbac.authorization.k8s.io
+  resources:
+  - clusterroles
+  - clusterrolebindings
+  - roles
+  - rolebindings
+  verbs:
+  - watch
+  - list
+  - get
+- apiGroups:
+  - ''
+  resources:
+  - serviceaccounts
+  verbs:
+  - get
+  - list
+  - watch
+- apiGroups:
+  - authentication.k8s.io
+  resources:
+  - tokenreviews
+  verbs:
+  - '*'
+- apiGroups:
+  - authorization.k8s.io
+  resources:
+  - subjectaccessreviews
+  verbs:
+  - '*'
+- apiGroups:
+  - apiextensions.k8s.io
+  resources:
+  - customresourcedefinitions
+  verbs:
+  - list
+  - watch


All this seems to be useless.

removed these.

jsafrane · 2020-12-15T16:42:30Z

assets/vsphere_problem_detector/04_clusterrole.yaml

+  - create
+  - patch
+  - update


The operator does not need to modify nodes.

removed these.

jsafrane · 2020-12-15T16:42:58Z

assets/vsphere_problem_detector/04_clusterrole.yaml

+- apiGroups:
+  - cloudcredential.openshift.io
+  resources:
+  - credentialsrequests
+  verbs:
+  - get
+  - list
+  - watch


The operator should not need to read credentialsrequests

Fix eventrecorder and reasoning

jsafrane · 2020-12-16T10:49:42Z

/lgtm

openshift-ci-robot · 2020-12-16T10:50:05Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gnufied, jsafrane

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [gnufied,jsafrane]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2020-12-16T11:22:45Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-12-16T11:35:48Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-ci-robot · 2020-12-16T13:18:36Z

@gnufied: All pull requests linked via external trackers have merged:

openshift/cluster-storage-operator#111

Bugzilla bug 1904497 has been moved to the MODIFIED state.

In response to this:

Bug 1904497: Add vsphere problem detector deployment

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 4, 2020

openshift-ci-robot requested review from bertinatto and huffmanca December 4, 2020 06:07

openshift-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Dec 4, 2020

gnufied force-pushed the add-vsphere-problem-detector-deployment branch from 034f8d2 to fab994e Compare December 8, 2020 20:37

openshift-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 8, 2020

gnufied changed the title ~~WIP: Add vsphere problem detector deployment~~ Add vsphere problem detector deployment Dec 8, 2020

openshift-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 8, 2020

Use deployment to deploy vsphere problem detector

0a30222

Add code to sync static assets Add generated binary assets Add code to create service monitor requests

gnufied force-pushed the add-vsphere-problem-detector-deployment branch from fab994e to 0a30222 Compare December 8, 2020 20:40

gnufied changed the title ~~Add vsphere problem detector deployment~~ Bug 1904497: Add vsphere problem detector deployment Dec 8, 2020

openshift-ci-robot added bugzilla/severity-medium Referenced Bugzilla bug's severity is medium for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. labels Dec 8, 2020

Change named port of vsphere-metrics service

f236f05

gnufied force-pushed the add-vsphere-problem-detector-deployment branch from 289b8ce to f236f05 Compare December 9, 2020 17:00

jsafrane reviewed Dec 9, 2020

View reviewed changes

gnufied added 5 commits December 10, 2020 12:31

Cleanup existing rules

b804822

Remove code duplication from deployment controller

7cf10d7

Vendor dependencies properly

2a3a65d

Add prometheus informers

2c205d8

Use informer for service monitor

d86d9f0

gnufied force-pushed the add-vsphere-problem-detector-deployment branch from 423ae36 to d86d9f0 Compare December 11, 2020 03:54

bertinatto reviewed Dec 11, 2020

View reviewed changes

gnufied commented Dec 11, 2020

View reviewed changes

reduce scope of rbac roles

26b8b88

Remove deployment controller customizations

27b1541

bertinatto reviewed Dec 11, 2020

View reviewed changes

gnufied force-pushed the add-vsphere-problem-detector-deployment branch from a5efca5 to 6f568c2 Compare December 14, 2020 16:30

jsafrane reviewed Dec 14, 2020

View reviewed changes

gnufied force-pushed the add-vsphere-problem-detector-deployment branch from 6f568c2 to 5d7df47 Compare December 14, 2020 19:42

jsafrane reviewed Dec 15, 2020

View reviewed changes

Remove unnecessary annotations and rbacs

60bc9ce

Fix eventrecorder and reasoning

gnufied force-pushed the add-vsphere-problem-detector-deployment branch from 5d7df47 to 60bc9ce Compare December 15, 2020 17:45

openshift-ci-robot assigned jsafrane Dec 16, 2020

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Dec 16, 2020

openshift-merge-robot merged commit 0fdc1a3 into openshift:master Dec 16, 2020

	ExtraControllers: []factory.Controller{
	newCertificateSyncerOrDie(clients, recorder),
	},

Bug 1904497: Add vsphere problem detector deployment #111

Bug 1904497: Add vsphere problem detector deployment #111

Conversation

gnufied commented Dec 4, 2020 • edited

openshift-ci-robot commented Dec 8, 2020

openshift-ci-robot commented Dec 8, 2020

gnufied commented Dec 9, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsafrane Dec 9, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gnufied commented Dec 11, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gnufied left a comment

Choose a reason for hiding this comment

openshift-ci-robot commented Dec 11, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gnufied commented Dec 15, 2020

gnufied commented Dec 15, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsafrane commented Dec 16, 2020

openshift-ci-robot commented Dec 16, 2020

openshift-bot commented Dec 16, 2020

openshift-bot commented Dec 16, 2020

openshift-ci-robot commented Dec 16, 2020

gnufied commented Dec 4, 2020 •

edited

jsafrane Dec 9, 2020 •

edited