STOR-1037: Add cluster-storage-operator to mgmt. cluster #1748

jsafrane · 2022-09-16T07:57:56Z

What this PR does / why we need it:

Move cluster-storage-operator + its operands (aws-ebs-csi-driver-operator + AWS EBS CSI driver control plane pods) to HyperShift mgmt. cluster.

cluster-storage-operator must be updated first!

Testing
Update control-plane-operator RBACs:

Build your own hypershift-operator image.
1. Clone openshift/hypershift git repo + apply this PR.
2. Build the custom image, e.g. quay.io/jsafrane/hypershift:latest:
```
make docker-build docker-push quay.io/jsafrane/hypershift:latest
```
Install hypershift-operator from the custom image: ./hypershift install --hypershift-image=quay.io/jsafrane/hypershift:latest ...

Install a guest cluster:

Build a release that contains this PR + cluster-storage-operator PR + AWS operator PR. For example, ask cluster-bot: build openshift/hypershift#1698,openshift/aws-ebs-csi-driver-operator#159,openshift/cluster-storage-operator#XXX
Use the release to install a guest cluster: ./hypershift create cluster aws --release-image=registry.build02.ci.openshift.org/ci-ln-ww5ihgt/release:latest ...

Checklist

Subject and description added to both, commit and PR.
Relevant issues have been referenced.
This change includes docs.
This change includes unit tests.

netlify · 2022-09-29T15:40:23Z

✅ Deploy Preview for hypershift-docs ready!

Name	Link
🔨 Latest commit	`a020c10`
🔍 Latest deploy log	https://app.netlify.com/sites/hypershift-docs/deploys/63526dc4467fee0009800da0
😎 Deploy Preview	https://deploy-preview-1748--hypershift-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

gnufied · 2022-10-07T17:06:03Z

...l-plane-operator/controllers/hostedcontrolplane/storage/assets/10_deployment-hypershift.yaml

+        - name: POWERVS_BLOCK_CSI_DRIVER_IMAGE
+          value: quay.io/openshift/origin-powervs-block-csi-driver:latest
+        - name: HYPERSHIFT_IMAGE
+          value: tbd


I thought we had this - quay.io/openshift/origin-control-plane:latest ?

jsafrane · 2022-10-10T14:52:53Z

/hold
openshift/cluster-storage-operator#318 must be merged first!

enxebre · 2022-10-20T11:03:50Z

cmd/install/assets/hypershift_operator.go

@@ -802,7 +802,7 @@ func (o HyperShiftOperatorClusterRole) Build() *rbacv1.ClusterRole {
 			},
 			{
 				APIGroups: []string{"apps"},
-				Resources: []string{"deployments", "statefulsets"},
+				Resources: []string{"deployments", "replicasets", "statefulsets"},


Note for myself: this will likely require refresh the HO CI role.

enxebre · 2022-10-20T11:10:38Z

control-plane-operator/controllers/hostedcontrolplane/manifests/storage.go

+
+func ClusterStorageOperatorRole(ns string) *rbacv1.Role {
+	role := &rbacv1.Role{}
+	role.Name = "cluster-storage-operator-role"


name is redundant with type "cluster-storage-operator-role" -> "cluster-storage-operator"

enxebre · 2022-10-20T11:11:07Z

control-plane-operator/controllers/hostedcontrolplane/manifests/storage.go

+
+func ClusterStorageOperatorRoleBinding(ns string) *rbacv1.RoleBinding {
+	roleBinding := &rbacv1.RoleBinding{}
+	roleBinding.Name = "cluster-storage-operator-role"


cluster-storage-operator-role-> cluster-storage-operator?

enxebre · 2022-10-20T11:12:12Z

...l-plane-operator/controllers/hostedcontrolplane/storage/assets/10_deployment-hypershift.yaml

@@ -0,0 +1,137 @@
+# *** AUTOMATICALLY GENERATED FILE - DO NOT EDIT ***


can you ref the source? automatically generated by what/how?

Added a README.md

enxebre · 2022-10-20T11:13:44Z

control-plane-operator/controllers/hostedcontrolplane/storage/assets/rolebinding.yaml

+kind: RoleBinding
+metadata:
+  labels:
+  name: cluster-storage-operator-role


cluster-storage-operator-role-> cluster-storage-operator?

enxebre · 2022-10-20T11:17:49Z

control-plane-operator/controllers/hostedcontrolplane/storage/params.go

+)
+
+var (
+	// map env. variable in CSO Deployment -> key in `images` map (= name of the image in payload).


Do we have any programatic guarantee that we don't end up with the operator referencing an image as hardcoded in the yaml, e.g. if it's missing here?
Can we unit test it somehow?

I reworked the env. var replacement into its own file + added an unit test

enxebre · 2022-10-20T11:18:55Z

/label tide/merge-method-squash
I dropped some minor comments.
Looks great to me otherwise, pending openshift/cluster-storage-operator#318

sferich888 · 2022-10-20T14:47:45Z

/label px-approved

jsafrane · 2022-11-01T12:53:29Z

/retest
do we have 4.11.z fixed?

jsafrane · 2022-11-02T11:01:39Z

/test e2e-aws

jsafrane · 2022-11-11T13:51:07Z

/retest

tsmetana · 2022-11-14T11:56:58Z

/retest

csrwng · 2022-11-14T18:07:03Z

/retest-required

enxebre · 2022-11-16T16:37:31Z

/test e2e-aws

enxebre · 2022-11-16T18:51:43Z

@jsafrane failure is legit.

=== RUN TestUpgradeControlPlane/EnsureNoPodsWithTooHighPriority
util.go:411: pod aws-ebs-csi-driver-controller-7fb697944c-8pnzm with priorityClassName system-cluster-critical has a priority of 2000000000 with exceeds the max allowed of 100002000

It will be installed by control-plane-operator in the mgmt cluster.

Right now, the operator needs more permissions that would be necessary, but we need to fix cluster-storage-operator and aws-ebs-csi-driver-operators first.

jsafrane · 2022-12-08T12:49:23Z

From guest clusters:

message: Cluster operators console, dns, image-registry, ingress, insights,
kube-storage-version-migrator, monitoring, node-tuning, openshift-samples,
service-ca, storage are not available

I don't see any nodes created for the guest clusters.
/retest

csrwng · 2022-12-08T18:08:35Z

/retest-required

openshift-ci · 2022-12-08T19:30:40Z

@jsafrane: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/periodics-4.13-images	`a020c10`	link	true	`/test periodics-4.13-images`
ci/prow/e2e-ibmcloud-iks	`a020c10`	link	false	`/test e2e-ibmcloud-iks`
ci/prow/e2e-ibmcloud-roks	`a020c10`	link	false	`/test e2e-ibmcloud-roks`
ci/prow/kubevirt-e2e-kubevirt-azure-ovn	`99a5e90`	link	false	`/test kubevirt-e2e-kubevirt-azure-ovn`
ci/prow/capi-provider-agent-sanity	`99a5e90`	link	false	`/test capi-provider-agent-sanity`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

csrwng · 2022-12-09T01:16:26Z

/approve

csrwng · 2022-12-09T01:16:45Z

/retest-required

openshift-ci · 2022-12-09T01:16:45Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: csrwng, jsafrane

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [csrwng]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

csrwng · 2022-12-09T02:37:47Z

/lgtm

derekwaynecarr · 2022-12-09T14:38:46Z

woot!

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 16, 2022

openshift-ci bot requested review from csrwng and sjenning September 16, 2022 07:59

jsafrane force-pushed the cso branch 5 times, most recently from d633962 to 8189970 Compare September 16, 2022 10:15

jsafrane force-pushed the cso branch from 9c6ec59 to 00ca6f4 Compare October 7, 2022 11:07

gnufied reviewed Oct 7, 2022

View reviewed changes

jsafrane force-pushed the cso branch 2 times, most recently from 0a5226e to 8dfc9b6 Compare October 10, 2022 09:37

jsafrane changed the title ~~WIP: STOR-1037: Add cluster-storage-operator to mgmt. cluster~~ STOR-1037: Add cluster-storage-operator to mgmt. cluster Oct 10, 2022

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 10, 2022

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 10, 2022

jsafrane mentioned this pull request Oct 13, 2022

STOR-1040: Render AWS CSI Driver credentials in the mgmt cluster #1751

Merged

jsafrane force-pushed the cso branch from 8dfc9b6 to 0b44aef Compare October 19, 2022 08:46

enxebre reviewed Oct 20, 2022

View reviewed changes

openshift-ci bot added the tide/merge-method-squash Denotes a PR that should be squashed by tide when it merges. label Oct 20, 2022

openshift-ci bot added the px-approved Signifies that Product Support has signed off on this PR label Oct 20, 2022

jsafrane force-pushed the cso branch from 0b44aef to f0bd4c7 Compare October 20, 2022 15:33

openshift-ci bot added the qe-approved Signifies that QE has signed off on this PR label Oct 29, 2022

jsafrane force-pushed the cso branch 3 times, most recently from 99cf0ab to 99a5e90 Compare December 2, 2022 11:41

jsafrane added 6 commits December 8, 2022 11:25

Add cluster-storage-operator

f6444a6

Don't install cluster-storage-operator in the guest cluster

b0eca7a

It will be installed by control-plane-operator in the mgmt cluster.

Fix RBACs for cluster-storage-operator

41d37c5

Right now, the operator needs more permissions that would be necessary, but we need to fix cluster-storage-operator and aws-ebs-csi-driver-operators first.

Add env. var replacer + unit test for it

7cdb85e

Add readme for asset files

0f9de52

Rename roles

f492d3e

jsafrane force-pushed the cso branch from 99a5e90 to f492d3e Compare December 8, 2022 10:27

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 9, 2022

openshift-ci bot assigned csrwng Dec 9, 2022

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Dec 9, 2022

openshift-merge-robot merged commit 6d862d1 into openshift:main Dec 9, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

STOR-1037: Add cluster-storage-operator to mgmt. cluster #1748

STOR-1037: Add cluster-storage-operator to mgmt. cluster #1748

jsafrane commented Sep 16, 2022 •

edited

netlify bot commented Sep 29, 2022 •

edited

gnufied Oct 7, 2022

jsafrane Oct 10, 2022

jsafrane commented Oct 10, 2022 •

edited

enxebre Oct 20, 2022

enxebre Oct 20, 2022

jsafrane Oct 21, 2022

enxebre Oct 20, 2022

jsafrane Oct 21, 2022

enxebre Oct 20, 2022

jsafrane Oct 21, 2022

enxebre Oct 20, 2022

jsafrane Oct 21, 2022

enxebre Oct 20, 2022

jsafrane Oct 20, 2022

enxebre commented Oct 20, 2022 •

edited

sferich888 commented Oct 20, 2022

jsafrane commented Nov 1, 2022

jsafrane commented Nov 2, 2022

jsafrane commented Nov 11, 2022

tsmetana commented Nov 14, 2022

csrwng commented Nov 14, 2022

enxebre commented Nov 16, 2022

enxebre commented Nov 16, 2022 •

edited

jsafrane commented Dec 8, 2022 •

edited

csrwng commented Dec 8, 2022

openshift-ci bot commented Dec 8, 2022 •

edited

csrwng commented Dec 9, 2022

csrwng commented Dec 9, 2022

openshift-ci bot commented Dec 9, 2022

csrwng commented Dec 9, 2022

derekwaynecarr commented Dec 9, 2022

		@@ -0,0 +1,137 @@
		# * AUTOMATICALLY GENERATED FILE - DO NOT EDIT *

STOR-1037: Add cluster-storage-operator to mgmt. cluster #1748

STOR-1037: Add cluster-storage-operator to mgmt. cluster #1748

Conversation

jsafrane commented Sep 16, 2022 • edited

netlify bot commented Sep 29, 2022 • edited

✅ Deploy Preview for hypershift-docs ready!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsafrane commented Oct 10, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

enxebre commented Oct 20, 2022 • edited

sferich888 commented Oct 20, 2022

jsafrane commented Nov 1, 2022

jsafrane commented Nov 2, 2022

jsafrane commented Nov 11, 2022

tsmetana commented Nov 14, 2022

csrwng commented Nov 14, 2022

enxebre commented Nov 16, 2022

enxebre commented Nov 16, 2022 • edited

jsafrane commented Dec 8, 2022 • edited

csrwng commented Dec 8, 2022

openshift-ci bot commented Dec 8, 2022 • edited

csrwng commented Dec 9, 2022

csrwng commented Dec 9, 2022

openshift-ci bot commented Dec 9, 2022

csrwng commented Dec 9, 2022

derekwaynecarr commented Dec 9, 2022

jsafrane commented Sep 16, 2022 •

edited

netlify bot commented Sep 29, 2022 •

edited

jsafrane commented Oct 10, 2022 •

edited

enxebre commented Oct 20, 2022 •

edited

enxebre commented Nov 16, 2022 •

edited

jsafrane commented Dec 8, 2022 •

edited

openshift-ci bot commented Dec 8, 2022 •

edited