add the registry operator to the default install.

[bparees]

No description provided.

[bparees]

/assign @derekwaynecarr

@legionus fyi. this will get the registry operator installed by default, until we tie into the installer differently. we'll have to keep these files in sync when important changes happen (which i assume will be rare)

[wking]

I don't understand the distinction between modules/bootkube and modules/tectonic, and we consume them both together, so maybe there isn't a distinction. @abhinavdahiya, @yifan-gu, do either of you know?

On the other hand, this will all be handled differently soon, so I'm fine with this PR landing without agonizing over the target location ;).

[aaronlevy]

IIRC the only difference was that anything in bootkube used the binary process for object creation (which you dig into in comment below) - and was vaguely thought of as "anything required to get the cluster running". Anything in tectonic had a bash script that created the objects in a hard-coded order - and were all the "on top of" components (e.g. monitoring). They probably could/should be collapsed at some point - but in near term likely doesn't matter.

[aaronlevy]

From in-person discussion with @abhinavdahiya he would prefer that anything not directly required to get a working control-plane ends up under the tectonic module for now - so I believe this should be added to:

installer/modules/tectonic/resources/tectonic.sh

Lines 130 to 171 in 741d5cc

	echo "Creating Initial Roles"
	kubectl delete -f rbac/role-admin.yaml
	kubectl create -f ingress/svc-account.yaml
	kubectl create -f rbac/role-admin.yaml
	kubectl create -f rbac/role-user.yaml
	kubectl create -f rbac/binding-admin.yaml
	kubectl create -f rbac/binding-discovery.yaml
	echo "Creating Cluster Config For Tectonic"
	kubectl create -f cluster-config.yaml
	kubectl create -f ingress/cluster-config.yaml
	echo "Creating Tectonic Secrets"
	kubectl create -f secrets/pull.json
	kubectl create -f secrets/ingress-tls.yaml
	kubectl create -f secrets/ca-cert.yaml
	kubectl create -f ingress/pull.json
	echo "Creating Operators"
	kubectl create -f security/priviledged-scc-tectonic.yaml
	kubectl create -f updater/tectonic-channel-operator-kind.yaml
	kubectl create -f updater/app-version-kind.yaml
	kubectl create -f updater/migration-status-kind.yaml
	wait_for_crd tectonic-system channeloperatorconfigs.tco.coreos.com
	kubectl create -f updater/tectonic-channel-operator-config.yaml
	kubectl create -f updater/operators/kube-core-operator.yaml
	kubectl create -f updater/operators/tectonic-channel-operator.yaml
	kubectl create -f updater/operators/kube-addon-operator.yaml
	kubectl create -f updater/operators/tectonic-alm-operator.yaml
	kubectl create -f updater/operators/tectonic-utility-operator.yaml
	kubectl create -f updater/operators/tectonic-ingress-controller-operator.yaml
	wait_for_crd tectonic-system appversions.tco.coreos.com
	kubectl create -f updater/app_versions/app-version-tectonic-cluster.yaml
	kubectl create -f updater/app_versions/app-version-kube-core.yaml
	kubectl create -f updater/app_versions/app-version-kube-addon.yaml
	kubectl create -f updater/app_versions/app-version-tectonic-alm.yaml
	kubectl create -f updater/app_versions/app-version-tectonic-utility.yaml
	kubectl create -f updater/app_versions/app-version-tectonic-ingress.yaml

[bparees]

@derekwaynecarr how do i assert what order these asset files are applied in?

[bparees]

/cc @eparis @yifan-gu

[wking]

how do i assert what order these asset files are applied in?

Looks like you may need entries here to get these injected into the Ignition payload. I'd guess they're then applied to the cluster in collation order, based on the presence of 01-tectonic-namespace.yaml and similar in the existing list.

[bparees]

Looks like you may need entries here to get these injected into the Ignition payload. I'd guess they're then applied to the cluster in collation order, based on the presence of 01-tectonic-namespace.yaml and similar in the existing list.

thanks, added.

[wking]

modules/tectonic injection happens here, so if these move into modules/tectonic we'd need additional entries there for the new assets.

modules/bootkube injection seems to happen in the bootkube image. I'm checking to see how that works...

[bparees]

/unassign @derekwaynecarr
/assign @eparis @aaronlevy

[wking]

modules/bootkube injection seems to happen in the bootkube image. I'm checking to see how that works...

We're passing --asset-dir=/assets to bootkube. Bootkube appends manifests to that path and then loads and pushes those assets. Loading uses filepath.Walk, which is lexically ordered.

[wking]

Bootkube appends manifests to that path and then loads and pushes those assets. Loading uses filepath.Walk, which is lexically ordered.

But this is all clobbered in pushing, where:

1. We re-sort (because we don't trust filepath.Walk?).
2. We split into namespaces, CRDs, and other, preserving the previous sort order within each subset.
3. We push all the namespaces in order.
4. We push all the CRDs in order.
5. We wait on the CRDs.
6. We push everything else.

[aaronlevy]

I was a little hesitant back when we changed this behavior - initially preferring that we let lexicographic ordering determine the absolute creation order, rather than split it up by object type first. However, at the time it was determined that there wasn't a good example where a user wouldn't want namespaces, for example, to be created up front and minimize user-error (and instead done in a more specific/manual order).

Either way, there are some objects that are created, but asynchronously available - and that behavior I think we probably want to preserve. For example, if I create a CRD the api endpoint for instances of that CR isn't available for a couple seconds - so if we hit a CRD we should ideally wait until its api endpoint is available before considering it "created".

[aaronlevy]

/assign @abhinavdahiya

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: bparees
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: aaronlevy

If they are not already assigned, you can assign the PR to them by writing /assign @aaronlevy in a comment when ready.

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[bparees]

@aaronlevy @abhinavdahiya updated per your latest comments.

[abhinavdahiya]

/retest

[abhinavdahiya]

This gives operator root on cluster. why not just bind to cluster-admin role?

[bparees]

one reason i can think of is that this makes it explicit what permissions we actually need/want.

[legionus]

This gives operator root on cluster. why not just bind to cluster-admin role?

We want to explicitly list the permissions that are required. Speaking specifically about this, we need these permissions to create clusterroles. We need to do this dynamically since the number of interfaces registry uses is vary from version to version.

[wking]

/retest

[bparees]

it sounds like the CVO is going to pick up our image's manifest content automatically soon so i'm closing this so we don't end up installing the operator twice.