Run conformance tests for OpenShift

[stefanprodan]

If we can get sponsorship to spin up OpenShift clusters, we could run the bootstrap test from https://github.com/fluxcd/flux2/blob/main/tests/bootstrap/main.go with the OpenShift patches from https://fluxcd.io/flux/installation/configuration/openshift/.

Tasks:

• find a sponsor that can give us free OpenShift clusters that we can spin in GitHub Actions
• modify the bootstrap patch for OpenShift
• create a dedicated GitHub workflow to run the above

[errordeveloper]

You may find this terraform repo useful for sounding up clusters with OCP (the commercial stable edition, which comes with unlimited scale trial of 60days) as well as OKD (the community edition, which is not necessarily stable).
https://github.com/cilium/openshift-terraform-upi

The repo is archived now, but it was all very generic.

You can use 'opening-install' otherwise, but it can be a bit limited. That terrform repo contains a script that can download the installed command from any given version/edition, which can be handy.

[errordeveloper]

I will share the role definition once I get to my desk.

[stefanprodan]

@errordeveloper my hope is that Marc can provide us free OpenShift clusters that we can spin from GH Actions https://x.com/mccode/status/1758126362647490856?s=20

[errordeveloper]

This is what you will want:

apiVersion: v1
kind: List
items:
  - apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      name: flux-scc
    rules:
      apiGroups:
        - security.openshift.io
      resources:
        - securitycontextconstraints
      resourceNames:
        - nonroot
      verbs:
        - use
  - apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: flux-scc-source-controller
      namespace: flux-system
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: Role
      name: flux-scc
    subjects:
      - kind: ServiceAccount
        name: flux-source-controller
  - apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: flux-scc-kustomize-controller
      namespace: flux-system
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: Role
      name: flux-scc
    subjects:
      - kind: ServiceAccount
        name: flux-kustomize-controller
  - apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: flux-scc-helm-controller
      namespace: flux-system
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: Role
      name: flux-scc
    subjects:
      - kind: ServiceAccount
        name: flux-helm-controller

In case you want to tweak a few things, I used this to generate the YAML above.

cue export --out yaml - <<EOF
_controllers: [
	"source-controller",
	"kustomize-controller",
	"helm-controller",
]

_namespace: "flux-system"

_role: {
	apiVersion: "rbac.authorization.k8s.io/v1"
	kind:       "ClusterRole"
	metadata: name: "flux-scc"
	rules: {
		apiGroups: [
			"security.openshift.io",
		]
		resources: [
			"securitycontextconstraints",
		]
		resourceNames: [
			"nonroot",
		]
		verbs: [
			"use",
		]
	}
}

_roleBindings: {}

for c in _controllers {
	_roleBindings: (c): {
		apiVersion: "rbac.authorization.k8s.io/v1"
		kind:       "RoleBinding"
		metadata: {
			name:      "flux-scc-\(c)"
			namespace: _namespace
		}
		roleRef: {
			apiGroup: "rbac.authorization.k8s.io"
			kind:     "Role"
			name:     "flux-scc"
		}
		subjects: [{
			kind: "ServiceAccount"
			name: "flux-\(c)"
		}]
	}
}

apiVersion: "v1"
kind:       "List"
items: [
	_role,
	for _, roleBinding in _roleBindings {roleBinding},
]
EOF

I've not tested it, as I don't have an OpenShift cluster around right now, but it's very much trivial.

[errordeveloper]

It's also possible that ClusterRole just like flux-scc exists already, if that might be somehow convenient to avoid creating one.

[stefanprodan]

Thanks @errordeveloper we'll try using the ClusterRole you posted once we have an OpenShift env. If this passed the test, then we can add it to the docs, and maybe as a flag to bootstrap as people can run a single command on OpenShift.

[errordeveloper]

...maybe as a flag to bootstrap as people can run a single command on OpenShift.

That would be handy for sure, with openshift-install there is also a way to create a cluster with extra manifests. Which might make sense to document or use in e2es.

[marccampbell]

We have been building a product specifically for this (testing OpenShift quickly in CI). I've tried the flux installation on it, and it succeeded. I have some GitHub actions and would be happy to provide ongoing credits for OpenShift usage for Flux, and contribute to the CI change needed. Some product docs are at https://docs.replicated.com/vendor/testing-how-to, but you can ignore anything related to distributing app. We can and will enable just testing in CI for Flux.

[stefanprodan]

@marccampbell I've been looking at the example workflows, so to get a cluster running for ten minutes all we need is create-customer and create-cluster? https://github.com/replicatedhq/replicated-actions/blob/main/example-workflows/development-helm.yaml#L63-L86