Migrate from fluxcd-community charts to Flux-Operator

[kingdonb]

This is close enough to what we discussed at the Cozystack meeting today, to push as a draft

The flux-operator chart is much simpler

• I will test upgrade locally on my home lab cozystack ☑️

I think we can just use kubectl apply twice, and avoid creating a second "flux-instances" package in core for now (edit: it worked!)

[kingdonb]

At this point I have tested it out locally enough to know that we need at least spec.distribution.registry in the FluxInstance - which is a shame, we'll have to figure out another way that users can add their own settings for registry here, in case they bought Flux Enterprise ;)

Otherwise, users should be able to simply edit the FluxInstance in-place and have their edits merged in, I'll begin testing that. The migration was a success, the first thing that I expected happened once I corrected the error in the FluxInstance mentioned above, was the two image controllers got torn down. But that didn't happen - This is a default configuration in Flux, they are "extra controllers"

I can try using this for a while to see if there are any issues, if it enables all of the customizability that I was hoping, give it a few install-tests, but unless we uncover something I haven't already seen, this is going to be a very easy transition!

[kingdonb]

I pinned the version 2.2.x here because I imagine we will want to pin some version, and I know the 2.2.x version is the one that's currently used. We may not need any migration for the 2.3 upgrade because Kubernetes API does the upgrade transparently, from v2beta2 to v2, without intervention.

But these templates will need to be changed at least:

% egrep -ir helm.toolkit.fluxcd.io/v2beta2 *
packages/core/platform/templates/helmreleases.yaml:apiVersion: helm.toolkit.fluxcd.io/v2beta2
packages/core/platform/templates/apps.yaml:{{- if .Capabilities.APIVersions.Has "helm.toolkit.fluxcd.io/v2beta2" }}
packages/core/platform/templates/apps.yaml:{{- $tenantRoot = lookup "helm.toolkit.fluxcd.io/v2beta2" "HelmRelease" "tenant-root" "tenant-root" }}
packages/core/platform/templates/apps.yaml:apiVersion: helm.toolkit.fluxcd.io/v2beta2

I left that stuff alone and opted for the version pin instead, we can upgrade Flux later, or ideally before the next release so we are caught up.

It should be easy enough to test the upgrade now, I guess I can build that out in a separate PR (and try to confirm if we don't really need any migrations!)

[kvaps]

Makes sense, thank you

[kingdonb]

(I added them to the components list, that worked)

[kvaps]

Now we have chicken-and-egg problem, Fluxcd can't be installed during the initial installation as CNIs are not installed yet, but CNI can't be installed because this construction expects that HelmRelease CR will be existing in a cluster

cozystack/scripts/installer.sh

Lines 25 to 32 in 838bee5

	install_basic_charts() {
	if [ "$BUNDLE" = "paas-full" ] || [ "$BUNDLE" = "distro-full" ]; then
	make -C packages/system/cilium apply resume
	fi
	if [ "$BUNDLE" = "paas-full" ]; then
	make -C packages/system/kubeovn apply resume
	fi
	}

cozystack/scripts/package-system.mk

Lines 10 to 11 in 838bee5

	apply: suspend ## Apply Helm release to a Kubernetes cluster
	kubectl get hr -n $(NAMESPACE) $(NAME) -o jsonpath='{.spec.values}' | helm upgrade -i -n $(NAMESPACE) $(NAME) . -f -

here is part of my cozystack log:

make: Leaving directory '/cozystack/packages/core/platform'
make: Entering directory '/cozystack/packages/core/fluxcd'
helm template -n cozy-fluxcd fluxcd . --no-hooks --dry-run=server -a admissionregistration.k8s.io/v1 -a apiextensions.k8s.io/v1 -a apiregistration.k8s.io/v1 -a apps/v1 -a authentication.k8s.io/v1 -a authorization.k8s.io/v1 -a autoscaling/v1 -a autoscaling/v2 -a batch/v1 -a certificates.k8s.io/v1 -a coordination.k8s.io/v1 -a discovery.k8s.io/v1 -a events.k8s.io/v1 -a flowcontrol.apiserver.k8s.io/v1 -a flowcontrol.apiserver.k8s.io/v1beta3 -a fluxcd.controlplane.io/v1 -a networking.k8s.io/v1 -a node.k8s.io/v1 -a policy/v1 -a rbac.authorization.k8s.io/v1 -a scheduling.k8s.io/v1 -a storage.k8s.io/v1 -a v1 | kubectl apply -n cozy-fluxcd -f-
serviceaccount/fluxcd-flux-operator unchanged
customresourcedefinition.apiextensions.k8s.io/fluxinstances.fluxcd.controlplane.io unchanged
clusterrolebinding.rbac.authorization.k8s.io/fluxcd-flux-operator unchanged
service/fluxcd-flux-operator unchanged
deployment.apps/fluxcd-flux-operator configured
kubectl apply -n cozy-fluxcd -f flux-instance.yaml
fluxinstance.fluxcd.controlplane.io/flux unchanged
make: Leaving directory '/cozystack/packages/core/fluxcd'
error: the server doesn't have a resource type "helmrepositories"
time="2024-06-15T15:17:59Z" level=fatal msg="failed to run" err="exit status 1"

[kvaps]

I was wrong, it seems happening right on the next step, here

cozystack/scripts/installer.sh

Line 46 in 838bee5

kubectl annotate helmrepositories.source.toolkit.fluxcd.io -A -l cozystack.io/repository reconcile.fluxcd.io/requestedAt=$(date +"%Y-%m-%dT%H:%M:%SZ") --overwrite

[kingdonb]

Perhaps we can add another wait step - the Flux-Operator FluxInstance has a ready status, let me take a look...

[kingdonb]

We could add the wait in the package itself, or in the installer script. I'll give it a local test, I built a new installer image here and I have a little bit of time I can try wiping and reloading the cluster, and follow the logs 👍

Thanks for the review!

[kingdonb]

Waiting where I have placed it makes this fail... I'll work it out and let you know when it's worked out here

[kvaps]

Is it possible to not hardcode the domain? Users might use the different one

[kingdonb]

We can make our own local values in the parent chart, and keep them for the FluxInstance - sure, I think so.

[kingdonb]

Everything that happens in the chart now is either a parameter or an override. 🎸 🚀 🪨

[kingdonb]

Maybe we can figure out how to pass-through to accept the domain as a parameter from the cozystack ConfigMap, before this merges - this will help me know how to add the distribution.registry later in a future build 👍

[kvaps]

@stefanprodan provided an elegant solution for removing cluster domain dependency from fluxcd:
controlplaneio-fluxcd/flux-operator#34 (comment)

I'd like to go this way right now as we won't break the exisiting user flow.

But it seems we still have to add cluster-domain and kubernetes-endpoint options into cozystack configmap.
So if specified, it must be used by fluxcd and other applications, because in some cases this might be required.

I'll prepare an issue to keep this in mind.

[kvaps]

done #168

[kingdonb]

I'm going to move 6d71499 (now e51f17b) to #167 and rebase that one on this one.

This is tested now, no errors and it works in a fresh install. It could use some upgrade testing, I think, but should also work. The upgrade was the first thing I tested.

[kingdonb]

I think that #167 is mergeable now, or this one.

But be aware that flux-operator chart is out of sync with upstream, as these PRs haven't merged yet:

• Helm chart: add hostNetwork option controlplaneio-fluxcd/charts#10
• flux-operator: Add extraEnvs and hostNetwork options controlplaneio-fluxcd/charts#11

They will take another release of flux-operator, which won't happen before Monday anyway, so I suggest we can wait for it then we can run make update so the version numbers are in sync (and we are not making a fork later, in case something over there doesn't merge as-is.)

[stefanprodan]

You may want to wait for controlplaneio-fluxcd/flux-operator#33 to get automated updates going forward. I will do a release with this next week, then you can add distribution.artifact to your FluxInstance template.