ConfigMap values reconciliation doesn't work after upgrade on EKS cluster

[saeedfazal]

Upgrade wego core to wego enterprise on EKS cluster.

The commands used for upgrade are as follows.

gitops install
pctl install --flux-namespace wego-system

gitops app add . --auto-merge=true
git pull

kubectl apply -f test/utils/scripts/entitlement-secret.yaml
kubectl create secret docker-registry docker-io-pull-secret \
--namespace wego-system \
--docker-username="${DOCKER_IO_USER}" \
--docker-password="${DOCKER_IO_PASSWORD}" 
kubectl create secret generic git-provider-credentials \
--namespace=wego-system \
--from-literal="GIT_PROVIDER_TOKEN=${GITHUB_TOKEN}"

WORKER_NODE=$(kubectl get node --selector='!node-role.kubernetes.io/master' -o name | head -n 1)
kubectl label "${WORKER_NODE}" wkp-database-volume-node=true

mccp upgrade

Merge the upgrade pull request. The wego core will be upgraded to enterprise after sometime.
Now change some of the values in weave-gitops-enterprise/artifacts/mccp-chart/helm-chart/ConfigMap.yaml e.g. natsURL and repositoryURL. Push/merge the changed values. The changed config map values never get reconciled.

Moreover the application dependency graph behaves inconsistently on EKS cluster and sometime shows the full graph and sometime not.

You can access he test EKS cluster with this issue at:

http://a1a7234aef8c04d439a1652bdd4f1e33-1161623819.eu-central-1.elb.amazonaws.com/
Cluster name: wge-upgrade
Cluster repository: https://github.com/saeedfazal/eks-cluster

NB: The same changes take effect on a kind cluster but they don't work on EKS cluster.

[foot]

If you delete the "relevant pods" e.g. capi-service after making and syncing the change does it work? 🤔 ..

e.g.

1. make sure the configmap changes are definitely in the cluster (kubectl get configmap...)
2. restart capi-service pod

[foot]

Believe this is because pctl creates the helmrelease w/ an interval of 0

From https://fluxcd.io/docs/components/helm/helmreleases/#reconciliation

ConfigMap and Secret values overrides. Changes to these do not trigger an immediate reconciliation, but will be handled upon the next reconciliation. This is to avoid a large number of upgrades occurring when multiple resources are updated.

But the next reconciliation never happens:

1. Fix pctl so it adds a non-zero interval
2. fix instructions to tell user to force a reconciliation after updating the configmap: flux reconcile helmrelease weave-gitops-enterprise-mccp-chart -n wego-system

[saeedfazal]

Tested - reconciliation works with non-zero helm release interval.

[foot]

Fixed we new upgrade command in weaveworks/weave-gitops#1069