fix: Stop creating immutable resource label that breaks "tutor k8s" on Tutor version changes

[fghaas]

Through the commonLabels directive in kustomization.yml, all resources get a label named "app.kubernetes.io/version", which is being set to the Tutor version at the time of initial deployment.

When the user then subsequently progresses to a new Tutor version, Kubernetes attempts to update this label — but for Deployment, ReplicaSet, and DaemonSet resources, this is no longer allowed as of kubernetes/kubernetes#50808. This causes tutor k8s start (at the kubectl apply --kustomize step) to break with errors such as:

Deployment.apps "redis" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"openedx-JIONBLbtByCGUYgHgr4tDWu1", "app.kubernetes.io/managed-by":"tutor", "app.kubernetes.io/name":"redis", "app.kubernetes.io/part-of":"openedx", "app.kubernetes.io/version":"12.1.7"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

Simply removing the app.kubernetes.io/version label from commonLabels in kustomization.yml will permanently fix this issue for newly created Kubernetes deployments, which will "survive" any future Tutor version changes thereafter.

However, existing production Open edX deployments will need to throw the affected Deployments away, and re-create them.

Also, add the Tutor version as a resource annotation instead, using the commonAnnotations directive.

See also:
kubernetes/client-go#508
https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonlabels/
https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonannotations/

[regisb]

I think that your testing should be sufficient. Could you please just add a mention of a breaking change to CHANGELOG-nightly.md?

[fghaas]

Could you please just add a mention of a breaking change to CHANGELOG-nightly.md?

Done, can you take a look to see whether you judge that that sufficiently describes the change?

[regisb]

As a rule of thumb, in the changelog I prefer when we describe what we are fixing instead of how. The former is of more interest to the reader. Thus, a better changelog would be something along the lines of "Fix upgrading Kubernetes deployments when upgrading from one minor tutor version to the next".
But I don't want to block your PR for a changelog line, so let's merge this :)

[fghaas]

I agree, but in this case your abbreviated version doesn't contain any explanation as to why this is a breaking change. (Why should a fix to upgrades be a breaking change?)

[regisb]

Indeed. The changelog should also explain that the deployments should be deleted prior to upgrading.