"tutor k8s" blows up on Tutor version update #531
Comments
|
It should probably be noted that if I understand correctly, the thing that's breaking here is not the attempted change to |
|
Hey Florian, thanks for your feedback on your usage of Kubernetes. I have seen the issues, pull requests and discussion topics you created -- sorry if I did not find the time to answer until now. I think it's great that you are giving us feedback on actual usage of Tutor running on k8s; this does not happen very frequently. Thus, I'd like to take this opportunity to allocate some time to improve the Regarding the specific issue of common labels that we are discussing here:
Based on these observations, can I suggest that you open a PR on top of the nightly branch of Tutor? |
I think the best approach here is to use a |
|
Right, thanks for bearing with me. I'm not too excited by the perspective of adding a labelTransformer object -- which would resolve our issue, but introduce yet another layer of complexity. Instead, could we move the tutor version from the commonLabels to the commonAnnotations? If we don't use the tutor version for querying, then maybe it makes more sense to consider it as an annotation? |
|
Right, if you don't actually want to query or reference anything by Tutor version, then I suppose it could also be an annotation. (I haven't used this before, mind you, but it does look like a sensible idea to me. :) ) |
|
Great. Would you have the time to try this and open a PR? If not I can do it myself. |
…n Tutor version changes Through the commonLabels directive in kustomization.yml, all resources get a label named "app.kubernetes.io/version", which is being set to the Tutor version at the time of initial deployment. When the user then subsequently progresses to a new Tutor version, Kubernetes attempts to update this label — but for Deployment, ReplicaSet, and DaemonSet resources, this is no longer allowed as of kubernetes/kubernetes#50808. This causes "tutor k8s start" (at the "kubectl apply --kustomize" step) to break with errors such as: Deployment.apps "redis" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"openedx-JIONBLbtByCGUYgHgr4tDWu1", "app.kubernetes.io/managed-by":"tutor", "app.kubernetes.io/name":"redis", "app.kubernetes.io/part-of":"openedx", "app.kubernetes.io/version":"12.1.7"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable Simply removing the app.kubernetes.io/version label from kustomization.yml will permanently fix this issue for newly created Kubernetes deployments, which will "survive" any future Tutor version changes thereafter. However, *existing* production Open edX deployments will need to throw the affected Deployments away, and re-create them. Also, add the Tutor version as a resource annotation instead, using the commonAnnotations directive. See also: kubernetes/client-go#508 https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonlabels/ https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonannotations/ Fixes overhangio#531.
|
I don't have bandwidth to test this today, but I've created a PR against |
|
So, I've not been able to test with
and the upgrade went through without a hitch. |
…n Tutor version changes Through the commonLabels directive in kustomization.yml, all resources get a label named "app.kubernetes.io/version", which is being set to the Tutor version at the time of initial deployment. When the user then subsequently progresses to a new Tutor version, Kubernetes attempts to update this label — but for Deployment, ReplicaSet, and DaemonSet resources, this is no longer allowed as of kubernetes/kubernetes#50808. This causes "tutor k8s start" (at the "kubectl apply --kustomize" step) to break with errors such as: Deployment.apps "redis" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"openedx-JIONBLbtByCGUYgHgr4tDWu1", "app.kubernetes.io/managed-by":"tutor", "app.kubernetes.io/name":"redis", "app.kubernetes.io/part-of":"openedx", "app.kubernetes.io/version":"12.1.7"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable Simply removing the app.kubernetes.io/version label from kustomization.yml will permanently fix this issue for newly created Kubernetes deployments, which will "survive" any future Tutor version changes thereafter. However, *existing* production Open edX deployments will need to throw the affected Deployments away, and re-create them. Also, add the Tutor version as a resource annotation instead, using the commonAnnotations directive. See also: kubernetes/client-go#508 https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonlabels/ https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonannotations/ Fixes overhangio#531.
…n Tutor version changes Through the commonLabels directive in kustomization.yml, all resources get a label named "app.kubernetes.io/version", which is being set to the Tutor version at the time of initial deployment. When the user then subsequently progresses to a new Tutor version, Kubernetes attempts to update this label — but for Deployment, ReplicaSet, and DaemonSet resources, this is no longer allowed as of kubernetes/kubernetes#50808. This causes "tutor k8s start" (at the "kubectl apply --kustomize" step) to break with errors such as: Deployment.apps "redis" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"openedx-JIONBLbtByCGUYgHgr4tDWu1", "app.kubernetes.io/managed-by":"tutor", "app.kubernetes.io/name":"redis", "app.kubernetes.io/part-of":"openedx", "app.kubernetes.io/version":"12.1.7"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable Simply removing the app.kubernetes.io/version label from kustomization.yml will permanently fix this issue for newly created Kubernetes deployments, which will "survive" any future Tutor version changes thereafter. However, *existing* production Open edX deployments will need to throw the affected Deployments away, and re-create them. Also, add the Tutor version as a resource annotation instead, using the commonAnnotations directive. See also: kubernetes/client-go#508 https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonlabels/ https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonannotations/ Fixes #531.
|
Closed by #533. |
…n Tutor version changes Through the commonLabels directive in kustomization.yml, all resources get a label named "app.kubernetes.io/version", which is being set to the Tutor version at the time of initial deployment. When the user then subsequently progresses to a new Tutor version, Kubernetes attempts to update this label — but for Deployment, ReplicaSet, and DaemonSet resources, this is no longer allowed as of kubernetes/kubernetes#50808. This causes "tutor k8s start" (at the "kubectl apply --kustomize" step) to break with errors such as: Deployment.apps "redis" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"openedx-JIONBLbtByCGUYgHgr4tDWu1", "app.kubernetes.io/managed-by":"tutor", "app.kubernetes.io/name":"redis", "app.kubernetes.io/part-of":"openedx", "app.kubernetes.io/version":"12.1.7"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable Simply removing the app.kubernetes.io/version label from kustomization.yml will permanently fix this issue for newly created Kubernetes deployments, which will "survive" any future Tutor version changes thereafter. However, *existing* production Open edX deployments will need to throw the affected Deployments away, and re-create them. Also, add the Tutor version as a resource annotation instead, using the commonAnnotations directive. See also: kubernetes/client-go#508 https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonlabels/ https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonannotations/ Fixes #531.
…n Tutor version changes Through the commonLabels directive in kustomization.yml, all resources get a label named "app.kubernetes.io/version", which is being set to the Tutor version at the time of initial deployment. When the user then subsequently progresses to a new Tutor version, Kubernetes attempts to update this label — but for Deployment, ReplicaSet, and DaemonSet resources, this is no longer allowed as of kubernetes/kubernetes#50808. This causes "tutor k8s start" (at the "kubectl apply --kustomize" step) to break with errors such as: Deployment.apps "redis" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"openedx-JIONBLbtByCGUYgHgr4tDWu1", "app.kubernetes.io/managed-by":"tutor", "app.kubernetes.io/name":"redis", "app.kubernetes.io/part-of":"openedx", "app.kubernetes.io/version":"12.1.7"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable Simply removing the app.kubernetes.io/version label from kustomization.yml will permanently fix this issue for newly created Kubernetes deployments, which will "survive" any future Tutor version changes thereafter. However, *existing* production Open edX deployments will need to throw the affected Deployments away, and re-create them. Also, add the Tutor version as a resource annotation instead, using the commonAnnotations directive. See also: kubernetes/client-go#508 https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonlabels/ https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/commonannotations/ Fixes #531.
Bug description
As one moves from one version of Tutor to another (I just saw this going from Tutor 12.1.6 to 12.1.7),
tutor k8sattempts to update theapp.kubernetes.io/versionlabel selector on all resources. As explained in kubernetes/client-go#508 (and probably elsewhere), this is not allowed for Deployments, causingtutor k8s startto break if applied to any Open edX environment previously deployed with an earlier version of Tutor. (Even though it will happily update the labels on the PVCs, ConfigMaps, and Services it touches before it gets to Deployments.)How to reproduce
tutor k8s quickstarttutor k8s start(orquickstart)Environment
Tutor 12.1.6 and 12.1.7, running against Kubernetes 1.18.16.
Additional context
Removing the offending line from
tutor/templates/kustomization.ymlwould remove this problem for newly created deployments. That is to say, any Open edX environment that's deployed to Kubernetes after this line has been removed from the Kustomization template will not have the label set, and won't break on Tutor updates. But any running cluster that does have it set it now, will break on a Tutor version update even after the line is removed, because Kustomization then tries to remove the label — which is also not allowed.Thus, removing the line from
tutor/templates/kustomization.ymlwould constitute a breaking change: users with existing Tutor-deployed Open edX environments will need to throw away all of their Deployments and then re-create them.The text was updated successfully, but these errors were encountered: