-
Notifications
You must be signed in to change notification settings - Fork 153
Description
Problem
When upgrading the toolhive-operator-crds Helm chart, CRDs are not automatically updated due to Helm's default behavior. This means users must manually apply CRD updates using kubectl, which is:
- Not intuitive for users expecting
helm upgradeto work - Easy to miss, leading to confusion when new CRD fields are available in the chart but not in the cluster
- Inconsistent with the upgrade experience
Current Workaround
Users must manually update CRDs after upgrading:
kubectl apply -f <path-to-crds>Note: Do NOT use kubectl replace --force as it will delete and recreate the CRD, destroying all existing custom resources in the process.
Proposed Solution
Add a Helm pre-upgrade hook to the toolhive-operator-crds chart that automatically applies CRD updates during chart upgrades. This can be implemented using a Job that runs kubectl apply on the CRDs.
Example implementation:
apiVersion: batch/v1
kind: Job
metadata:
name: {{ include "toolhive-operator-crds.fullname" . }}-upgrade-crds
annotations:
"helm.sh/hook": pre-upgrade
"helm.sh/hook-weight": "-5"
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
spec:
template:
spec:
serviceAccountName: {{ include "toolhive-operator-crds.fullname" . }}-upgrade
restartPolicy: Never
containers:
- name: upgrade-crds
image: bitnami/kubectl:latest
command:
- /bin/sh
- -c
- |
kubectl apply -f /crds/
volumeMounts:
- name: crds
mountPath: /crds
volumes:
- name: crds
configMap:
name: {{ include "toolhive-operator-crds.fullname" . }}-filesImportant: The hook must use kubectl apply (not kubectl replace --force) to safely merge CRD updates without deleting existing resources.
Benefits
- Better UX:
helm upgradewill "just work" as users expect - Prevents issues: Ensures CRDs are always in sync with the chart version
- Prevents data loss: Using
kubectl applysafely updates CRDs without deleting custom resources - Consistency: Aligns with how many other operator charts handle CRD upgrades (e.g., cert-manager, prometheus-operator)
References
- Helm CRD Best Practices
- cert-manager's approach to CRD upgrades
- Recent example where this caused confusion: User upgraded to v0.0.43 expecting
insecureAllowHTTPfield to be available, but CRDs weren't actually updated
Alternative Considered
Document the manual upgrade process clearly in the chart README - but this is a poor UX compared to automatic upgrades.