Cannot upgrade from admissionWebhook:disabled to admissionWebhook:enabled when in DB mode

[agringeri]

When running Kong in DB-mode with migration jobs enabled, it is not possible to enable the admission webhook from previously disabled state. The webhook-cert is a volume that is mounted to the pre-upgrade-migration job, and that job must run to completion before any other templates are generated (helm.sh/hook: "pre-upgrade"). However, the pod cannot start because the webhook-cert can't be mounted since the key-pair/secret has not been generated, and thus the upgrade can't complete successfully.

A few possible solutions/workarounds:

• Don't run the pre-upgrade migration job when enabling the webhook, and re-enable it later. More of a workaround than a solution
• Don't mount the webhook-cert to the migration jobs, since it isn't needed.
• Add a pre-upgrade helm hook to the key pair secret (with a lower hook-weight than the migrations-job), so that it exists before the pre-upgrade-migration job runs. Side-effect of disassociating the secret with the helm release.

[rainest]

Yeah, this 🐔 🥚 pattern of issue is something we run into fairly often: something's needed, but won't be created until the main deployment gets it, and that change will only occur after hooks run, and since those hooks require the thing in question...

Creating those resources in the hooks doesn't work well for the reason you noted: due to Helm treating hook resources as unmanaged, we get stuck between the choice of having the resource be available during the hook and having it managed as part of the release, and we often want the latter.

The middle option (don't mount this for migration jobs because they don't need it) is probably ideal, but is hard to do well. As-is most of that mount logic doesn't pay attention to what type of Kong Pod it's used for, and while it'd be good to have a more advanced system that separates out mounts that are always needed (e.g. plugin code, which can contain migrations) and that are only only needed for the main Deployment Pods, adding it would be a fairly significant refactor of the templates.

The first/workaround option is what I'd suggest for now. While it's not seamless, it's usually fairly simple now that we provide toggles for the migration jobs. Migration jobs are only necessary if you're upgrading the Kong image version, so for any other change they do spawn by default but are no-ops.

If you disable them when enabling the webhook, are you able to helm upgrade successfully, and do you see any issues if you re-enable them and upgrade again? The phased upgrade instructions for 1.9 are pretty much that, albeit for a different purpose.

Assuming that works, we should probably add a generic version of those instructions to the FAQ--something along the lines of "if you see pre-upgrade jobs failing because they need some ConfigMap or whatever, disable them and run a Helm upgrade without changing your Kong image version: that first upgrade should create the resource in question, and you can re-enable migration jobs after."

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.