DO / helm: Can't Upgrade

[boldtrn]

Bug description

I followed the upgrading instructions for DO/helm

helm upgrade -f values.yaml --timeout 20m --namespace posthog posthog posthog/posthog --atomic

Error: UPGRADE FAILED: release posthog failed, and has been rolled back due to atomic being set: post-upgrade hooks failed: job failed: BackoffLimitExceeded

When I tried to rerun the upgrade with a higher timeout value, as I thought that might have been the issue

helm upgrade -f values.yaml --timeout 120m --namespace posthog posthog posthog/posthog --atomic

Error: UPGRADE FAILED: release posthog failed, and has been rolled back due to atomic being set: post-upgrade hooks failed: warning: Hook post-upgrade posthog/templates/migrate.job.yaml failed: jobs.batch "posthog-migrate" already exists

Expected behavior

Run the latest version.

Environment

• PostHog Cloud
• self-hosted PostHog (ClickHouse-based), version/commit: please provide
• self-hosted PostHog (Postgres-based, legacy), version/commit: please provide

[jamesefhawkins]

Hey @tiina303 I thought maybe you could help here?

[tiina303]

Hi @boldtrn

The second error about the migrate job happens because it was left around from the previous upgrade attempt, we just need to kill that job before running the upgrade again kubectl delete job posthog-migrate -n posthog.

Regarding to why the initial upgrade failed I'm would need a bit more information:

1. What chart version are you currently on helm get manifest posthog | grep "HELM_INSTALL_INFO" -A1 (maybe it's a major upgrade that needs some manual steps to be performed too & that's why the upgrade failed)?
2. While you were running the upgrade did any of the pods fail/restart loop, what were the logs in those pods, were any pods unschedulable what was the kubectl get pod <pod> -o yaml output for those?

[boldtrn]

@tiina303 thanks a lot for the additional help.

Running kubectl delete job posthog-migrate -n posthog and then re-running the upgrade again fixed the issue.

Running helm get manifest posthog returns Error: release: not found. Not sure if this indicates a potential issue?

I will close this issue, as the actual problem is resolved, but I would be happy to provide some more information to find out the underlying issue.

[tiina303]

Running helm get manifest posthog returns Error: release: not found. Not sure if this indicates a potential issue?

Sorry I forgot the -n posthog flag, there release is typically installed in the posthog namespace, but if the issue is fixed that's not needed, I was trying to be proactive about further investigation in case it wasn't.