Investigate downtime during deployment

[guidoiaquinti]

Bug description

A customer running on AWS is reporting some downtime during helm operations. They are running with 2 replicas for web/events.

Expected behavior

No downtime expected

How to reproduce

"It appears that helm creates two new nodes, and terminates the old ones. But it terminates them before the new pods become ready / healthy. For example I can see this shortly after an upgrade, causing NGINX to show a 503 for all requests."

NAME                                                READY   STATUS    RESTARTS   AGE
posthog-web-c4bc4b487-4msjb                         0/1     Running   0          54s
posthog-web-c4bc4b487-cxzhn                         0/1     Running   0          54s

Environment

• Deployment platform (gcp/aws/...): AWS (possibly everywhere)
• Chart version/commit: latest
• Posthog version: latest

Additional context

• See private Slack channel

[Legion2]

I think the downtime is related to the way the deployments are added via Helm, all deployments are annotated as helm hooks, so helm will create and delete them as part of this lifecycle.

charts-clickhouse/charts/posthog/templates/web-deployment.yaml

Lines 11 to 15 in 4273cd1

	# This is what defines this resource as a hook. Without this line, the
	# job is considered part of the release.
	"helm.sh/hook": "post-install,post-upgrade"
	"helm.sh/resource-policy": "keep"
	"helm.sh/hook-weight": "1"

I think this is a mistake, that the deployments are configured as hooks.

[guidoiaquinti]

I think the downtime is related to the way the deployments are added via Helm, all deployments are annotated as helm hooks, so helm will create and delete them as part of this lifecycle.

charts-clickhouse/charts/posthog/templates/web-deployment.yaml

Lines 11 to 15 in 4273cd1

	# This is what defines this resource as a hook. Without this line, the
	# job is considered part of the release.
	"helm.sh/hook": "post-install,post-upgrade"
	"helm.sh/resource-policy": "keep"
	"helm.sh/hook-weight": "1"

I think this is a mistake, that the deployments are configured as hooks.

On point! Here's an attempt to remove them but it's currently blocked by some issues in posthog/posthog