prod: authentik rollout wedged — chart pins 2024.12.3 but DB migrated to 2026.5.5 (selfHeal downgrade loop; fuzefront app Degraded)

[izzywdev]

@claude

Summary

In prod (Contabo k3s, namespace fuzefront, Argo CD app fuzefront), the authentik-server rollout is permanently wedged and it is the sole remaining reason the fuzefront Argo Application is Degraded (the app is otherwise Synced, all other workloads Healthy). SSO is not down — the old 2026.5.5 pod still serves — but the rollout can never converge and Argo selfHeal keeps fighting it.

Root cause (verified against the live cluster + this repo)

The chart pins an older authentik image than the one that actually migrated the prod DB:

• deploy/helm/fuzefront/values.yaml pins authentik ... tag: "2024.12.3".
• 2026.5.5 appears nowhere in this repo (0 matches).

Sequence:

1. ~31h ago authentik was upgraded to 2026.5.5 directly on the cluster (out-of-band; never committed). That ran authentik's irreversible forward DB migrations and is the healthy 1/1 pod serving SSO now.
2. Argo selfHeal saw live (2026.5.5) != git (2024.12.3) and reverted the image to 2024.12.3 (new RS authentik-server-6df5b695ff, ~28h old). 2024.12.3 cannot run against the 2026.5.5-migrated DB, so it crashloops. The old 2026.5.5 RS (authentik-server-f6f79cf5f) lingers 1/1 only because the new pod never became Ready (maxUnavailable=0).

Live evidence:

Deployment authentik-server desired image: ghcr.io/goauthentik/server:2024.12.3
RS authentik-server-6df5b695ff  image 2024.12.3  desired=1 ready=0  (crashloop, ~28h)
RS authentik-server-f6f79cf5f   image 2026.5.5   desired=1 ready=1  (serving, ~31h)
authentik-worker                image 2024.12.3  (tasks erroring against DB)

Crash logs from the 2024.12.3 pods (schema is ahead of the code):

column authentik_core_authenticatedsession.session_key does not exist   (HINT: session_id)
column authentik_core_authenticatedsession.expiring does not exist
relation "authentik_events_systemtask" does not exist

The authentik-config-apply hook then fails on every sync (can't reach a healthy authentik), which is what pins the Argo app at Degraded.

Fix (GitOps — align the chart to the already-migrated DB)

Authentik DB migrations are one-way — you cannot downgrade to 2024.12.3. Roll the chart forward to match the DB:

# deploy/helm/fuzefront/values.yaml  (authentik server image)
-    tag: "2024.12.3"
+    tag: "2026.5.5"

Commit to master → Argo syncs → the 2026.5.5 RS becomes Ready → old RS scales to 0 → authentik-config-apply hook succeeds → fuzefront app goes Healthy. Must land via Git→Argo; do NOT kubectl set image (that out-of-band change is exactly what caused this — selfHeal reverts it).

Also align the non-prod references for consistency (they do not affect prod): docker-compose.yml, docker-compose.e2e.yml, and the deploy/e2e/provision-authentik-oidc.sh comments all still say 2024.12.3.

Acceptance criteria

• deploy/helm/fuzefront/values.yaml authentik server tag: "2026.5.5".
• New authentik-server RS on 2026.5.5 reaches 1/1; the 2024.12.3 RS scales to 0.
• authentik-worker on 2026.5.5, no schema errors in its log.
• authentik-config-apply hook completes; Argo fuzefront Application = Synced / Healthy.

STATE

• Done: root-caused in prod. Confirmed chart pins 2024.12.3, DB migrated to 2026.5.5 (healthy old RS proves the schema), Argo selfHeal reverting the out-of-band upgrade.
• Remaining: one-line values.yaml tag bump (+ optional non-prod alignment) → merge → Argo sync.
• Decisions needed by handler: confirm 2026.5.5 is the intended target version (it is the version already running + already migrated; forward-only, so this is the safe choice).
• Context: surfaced while driving the fuzefront Argo app to Healthy after the prod: chat-service migrate crash-loops — running against shared fuzefront_platform DB (fuzefront app Degraded) #499/prod: email-service rollout wedged — SMTP_USER/SMTP_PASS missing from fuzefront-secrets SealedSecret #500/prod: chat-db-migrate STILL fails after #508 — migration 001 has cross-DB FK to users/organizations (relation "users" does not exist) #510 chain (all resolved). This authentik regression is independent and predates that work (~28h). NOT caused by any manual action in this investigation.

DONE:

[claude]

Claude finished @izzywdev's task in 1s —— View job

---

I'll analyze this and get back to you.

[izzywdev]

PR #556 rolls the chart forward to authentik 2026.5.5 (deploy/helm/fuzefront/values.yaml; values-prod.yaml has no authentik tag override, so prod inherits it). This un-wedges the rollout.

Heads-up on the non-prod alignment the issue also asked for: bumping the docker-compose / E2E harness to 2026.5.5 broke CI (OIDC Plumbing E2E + E2E (sign-in)) with authorize returned HTTP 200 without redirect (consent flow?) -> POST /api/v1/security/session -> 503. authentik 2026.5.5 returns the final redirect as an HTTP-200 redirect challenge, and the security service's headless flow driver rejects it — a security-service code change, out of scope here. Reverted those files so this PR is a clean one-line prod fix; tracked the harness migration as #557.