Skip to content

Authentik worker exhausts app-db connections on startup #31

Description

@pounde

Problem

The Authentik Celery/Dramatiq worker exhausts all available PostgreSQL connections on app-db within 10-15 seconds of starting. Once saturated, Authentik returns 500 errors on all requests, breaking LDAP authentication, forward auth, and the entire downstream startup chain (init-ldap-ready times out → TAK Server never starts).

What We Know

The worker creates hundreds of psycopg3 connection pools during startup. We observed pool numbers up to pool-407 in the worker logs. Each pool holds at least one connection, easily exceeding max_connections=200.

It's not caused by any FastTAK code change. We tested incrementally on main, on the pre-LDAP-race-fix commit, and on the feat/deploy-modes branch — identical behavior in all cases. The worker alone (no other services) saturates the DB in seconds.

CONN_MAX_AGE is irrelevant. Tested with values of 300, 1, and 0 (Django default). The psycopg3 pools inside Dramatiq's PostgreSQL scheduler don't respect Django's connection lifecycle. Connections are opened and never released regardless of the setting.

Fresh DB is fine, accumulated state triggers it. After docker compose down -v (destroying volumes), a fresh startup holds steady at ~16 connections. With accumulated state from normal usage (even low-activity dev — fewer than 100 users), the startup burst exceeds 200 connections and never recovers.

A restart of Authentik (not the DB) temporarily fixes it. Restarting just the Authentik containers clears the stale pools, and reconnection is more gradual. But a cold start (DB + Authentik together) always hits the ceiling if there's accumulated state.

What We're Not Sure About

  • What specific accumulated state triggers the burst? Expired sessions? Event logs? Outpost state? Dramatiq task queue? We haven't identified which tables/rows scale the startup connection count.
  • Is this version-specific? We're on Authentik 2026.2.1. Since 2025.8, Authentik migrated tasks to Postgres, and 2025.10 moved caching + embedded outpost to Postgres ("~50% more database connections" per release notes). This may be a regression in newer versions.
  • Does Authentik have any internal cleanup/housekeeping that should prevent this accumulation? If so, it's not keeping up.

Reproduction

  1. Run the stack for a few days with normal activity (user creation, enrollment, logins)
  2. docker compose down (without -v — preserving volumes)
  3. docker compose up -d app-db redis → wait for healthy
  4. docker compose up -d authentik-server → stable at ~20 connections
  5. docker compose up -d authentik-worker → 200+ connections in 10-15 seconds, never recovers

Workaround: docker compose down -v to destroy volumes, then fresh start. Not acceptable for production.

Potential Fixes

1. PgBouncer (most promising)

Add a PgBouncer container between Authentik and app-db in transaction-pool mode. Caps real DB connections at ~20 regardless of how many client connections the worker opens. One container, three env var changes. Node-RED stays direct to app-db.

2. Increase max_connections + monitor

Bump max_connections to 500 and add monitoring/alerting when connections approach the limit. Band-aid — the worker may still exceed 500 with enough accumulated state.

3. Periodic DB maintenance

Scheduled cleanup of Authentik's accumulated state (expired sessions, old events, stale task queue). Requires understanding which tables are the trigger — not yet identified.

4. Delayed worker startup

Start authentik-worker after all other services are healthy, so the connection storm doesn't block the startup chain. Doesn't fix the root cause but limits blast radius.

Upstream

This is a known, unresolved issue in Authentik:

Authentik provides no configuration to limit the worker's internal psycopg3 pool size. CONN_MAX_AGE only controls Django ORM connections, not the Dramatiq scheduler's pools.

Environment

  • Authentik 2026.2.1
  • PostgreSQL 15 (PostGIS) via app-db
  • max_connections=200
  • macOS (ARM64, Docker Desktop)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions