Authentik will not boot anymore

[samip5]

Describe the bug
It seems that for whatever reason the Authentik core will not want to boot.

To Reproduce
Steps to reproduce the behavior:

1. Update to 2023.6.1
2. Notice that it stops responding
3. Check logs and see that it fails to boot
4. See error

Expected behavior
I would have expected it to boot.

Logs

{"error":"authentik core not running yet","event":"failed to proxy to backend","level":"warning","logger":"authentik.router","timestamp":"2023-07-24T11:27:50Z"}
1690198078.8367748 [critical ] WORKER TIMEOUT (pid:21)        [gunicorn.error] 
1690198078.85575 [critical ] WORKER TIMEOUT (pid:22)        [gunicorn.error] 
1690198079.4460566 [warning  ] Worker with pid 21 was terminated due to signal 6 [gunicorn.error] 
1690198079.4468806 [warning  ] Worker with pid 22 was terminated due to signal 6 [gunicorn.error] 
1690198079.465585 [info     ] Booting worker with pid: 23    [gunicorn.error] 
1690198079.4971247 [info     ] Booting worker with pid: 24    [gunicorn.error] 
{"error":"authentik core not running yet","event":"failed to proxy to backend","level":"warning","logger":"authentik.router","timestamp":"2023-07-24T11:28:01Z"}
{"error":"authentik core not running yet","event":"failed to proxy to backend","level":"warning","logger":"authentik.router","timestamp":"2023-07-24T11:28:08Z"}
{"error":"authentik core not running yet","event":"failed to proxy to backend","level":"warning","logger":"authentik.router","timestamp":"2023-07-24T11:28:11Z"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1690198097.0827873, "path": "authentik.admin.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1690198097.1115623, "path": "authentik.crypto.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1690198097.123654, "path": "authentik.events.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1690198097.1378193, "path": "authentik.outposts.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1690198097.1707942, "path": "authentik.policies.reputation.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1690198097.438546, "path": "authentik.providers.scim.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1690198097.4600086, "path": "authentik.sources.ldap.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1690198097.5211961, "path": "authentik.sources.plex.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1690198097.5357234, "path": "authentik.stages.authenticator_static.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1690198097.5426495, "path": "authentik.stages.authenticator_totp.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1690198097.5540798, "path": "authentik.stages.authenticator_validate.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1690198098.9144554, "path": "authentik.blueprints.settings"}
{"event": "Booting authentik", "level": "info", "logger": "authentik.lib.config", "timestamp": 1690198098.9437504, "version": "2023.6.1"}
{"event": "Enabled authentik enterprise", "level": "info", "logger": "authentik.lib.config", "timestamp": 1690198098.9524992}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1690198098.9544783, "path": "authentik.enterprise.settings"}
{"error":"authentik core not running yet","event":"failed to proxy to backend","level":"warning","logger":"authentik.router","timestamp":"2023-07-24T11:28:20Z"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1690198101.5573373, "path": "authentik.admin.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1690198101.5747912, "path": "authentik.crypto.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1690198101.5902755, "path": "authentik.events.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1690198101.6025028, "path": "authentik.outposts.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1690198101.6251688, "path": "authentik.policies.reputation.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1690198101.6504257, "path": "authentik.providers.scim.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1690198101.6637986, "path": "authentik.sources.ldap.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1690198101.672482, "path": "authentik.sources.plex.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1690198101.687867, "path": "authentik.stages.authenticator_static.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1690198101.6956124, "path": "authentik.stages.authenticator_totp.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1690198101.7027323, "path": "authentik.stages.authenticator_validate.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1690198101.7475748, "path": "authentik.blueprints.settings"}
{"event": "Booting authentik", "level": "info", "logger": "authentik.lib.config", "timestamp": 1690198101.7570236, "version": "2023.6.1"}
{"event": "Enabled authentik enterprise", "level": "info", "logger": "authentik.lib.config", "timestamp": 1690198101.9477386}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1690198101.949856, "path": "authentik.enterprise.settings"}
{"event": "Loaded GeoIP database", "last_write": 1688989663.0, "level": "info", "logger": "authentik.events.geo", "pid": 24, "timestamp": "2023-07-24T11:28:29.422938"}
1690198109.6534352 [critical ] WORKER TIMEOUT (pid:23)        [gunicorn.error] 
1690198109.6757782 [critical ] WORKER TIMEOUT (pid:24)        [gunicorn.error] 
1690198109.7683585 [warning  ] Worker with pid 24 was terminated due to signal 6 [gunicorn.error] 
1690198110.3173668 [info     ] Booting worker with pid: 25    [gunicorn.error] 
1690198110.330595 [warning  ] Worker with pid 23 was terminated due to signal 6 [gunicorn.error] 
{"error":"authentik core not running yet","event":"failed to proxy to backend","level":"warning","logger":"authentik.router","timestamp":"2023-07-24T11:28:31Z"}
1690198112.078749 [info     ] Booting worker with pid: 26    [gunicorn.error] 

Version and Deployment (please complete the following information):

• authentik version: 2023.6.1
• Deployment: Helm

Additional context
It had been working fine ish, and now it's terminating due to signal 6 what ever that means..

[alexbarcelo]

The same happened to myself when upgrading to 2023.8.1 (I had a quite outdated 2023.5.x version and now proceeded to upgrade aggressively).

The error line seems to be the same in my scenario:

{"error":"authentik core not running yet","event":"failed to proxy to backend","level":"warning","logger":"authentik.router","timestamp":"2023-08-30T14:03:21Z"}

followed by gunicorn timeout error, rinse and repeat.

[alexbarcelo]

Update: after doing a hundred upgrade-downgrade procedures and acomplishing nothing, I left the system broken and 20 minutes later the issue solved itself.

Maybe the gunicorn timeout and the initialization time are close and you need a bit of luck? (I am running in a constrained kubernetes environment homelab, not very powerful). That is the only explanation I can come up with, but I really have no idea.

[jmtaillon]

I'm also on a rather constrained homelab and found a workaround for the initialization timeout. My environment started to do this a few days ago after running solid for about a month on this version and each gunicorn worker was hitting 100% cpu utilization during initialization. The restart of the workers would occur every 30 seconds and do it again (which is a gunicorn default timeout).

To work around this, I reduced the number of workers via the Authentik environment variables and also used the following variable to tell gunicorn to wait longer (you can go higher than this, but probably keep it reasonable).

GUNICORN_CMD_ARGS="--timeout=90"

There is no specific Authentik env variable for this, so this will only work so long as something isn't passed directly into gunicorn via the command line (see the gunicorn config docs for priority), but I imagine the gunicorn.config.py file would be the place any defaults or overrides would be set in the future. A dedicated Authentik config var for slower systems would be great!

[rizwan95]

I'm also on a rather constrained homelab and found a workaround for the initialization timeout. My environment started to do this a few days ago after running solid for about a month on this version and each gunicorn worker was hitting 100% cpu utilization during initialization. The restart of the workers would occur every 30 seconds and do it again (which is a gunicorn default timeout).

To work around this, I reduced the number of workers via the Authentik environment variables and also used the following variable to tell gunicorn to wait longer (you can go higher than this, but probably keep it reasonable).

GUNICORN_CMD_ARGS="--timeout=90"

There is no specific Authentik env variable for this, so this will only work so long as something isn't passed directly into gunicorn via the command line (see the gunicorn config docs for priority), but I imagine the gunicorn.config.py file would be the place any defaults or overrides would be set in the future. A dedicated Authentik config var for slower systems would be great!

This is the solution! Thank you very much!

[authentik-automation]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.