breakfix: Deadlock starting with OIDC provider behind caddy

[pinpox]

Describe the issue

Caddy can't start if the OIDC provider itself is behind the proxy.

I'm running an OIDC provider (dex) behind caddy as a reverse proxy at login.mydomain.tld. Adding caddy-security, makes caddy fail to start with the following error:

Oct 04 21:26:13 kfbox caddy[2798169]: Error: loading initial config: loading new config: loading security app module: provision security: server initialization failed: failed configuring identity provider: failed to fetch metadata for OAuth 2.0 authorization server: Get "https://login.mydomain.tld/.well-known/openid-configuration": dial tcp 46.38.XXX.XX:443: connect: connection refused

This seems to be related because of a deadlock:
Caddy is not running -> Caddy can't reach login.mydomain.tld/.well-known/openid-configuration to start with caddy-security -> Caddy can't start

Configuration

Paste full Caddyfile below:

{
        order authenticate before respond
        order authorize before basicauth

        security {
                oauth identity provider github {env.GITHUB_CLIENT_ID} {env.GITHUB_CLIENT_SECRET}

                oauth identity provider generic {
                        realm generic
                        driver generic
                        client_id {env.GENERIC_CLIENT_ID}
                        client_secret {env.GENERIC_CLIENT_SECRET}
                        scopes openid email profile
                        base_auth_url https://login.mydomain.tld
                        metadata_url https://login.mydomain.tld/.well-known/openid-configuration
                }

                authentication portal myportal {
                        crypto default token lifetime 3600
                        crypto key sign-verify {env.JWT_SHARED_KEY}
                        cookie domain mydomain.tld
                        enable identity provider github
                        enable identity provider generic
                        ui {
                                links {
                                        "My Identity" "/whoami" icon "las la-user"
                                }
                        }

                        transform user {
                                match realm github
                                action add role authp/user
                                ui link "Test site 1" https://static-site-one.mydomain.tld/ icon "las la-star"
                                ui link "Test site 2" https://static-site-two.mydomain.tld/ icon "las la-star"
                        }

                        transform user {
                                match realm github
                                match sub github.com/pinpox
                                action add role authp/admin
                        }
                }
                authorization policy mygenericpolicy {
                        set auth url https://caddy-auth.mydomain.tld/oauth2/generic
                        crypto key verify {env.JWT_SHARED_KEY}
                        allow roles authp/admin authp/user
                        validate bearer header
                        inject headers with claims
                }

                authorization policy mypolicy {
                        set auth url https://caddy-auth.mydomain.tld/oauth2/github
                        crypto key verify {env.JWT_SHARED_KEY}
                        allow roles authp/admin authp/user
                        validate bearer header
                        inject headers with claims
                }
        }

        log {
                level ERROR
        }
}

mydomain.tld {
        log {
                output file /var/log/caddy/access-mydomain.tld.log
        }

        root * /nix/store/wlk5r9jzm2x0hnch42570kzbd2gk8vq4-page
        encode zstd gzip
        file_server
}

caddy-auth.mydomain.tld {
        log {
                output file /var/log/caddy/access-caddy-auth.mydomain.tld.log
        }

        authenticate with myportal
}

login.mydomain.tld {
        log {
                output file /var/log/caddy/access-login.mydomain.tld.log
        }

        reverse_proxy 127.0.0.1:5556
}

static-site-one.mydomain.tld {
        log {
                output file /var/log/caddy/access-static-site-one.mydomain.tld.log
        }

        authorize with mypolicy
        encode gzip
        root * /nix/store/r004sc6x1dmsld1a7igkpqjbrbk5vnm3-index.html/html
        file_server
}

static-site-two.mydomain.tld {
        log {
                output file /var/log/caddy/access-static-site-two.mydomain.tld.log
        }

        authorize with mygenericpolicy
        encode gzip
        root * /nix/store/n4jg2dxc9cc24wv8afn1f05ig37fxcmd-index.html/html
        file_server
}

Version Information

Provide output of caddy list-modules -versions | grep git below:

 v1.1.20

Expected behavior

Caddy should start and retry reaching the metadata url or start the reverse-proxy entries first, then the caddy-security related stuff

Additional context

Add any other context about the problem here.

[greenpau]

@pinpox , I am not sure whether this applies, but there are 2 options available for you: delay and retry.

https://authp.github.io/docs/authenticate/oauth/backend-oauth2-endpoint

Keeping it open and waiting on your response.

[ptman]

That is great, but even better would be infinite retries. What sort of status (/metrics ?) can one get about the state of authp?

[pinpox]

@pinpox , I am not sure whether this applies, but there are 2 options available for you: delay and retry.

https://authp.github.io/docs/authenticate/oauth/backend-oauth2-endpoint

Keeping it open and waiting on your response.

I managed to get it working with:

            delay_start 5
            retry_attempts 5
            retry_interval 10

Thanks for the hint!