Skip to content

Troubleshooting

Jump to bottom
fuomag9 edited this page Sep 26, 2026 · 8 revisions

Troubleshooting

When things don't work.

Table of Contents

  1. Installation Issues
  2. Authentication Issues
  3. Certificate Issues
  4. OAuth Issues
  5. Permission Issues
  6. Networking Issues
  7. Database Issues
  8. Performance Issues
  9. Secret Decryption Issues
  10. Forward Auth Issues
  11. Instance Sync Issues
  12. WAF Issues

Installation Issues

Container Won't Start

Symptom: Container exits immediately after starting

Solutions:

  1. Check logs:

    docker compose logs web
docker compose logs caddy

  2. Common causes:

  3. Verify environment variables:

    docker compose config

  4. Check for port conflicts:

    lsof -i :3000
lsof -i :80
lsof -i :443

Compose Refuses to Start: Required Variable Missing

Error: docker compose up stops before creating any container with a message ending in ERROR - SESSION_SECRET is required, ERROR - ADMIN_PASSWORD is required or ERROR - CLICKHOUSE_PASSWORD is required

Cause: .env.example leaves SESSION_SECRET, ADMIN_PASSWORD and CLICKHOUSE_PASSWORD empty (since v1.13.1), and docker-compose.yml refuses to start while a required one is empty. CLICKHOUSE_PASSWORD is only required while the clickhouse profile is active.

Solution: fill them in .env:

openssl rand -base64 32   # use one output for SESSION_SECRET, another for CLICKHOUSE_PASSWORD

Choose your own ADMIN_PASSWORD (12+ characters, upper- and lowercase, a digit and a symbol), then run docker compose up -d.

Database Migration Errors

Symptom: Migration failed or database errors on startup

Solutions:

Option 1: Reset database (deletes all data):

docker compose down
rm -rf data/caddy-proxy-manager.db*
docker compose up -d

Option 2: Check permissions:

ls -la data/
chmod 600 data/caddy-proxy-manager.db

Option 3: Restore from backup:

cp backup/caddy-proxy-manager.db data/
docker compose restart web

Port Already in Use

Symptom: Error: bind: address already in use

Solutions:

  1. Find process using port:

    # Port 3000
lsof -i :3000

# Port 80
lsof -i :80

# Port 443
lsof -i :443

  2. Stop conflicting service:

    sudo systemctl stop apache2  # or nginx, etc.

  3. Or change ports in docker-compose.yml:

    ports:
  - "8080:3000"  # Web UI on 8080
  - "8000:80"    # HTTP on 8000
  - "8443:443"   # HTTPS on 8443

    If you move Caddy off 80/443 and use forward auth, set FORWARD_AUTH_ALLOWED_PORTS=8000,8443 (see Portal Says the Port Is Not Allowed); if you move the web UI, update BASE_URL.

Authentication Issues

App Refuses to Start: SESSION_SECRET Required

Error: SESSION_SECRET environment variable is required in production

Solution:

  1. Generate session secret:

    openssl rand -base64 32

  2. Add to .env file:

    SESSION_SECRET="<output of openssl rand -base64 32>"

  3. Recreate the web container (docker compose restart does not re-read .env):

    docker compose up -d

App Refuses to Start: Example Secret or Password

Error: SESSION_SECRET is using a known insecure placeholder value. … or Admin credentials validation failed: with ADMIN_PASSWORD is an example value from the documentation; choose your own password

Cause: Since v1.13.1, the web container refuses the placeholder secrets (including your-secure-session-secret-here-min-32-chars from older .env.example files) and the example admin passwords from earlier READMEs and .env.example files.

Solution:

  1. Generate a new secret with openssl rand -base64 32 and choose your own ADMIN_PASSWORD.
  2. Run docker compose up -d.

Stored secrets encrypted under a placeholder secret are re-encrypted with the new one automatically on that start; nothing has to be re-entered. Replacing any other secret needs SESSION_SECRET_PREVIOUS (see Security Configuration#secret-rotation).

Password Validation Failed

Error: Admin credentials validation failed: followed by e.g. ADMIN_PASSWORD must be at least 12 characters long or ADMIN_PASSWORD must include both uppercase and lowercase letters. In the UI and API: "Password must be at least 12 characters long, …" (or "New password …" when changing it).

Solution:

Ensure password meets ALL requirements:

  • 12+ characters (at most 256 for passwords set in the app)
  • Uppercase letters (A-Z)
  • Lowercase letters (a-z)
  • Numbers (0-9)
  • Special characters (!@#$%^&*)
  • Not an example password from the documentation

Example:

ADMIN_PASSWORD="<choose-your-own: 12+ chars, upper, lower, digit, symbol>"

Since v1.13.1 the same policy also applies to users created by an admin and to self-registration; password changes already enforced it.

Login Fails with Correct Credentials

Symptom: Can't log in even with correct username/password

Solutions:

  1. Check environment variables loaded:

    docker compose exec web env | grep ADMIN

  2. Recreate the web container after .env changes (docker compose restart keeps the old values):

    docker compose up -d

  3. Password changed in the UI? Since v1.13.1, ADMIN_PASSWORD is applied only when the admin is created or when ADMIN_USERNAME/ADMIN_PASSWORD change, so a password changed on the Profile page is kept across restarts and the old environment value no longer works.

    • To reset it, set ADMIN_PASSWORD to a new value and run docker compose up -d. This also restores the admin role, re-activates the account if it was disabled and signs out its sessions.
    • On the first start after upgrading, a stored password that differs from ADMIN_PASSWORD is kept, with the warning ADMIN_PASSWORD differs from the stored admin password; keeping the stored password because it was probably changed in the UI. …. Change ADMIN_PASSWORD once more to force it.
    • Since v1.13.1, a changed ADMIN_USERNAME that another account already signs in with, or has as its email address, is not applied and nothing changes (log: ADMIN_USERNAME "…" is not applied: another account already signs in with it or has it as its email address. …). Sign in with the previous username, or give that account a different username or email address on the Users page first.
    • This also applies when you change only ADMIN_PASSWORD after the primary admin's username or email was changed on the Users page: applying the environment credentials resets them to ADMIN_USERNAME, so if another account holds that name (or <ADMIN_USERNAME>@localhost) the password is not reset either and the same error is logged. Choose an ADMIN_USERNAME no other account uses, or have another administrator give that account a different username or email address first.

    See Environment Variables Reference#how-admin_username-and-admin_password-are-applied.

  4. Use the sign-in username: the login page signs in by username (ignoring case), not by email. Since v1.13.1 the username is the account's email address only when that address is a valid username (3–255 characters from A-Z a-z 0-9 _ . @ -, so not one containing +) that no other account uses; otherwise an administrator sets it. The user's Profile page shows it as Sign-in username. If it shows none, see No Sign-in Username on the Profile Page.

  5. Check for rate limiting (see Rate Limited):

    • Dashboard login: wait 10 seconds
    • Forward auth portal: wait LOGIN_BLOCK_MS (15 minutes by default)
    • Or restart: docker compose restart web

Can't Set a Password on an OAuth-Only Account

Error: "Please sign in again before setting a password."

Cause: Since v1.13.1, an account without a password can add one only within 10 minutes of signing in.

Solution: Sign out, sign in again with OAuth, and set the password right away on the Profile page.

No Sign-in Username on the Profile Page

Symptom: The Profile page shows no Sign-in username and says "Your account has no sign-in username the login page can use, so you cannot sign in there with a password. An administrator has to set a sign-in username for your account. This page then shows it." (on an account without a password: "…, then you can set your password here."). The Users page shows no sign-in username for the account.

Cause: Since v1.13.1, CPM no longer makes a username up from an email address. An account gets its own email address as username only when that address is 3–255 characters from A-Z a-z 0-9 _ . @ - (an address with + is not) and no other account signs in with it or has it as its email address. Older releases could also store a username the login page cannot use, such as an email with + or a mixed-case username.

Solution: An administrator sets a username:

  1. Open Users, click the edit icon on the account, enter a Username and click Save, or send PUT /api/v1/users/:id with {"username": "alice"} (see Feature Guide REST API#users).
  2. Use 3–255 characters of lowercase letters, digits and _ . @ - that are not another account's username, email address or forward-auth portal name; otherwise the save is refused with the reason (e.g. "Another account already signs in with this name or has it as its email address").
  3. Tell the user. They sign in at /login with that username and their password; an account without a password sets one on the Profile page first.

If the Profile page instead says "Your password cannot be used on the sign-in page yet. Change it once here to enable password sign-in.", either the account's own email address qualifies as a username or its password was set on an older release. Changing the password once (signed in another way, such as with OAuth) enables password sign-in, and in the first case makes the email address the username. In the first case an administrator can also set a username as above.

See Feature Guide User Management#sign-in-usernames.

Sign-in Username Warnings on Startup

Symptom: Since v1.13.1, the web container log shows warnings like these on every start:

Sign-in username "alice-cpm@example.com" of user 5 is an email address other than the account's own and can be somebody else's; check it on the Users page
Sign-in username "bob" of user 7 is also another account's username, email address or forward-auth portal name; give one of them a different username on the Users page

Cause: CPM checks the stored sign-in usernames on every start and reports, without changing anything:

  • An email address other than the account's own. Usually a username the withdrawn v1.13.0 release generated from the email address (alice+cpm@example.com → alice-cpm@example.com, or alice-2@example.com after a collision), or the account's address before its email changed. Somebody else can own that address.
  • A name that also reaches another account: another account's username, email address or forward-auth portal name (the <name> of an email <name>@localhost), ignoring case.

v1.13.0 logged the usernames it generated on startup as Gave user <id> the sign-in username <username>, but not those it generated on account creation or password and profile changes; this check covers both.

Solution:

  1. Find the account: GET /api/v1/users/<id> returns its email address, which the Users page search finds.
  2. Click the edit icon and set a Username the user should sign in with, such as one that is not an email address (alice) or the account's own email address when it qualifies. For a shared name, give one of the accounts a different username, or change the email address that collides.
  3. Tell the user their new username.

The warning stops once the username no longer matches. See Feature Guide User Management#upgrading-from-v1130.

Rate Limited

Symptom: "Too many requests. Please try again later." on the dashboard login, "Too many login attempts. Please try again later." on the forward auth portal, or "Too many attempts. Please try again later." when changing a password

Solutions:

Option 1: Wait. Dashboard sign-in allows 3 requests per 10 seconds per client (Better Auth's built-in limit, which only AUTH_RATE_LIMIT_ENABLED=false turns off). Portal logins and password changes are blocked for LOGIN_BLOCK_MS (15 minutes by default).

Option 2: Restart container (clears in-memory rate limit):

docker compose restart web

Option 3: Adjust rate limiting:

The forward auth portal login, password changes and account linking use the LOGIN_* variables. AUTH_RATE_LIMIT_MAX/AUTH_RATE_LIMIT_WINDOW only affect Better Auth's /api/auth endpoints other than sign-in and sign-up. The stock docker-compose.yml passes none of them through, so add them to the web service's environment: and run docker compose up -d:

services:
  web:
    environment:
      LOGIN_MAX_ATTEMPTS: "10"      # More attempts allowed
      LOGIN_BLOCK_MS: "300000"      # Shorter block (5 min)

The portal also limits failures per account from all clients combined (65 per hour with the defaults), so an account can be blocked even when your own client is not. See Environment Variables Reference#rate-limiting.

Certificate Issues

Certificate Not Obtained

Symptom: HTTPS not working, shows Caddy default page

Solutions:

  1. Check Caddy logs:

    docker compose logs caddy | grep -i acme
docker compose logs caddy | grep -i certificate

  2. Verify DNS resolves:

    dig +short your-domain.com
# Should return your server's public IP

  3. Check ports accessible:

    # From external machine
nc -zv your-server-ip 80
nc -zv your-server-ip 443

  4. Verify ACME email set:

    • Settings → General → ACME Email

  5. Check Let's Encrypt rate limits:

    • Visit crt.sh
    • Search for your domain
    • If 50+ certs/week, you're rate limited

Certificate Validation Failed

Error: Failed to obtain certificate

Common causes:

  1. DNS not propagated:

    • Wait up to 48 hours for DNS propagation
    • Verify: nslookup your-domain.com

  2. Firewall blocking:

    • Check server firewall: sudo ufw status
    • Check cloud provider security groups

  3. Domain doesn't point to server:

    • Verify IP: dig +short your-domain.com
    • Should match: curl ifconfig.me

  4. Cloudflare proxy enabled incorrectly:

    • Disable Cloudflare proxy (orange cloud → gray cloud)
    • Or configure DNS-01 challenge

Wildcard Certificate Fails

Symptom: Wildcard domain *.example.com doesn't get certificate

Solution:

Wildcard certificates require DNS-01 challenge:

  1. Configure a DNS provider:

  2. Verify configuration:

    • Settings → DNS Providers → provider configured and set as default

  3. Check logs:

    docker compose logs caddy | grep -i dns

OAuth Issues

OAuth Redirect URI Mismatch

Error: redirect_uri_mismatch

Solutions:

  1. Verify BASE_URL:

    # In .env
BASE_URL="https://proxy.example.com"  # No trailing slash

  2. Configure redirect URI in OAuth provider:

    {BASE_URL}/api/auth/callback/{provider-id}

    The exact callback URL is shown in Settings → OAuth Providers.

  3. Recreate the web container after changes (docker compose restart does not re-read .env):

    docker compose up -d

OIDC Discovery Failed

Error: Failed to discover OIDC endpoints

Solutions:

  1. Test discovery endpoint:

    curl https://auth.example.com/application/o/app/.well-known/openid-configuration

  2. Check trailing slash in OAUTH_ISSUER:

    # Some providers require trailing slash
OAUTH_ISSUER="https://auth.example.com/application/o/app/"

  3. Use manual endpoints if discovery fails:

    OAUTH_AUTHORIZATION_URL="https://..."
OAUTH_TOKEN_URL="https://..."
OAUTH_USERINFO_URL="https://..."

OAuth Login Button Not Showing

Symptom: OAuth login option doesn't appear on login page

Solutions:

  1. Verify OAUTH_ENABLED:

    OAUTH_ENABLED=true  # Must be lowercase "true"

  2. Check all required variables:

    OAUTH_CLIENT_ID="..."
OAUTH_CLIENT_SECRET="..."
OAUTH_ISSUER="..."

  3. Recreate the web container (docker compose restart does not re-read .env):

    docker compose up -d

  4. Check logs:

    docker compose logs web | grep -i oauth

Permission Issues

Docker Volumes vs Bind Mounts

Important: As of v1.0, Caddy Proxy Manager uses Docker named volumes by default instead of bind mounts.

If you're experiencing permission issues:

  1. Using default Docker volumes (recommended): Permission issues should be rare with volumes as Docker handles permissions automatically.

  2. Switched to bind mounts (./data, ./caddy-data, etc.): You need to ensure proper file ownership matches the container's PUID/PGID.

Quick fix for bind mount permission issues:

# Stop containers
docker compose down

# Fix ownership (use default container UIDs)
sudo chown -R 10001:10001 data/
sudo chown -R 10000:10000 caddy-data/ caddy-config/ caddy-logs/

# Or rebuild containers to match your user
PUID=$(id -u) PGID=$(id -g) docker compose up --build -d
sudo chown -R $(id -u):$(id -g) data/ caddy-data/ caddy-config/ caddy-logs/

For detailed information including:

  • How to switch between volumes and bind mounts
  • Step-by-step permission troubleshooting
  • Migration guides
  • Advanced ACL configurations

See: Rootless Docker Operation#docker-volumes-vs-bind-mounts

Permission Denied on /app/data

Error: EACCES: permission denied, open '/app/data/caddy-proxy-manager.db'

Solutions:

Option 1: Fix ownership:

sudo chown -R 10001:10001 data/
docker compose restart web

Option 2: Use rootless with your user:

PUID=$(id -u) PGID=$(id -g) docker compose up --build -d

See Rootless Docker Operation.

Can't Edit Files on Host

Symptom: Permission denied when editing files in data/ directory

Solutions:

Option 1: Change PUID to your user (recommended for dev):

PUID=$(id -u) PGID=$(id -g) docker compose up --build -d

Option 2: Use sudo:

sudo nano data/caddy-proxy-manager.db

Option 3: Add yourself to group:

sudo groupadd -g 10001 caddypm
sudo usermod -aG caddypm $(whoami)
# Log out and back in
sudo chgrp -R caddypm data/
sudo chmod -R g+rw data/

Backup Job Can No Longer Read the Database

Symptom: After upgrading, a host-side backup or monitoring job gets Permission denied on caddy-proxy-manager.db (or its -wal/-shm files)

Cause: Since v1.13.1, the web container removes the world permission bits from the database files whenever it opens the database, because they hold password hashes, session tokens and encrypted secrets. Owner and group bits are left unchanged.

Solution: Run the job as the files' owner, or give it the files' group (as in Option 3 above, with read access: sudo chmod g+r data/caddy-proxy-manager.db*). Adding world read access back does not last: it is removed again on the next start. If the container cannot change the mode, it logs Could not restrict permissions on <file>: … and keeps running.

Networking Issues

Can't Connect to Caddy API

Error: Failed to connect to Caddy API at http://caddy:2019

Solutions:

  1. Check Caddy container running:

    docker ps | grep caddy

  2. Verify network connectivity:

    docker compose exec web wget -O- http://caddy:2019/config/

  3. Check CADDY_API_URL:

    # In .env or docker-compose.yml
CADDY_API_URL="http://caddy:2019"

  4. Verify same Docker network:

    docker network inspect caddy-network

502 Bad Gateway

Symptom: Caddy returns 502 error for proxy host

Solutions:

  1. Check upstream service running:

    curl http://upstream-host:port

  2. For Docker containers, verify network:

    docker network inspect caddy-network

  3. Check container name correct:

    docker ps --format "{{.Names}}"

  4. Use host.docker.internal for host services:

    # macOS/Windows
http://host.docker.internal:8080

# Linux
http://172.17.0.1:8080

  5. Check logs:

    docker compose logs caddy | grep -i upstream

WebSocket Connection Failed

Symptom: WebSocket connections fail through proxy

Solutions:

  1. Verify backend supports WebSocket

  2. Add WebSocket headers:

    Connection: {http.request.header.Connection}
Upgrade: {http.request.header.Upgrade}

  3. Check for timeout issues:

    • WebSocket connections are long-lived
    • May need timeout adjustments (future feature)

Database Issues

Database Locked

Error: database is locked

Solutions:

  1. Close all connections:

    docker compose restart web

  2. Check for multiple instances:

    docker ps | grep caddy-proxy-manager
# Should see only 1 web container

  3. Remove lock file:

    rm -f data/caddy-proxy-manager.db-wal
rm -f data/caddy-proxy-manager.db-shm
docker compose restart web

Corrupted Database

Symptom: Database errors, data missing, crashes

Solutions:

Option 1: Restore from backup:

docker compose down
cp backup/caddy-proxy-manager.db data/
docker compose up -d

Option 2: Export and recreate:

# Export (if possible)
sqlite3 data/caddy-proxy-manager.db .dump > backup.sql

# Recreate
rm data/caddy-proxy-manager.db
docker compose up -d

# Import (manual process)

Option 3: Start fresh:

docker compose down
rm -rf data/
docker compose up -d

Performance Issues

Slow Response Times

Solutions:

  1. Check system resources:

    docker stats

  2. Increase container resources:

    # docker-compose.yml
services:
  web:
    deploy:
      resources:
        limits:
          memory: 1G
          cpus: '1.0'

  3. Check upstream performance:

    • Test upstream directly
    • May be backend slowness

  4. Enable metrics for monitoring:

    • Settings → Metrics → Enable

High Memory Usage

Solutions:

  1. Check logs size:

    docker compose logs --tail=100 web

  2. Limit log size:

    # docker-compose.yml
services:
  web:
    logging:
      options:
        max-size: "10m"
        max-file: "3"

  3. Restart containers periodically:

    docker compose restart

Secret Decryption Issues

HKDF Decryption Failed / Stored Token Can't Be Decrypted

Error:

[secret] Failed to decrypt stored secret for DNS provider "cloudflare" credential "api_token": decryption failed with the current key and SESSION_SECRET_PREVIOUS, and the legacy key grace period has expired. ...

or, at startup:

[secret] setting "dns_provider" cannot be decrypted with SESSION_SECRET or SESSION_SECRET_PREVIOUS; re-enter it in the UI or set SESSION_SECRET_PREVIOUS to the secret it was stored with.
N stored secret(s) listed above could not be decrypted with SESSION_SECRET or SESSION_SECRET_PREVIOUS; ...

Releases up to v1.12.0 print [secret] Failed to decrypt stored secret …: HKDF decryption failed and the legacy key grace period has expired. … instead.

What's happening:

Sensitive values are stored in the database encrypted at rest (enc:v1: prefix) using AES-256-GCM with a key derived from SESSION_SECRET. If SESSION_SECRET changes — for example after your environment variables were reset, or you generated a new secret — previously stored values can no longer be decrypted unless the old secret is in SESSION_SECRET_PREVIOUS.

Since v1.13.1, the web container tries every stored secret at startup: values that SESSION_SECRET_PREVIOUS (or a public placeholder secret) decrypts are re-encrypted with the current SESSION_SECRET, values that no key decrypts are left as stored and listed in the log, and OAuth sign-in tokens that no key decrypts are cleared (Cleared N stored OAuth sign-in token(s) that no key decrypts; …), since CPM does not use them.

Data stored encrypted with SESSION_SECRET:

  • DNS provider credentials, e.g. the Cloudflare API token (Settings → DNS Providers) — used for DNS-01 challenges. Credentials saved in plaintext by older versions (e.g. through PUT /api/v1/settings/dns-provider) are encrypted at startup, logged as Encrypted N DNS provider credential(s) that were stored in plaintext
  • OAuth provider client ID / secret (Settings → OAuth Providers)
  • OAuth sign-in tokens of linked accounts
  • Instance API tokens (Instances)
  • Instance sync master token (Settings → Instance Sync)
  • Imported certificate private keys (Certificates)
  • CA private keys (Certificates → CA / mTLS), since v1.13.1

Solutions:

Option 1: Set SESSION_SECRET_PREVIOUS (recommended if you still have the old secret)

Keep the new SESSION_SECRET, put the old value (e.g. from a backup of your .env file or compose config) in SESSION_SECRET_PREVIOUS, and run docker compose up -d. Startup re-encrypts everything with the new secret (Re-encrypted N stored secret(s) with the current SESSION_SECRET); remove SESSION_SECRET_PREVIOUS after that start. See Security Configuration#secret-rotation.

Option 2: Re-enter the affected token/secret

Open the relevant settings page, re-enter the token/password field, and save. The value is re-encrypted with the current key. For example, if the error mentions a DNS provider credential:

  1. Go to Settings → DNS Providers
  2. Edit the provider named in the error (e.g. cloudflare)
  3. Re-enter the API token (reset it with your DNS provider if needed)
  4. Save

Option 3: Restore the previous SESSION_SECRET

Restoring the old value as SESSION_SECRET also makes all stored values decryptable again. To move to a new secret later, use Option 1.

Option 4: LEGACY_KEY_CUTOFF_DATE (legacy-format secrets only)

Secrets stored by older versions used a legacy key derivation. They are supported only until a cutoff date (default 2026-06-01). If your SESSION_SECRET is unchanged but you see the grace-period error above, you can temporarily extend or disable the cutoff:

# In .env — ISO 8601 date, or "never" to disable entirely
LEGACY_KEY_CUTOFF_DATE=never

Note: This does not help if SESSION_SECRET itself changed. The legacy key is also derived from SESSION_SECRET, so a changed secret breaks both keys. Use Option 1, 2 or 3 instead.

Find which stored values are encrypted:

sqlite3 data/caddy-proxy-manager.db "SELECT key FROM settings WHERE value LIKE '%enc:v1:%';"

DNS provider credentials, the instance sync master token, and other settings live in the settings table; instance API tokens are in the instances table (apiToken column), certificate private keys in the certificates table and CA private keys in the ca_certificates table (privateKeyPem column in both).

See Environment Variables Reference for SESSION_SECRET, SESSION_SECRET_PREVIOUS and LEGACY_KEY_CUTOFF_DATE details.

Issuing a Client Certificate Fails: CA Private Key

Error: "The CA private key cannot be decrypted with the current SESSION_SECRET. Restore the previous secret via SESSION_SECRET_PREVIOUS or create a new CA." in the Issue Client Certificate dialog

Cause: Since v1.13.1, CA private keys are encrypted with SESSION_SECRET, and this CA's key was stored under a secret that is no longer configured.

Solution: Set SESSION_SECRET_PREVIOUS to the secret the key was stored with and run docker compose up -d (the key is then re-encrypted with the current secret), or create a new CA. Client certificates already issued keep working, because validating them only needs the CA certificate.

On an instance sync slave, "This CA has no stored private key — cannot issue client certificates" is expected: CA private keys stay on the master. Issue client certificates on the master.

Forward Auth Issues

Portal Says the Port Is Not Allowed

Error: The portal shows "This site is served on port 8443, which is not allowed for forward authentication. Ask the administrator to add it to FORWARD_AUTH_ALLOWED_PORTS." and the web log shows:

[forward-auth] Rejected app.example.com:8443 because port 8443 is not listed in FORWARD_AUTH_ALLOWED_PORTS. If forward-auth protected sites are served on port 8443, add it to FORWARD_AUTH_ALLOWED_PORTS (comma-separated) and recreate the web container (docker compose up -d).

Cause: Since v1.13.1, logins, redirects and sessions for protected sites on a port other than 80/443 are refused unless the port is listed in FORWARD_AUTH_ALLOWED_PORTS, because Caddy matches proxy hosts by hostname only. After upgrading, existing forward-auth sessions on such a port stop validating too.

Solution:

# In .env (docker-compose.yml passes it to the web container)
FORWARD_AUTH_ALLOWED_PORTS=8443

Then run docker compose up -d. See Environment Variables Reference.

Many Users Are Rate Limited on the Portal at Once

Symptom: Portal logins fail with "Too many login attempts. Please try again later." for many users together, typically behind a CDN or another proxy in front of Caddy

Cause: Since v1.13.1, the portal counts failures per client using the rightmost X-Forwarded-For entry and no longer trusts a client-sent X-Real-IP. Behind a CDN, that entry is the CDN edge, so users behind the same edge share one limit.

Solution:

  • Behind a CDN, set TRUSTED_CLIENT_IP_HEADER to the CDN's client IP header (e.g. cf-connecting-ip) and run docker compose up -d, but only if the origin accepts connections from the CDN alone; otherwise clients can forge the header.
  • Leave it unset when Caddy is the outermost proxy.
  • A single account can also be blocked by failures from many clients combined (65 per hour with the defaults); wait LOGIN_BLOCK_MS (15 minutes by default) or restart the web container.

See Feature Guide Forward Auth#portal-login-rate-limits.

Instance Sync Issues

Sync errors appear on the master under Settings → Instance Sync for each slave; slaves configured with INSTANCE_SLAVES report them in the master's log. See Feature Guide Instance Sync for sealed sync and sync key pinning (since v1.13.1).

Slave Sync Key Changed

Error: "Slave sync key changed; verify the slave, then pin its new key or reset its key pin", with this in the master's log:

Instance sync: slave "<name>" presented sync key <new>, but <pinned> is pinned and the slave sent no valid rotation proof; not syncing…

Cause: The master pinned the slave's sync key, which the slave derives from its SESSION_SECRET, and the slave now presents another one: its SESSION_SECRET changed (reinstall, or a rotation without SESSION_SECRET_PREVIOUS or with it removed too early), or something else answers at the slave's URL.

Solution:

  1. Rotated secret: put the old value in the slave's SESSION_SECRET_PREVIOUS, run docker compose up -d on the slave, then click Sync now on the master. The master logs Instance sync: slave "<name>" proved its new sync key <new> with the pinned key <old>; pinned the new key. Only then remove SESSION_SECRET_PREVIOUS from the slave.
  2. Otherwise: compare the key id in the master's log with the one the slave shows on its own Settings → Instance Sync page. If they match, pin the slave's new key on the master (Key pin). A key the slave does not show means something else answered at its address.

For a slave pinned with syncPublicKey or syncKeyId in INSTANCE_SLAVES, the error is "Slave sync key does not match the key configured in INSTANCE_SLAVES": update the entry with the slave's new key.

Sync Timed Out

Error: "Sync timed out"

Cause: Since v1.13.1, the master stops a sync request after INSTANCE_SYNC_TIMEOUT_MS (default 60 seconds), which covers the upload and the slave's apply including its Caddy reload. The slave may still finish applying the configuration.

Solution: Check the slave's logs and reachability. For large configurations or slow slaves, raise INSTANCE_SYNC_TIMEOUT_MS on the master (at most 300000) and run docker compose up -d.

Sync Key Request Failed

Error: "Sync key request failed with HTTP ", "Slave returned an invalid sync key" or "Slave did not acknowledge the sync (unexpected response)"

Cause: Before every sync the master fetches the slave's sync key with GET /api/instances/sync (since v1.13.1), and it does not follow redirects.

Error Usual cause
HTTP 401 Wrong sync token
HTTP 404 Wrong base URL, or a proxy or virtual host answering instead of CPM
HTTP 302, "Slave returned an invalid sync key", "Slave did not acknowledge the sync" A login page, redirect or other non-CPM response in front of the slave
HTTP 405 The slave has a pinned key but answers like v1.12.0 or earlier: it was downgraded, or a proxy refuses GET

Solution: Proxies in front of a slave must pass both GET and POST on /api/instances/sync, with the Authorization header and the query string, and must not cache the GET reply. After deliberately downgrading a slave to v1.12.0 or earlier, reset its key pin on the master.

Slave Fails to Apply Synced Config During an Upgrade

Symptom: The slave's Settings → Instance Sync page shows "Failed to apply synchronized configuration" while master and slaves run different releases

Cause: A master on v1.12.0 or earlier sends DNS provider credentials encrypted with its own SESSION_SECRET, and a slave on v1.12.0 or earlier expects them that way from any master.

Solution: Upgrade slaves before, or together with, the master. Until the master is upgraded, give each slave the master's secret as SESSION_SECRET or in SESSION_SECRET_PREVIOUS; until a slave is upgraded, it needs the master's current secret as its own SESSION_SECRET.

WAF Issues

Custom Directive Lines Are Not Sent to Caddy

Symptom: The web log shows

[waf] global WAF settings: 1 custom directive line(s) are not sent to Caddy and have no effect:
  "<line>" → <reason>

(or the same for proxy host "<name>" (<domains>)), or saving a host or the WAF settings is rejected with waf.custom_directives contains N line(s) that will be dropped and never sent to Caddy: …

Cause: Directives outside the allowed set (such as Include) were never sent to Caddy. Since v1.13.1, rules that could read files, run programs or switch the WAF off (for example @pmFromFile, @inspectFile, @validateSchema, setenv, ctl:ruleEngine), lines Coraza cannot parse, directives continued over several lines with a trailing \, and rules reusing the id: of an earlier rule are left out as well. When one rule of a chain is dropped, the whole chain is dropped. Rules stored by older versions are kept but only reported in the log (with the source they come from); a save is rejected only for lines it newly drops.

Solution: Rewrite or remove each listed line according to its reason, write each directive on one line, and give every rule a unique id:. See Feature Guide WAF for the full list of rules.

Getting Help

Gathering Debug Information

When opening an issue, include:

  1. Error logs:

    docker compose logs web > web-logs.txt
docker compose logs caddy > caddy-logs.txt

  2. Environment info:

    docker --version
docker compose version
uname -a

  3. Configuration (sanitized):

    • Remove secrets (SESSION_SECRET, passwords, API tokens)
    • Include docker-compose.yml
    • Include relevant .env variables

  4. Steps to reproduce

  5. Expected vs actual behavior

Useful Commands

View logs:

# Follow logs
docker compose logs -f web
docker compose logs -f caddy

# Last 100 lines
docker compose logs --tail=100 web

# Search logs
docker compose logs web | grep -i error

Restart services:

# Restart all
docker compose restart

# Restart specific service
docker compose restart web
docker compose restart caddy

Container shell:

# Web container
docker compose exec web sh

# Caddy container
docker compose exec caddy sh

Check configuration:

# Verify docker-compose config
docker compose config

# Check environment variables
docker compose exec web env

Related Documentation

Still need help? Open an issue with debug information.

Clone this wiki locally