Summary
Live incident found and fixed 2026-08-11 while deploying dashboard changes: the VPS's live /root/vps/traefik/dynamic.yml still had the repo's committed placeholder hostname (auth.honeypot.example, 35 occurrences) instead of the real domain, even though .github/workflows/deploy.yml's "Deploy Traefik dynamic config with the real domain substituted" step exists specifically to sed this in CI on every VPS deploy.
Impact while live
Every Traefik router keyed on Host(\auth.honeypot.example`)(keycloak, keycloak-embedded-frames, keycloak-admin-console, keycloak-account-console) matched **zero** real traffic.https://auth.xore.rocks/realms/apiary/.well-known/openid-configuration` returned a consistent 404 (verified 5x). Any fresh dashboard container start failed immediately — newOIDCAuth() requires OIDC discovery unconditionally at startup — while already-running replicas were unaffected (cached their own successful discovery from before whatever broke this). Confirmed no user-facing outage occurred only because hp-dashboard-b had been running continuously since before the break.
What's not yet known
Why the substitution didn't take effect live despite the CI step existing and the file content otherwise matching main exactly (confirmed via diff — only the domain differed, no other drift). Candidates worth checking: whether the DOMAIN secret was unset/empty during a recent run (the step has an explicit guard for this — if [ -z "$DOMAIN" ]; then exit 1, which should have failed loudly, not silently left placeholders), whether a run targeted home only and never reached the vps job's Traefik-specific step, or whether an out-of-band manual edit reverted the substituted file after a prior correct deploy.
What was done live
Backed up the live file (/root/vps-backups/pre-domain-fix-<timestamp>.tar.gz), regenerated it from the current main checkout with the same sed "s/honeypot\.example/xore.rocks/g" substitution CI uses, validated as YAML, and deployed it in-place (cat > /root/vps/traefik/dynamic.yml, matching the CI step's own documented "never rename/mv — Traefik's bind mount tracks the inode" caution). Verified OIDC discovery returns 200 consistently afterward, then confirmed deploy-dashboard-rolling.sh uses it successfully.
Ask
Root-cause why the automated substitution didn't hold on the live host (or when it was lost), and consider a startup-time or scheduled health check that would catch "OIDC discovery unreachable" before it's discovered mid-deploy by a container that won't start — the CI's own "Verify the live Traefik configuration survived" step already checks for the placeholder string post-deploy, but nothing appears to check it between deploys.
Summary
Live incident found and fixed 2026-08-11 while deploying dashboard changes: the VPS's live
/root/vps/traefik/dynamic.ymlstill had the repo's committed placeholder hostname (auth.honeypot.example, 35 occurrences) instead of the real domain, even though.github/workflows/deploy.yml's "Deploy Traefik dynamic config with the real domain substituted" step exists specifically tosedthis in CI on every VPS deploy.Impact while live
Every Traefik router keyed on
Host(\auth.honeypot.example`)(keycloak, keycloak-embedded-frames, keycloak-admin-console, keycloak-account-console) matched **zero** real traffic.https://auth.xore.rocks/realms/apiary/.well-known/openid-configuration` returned a consistent 404 (verified 5x). Any fresh dashboard container start failed immediately —newOIDCAuth()requires OIDC discovery unconditionally at startup — while already-running replicas were unaffected (cached their own successful discovery from before whatever broke this). Confirmed no user-facing outage occurred only becausehp-dashboard-bhad been running continuously since before the break.What's not yet known
Why the substitution didn't take effect live despite the CI step existing and the file content otherwise matching
mainexactly (confirmed via diff — only the domain differed, no other drift). Candidates worth checking: whether theDOMAINsecret was unset/empty during a recent run (the step has an explicit guard for this —if [ -z "$DOMAIN" ]; then exit 1, which should have failed loudly, not silently left placeholders), whether a run targetedhomeonly and never reached thevpsjob's Traefik-specific step, or whether an out-of-band manual edit reverted the substituted file after a prior correct deploy.What was done live
Backed up the live file (
/root/vps-backups/pre-domain-fix-<timestamp>.tar.gz), regenerated it from the currentmaincheckout with the samesed "s/honeypot\.example/xore.rocks/g"substitution CI uses, validated as YAML, and deployed it in-place (cat > /root/vps/traefik/dynamic.yml, matching the CI step's own documented "never rename/mv — Traefik's bind mount tracks the inode" caution). Verified OIDC discovery returns 200 consistently afterward, then confirmeddeploy-dashboard-rolling.shuses it successfully.Ask
Root-cause why the automated substitution didn't hold on the live host (or when it was lost), and consider a startup-time or scheduled health check that would catch "OIDC discovery unreachable" before it's discovered mid-deploy by a container that won't start — the CI's own "Verify the live Traefik configuration survived" step already checks for the placeholder string post-deploy, but nothing appears to check it between deploys.