Correct the deploy runbook against a real first deploy - #168
Merged
Conversation
Three things the runbook got wrong, all found by following it end to end against Render and Neon rather than by reading it. The keep-warm ping now targets /health/live instead of the root path. Root 404s, and cron-job.org counts any non-2xx as a failed execution: it alerts every five minutes and disables jobs that keep failing. A keep-warm ping that switches itself off is the worst possible failure here, because the app silently returns to 40-60s cold starts with nothing to say so. /health/live answers 200 and is registered with Predicate = _ => false, so it runs no checks at all and still never wakes Neon's compute - which was the whole reason the doc steered away from /health. It already conceded this in a parenthetical and picked root anyway as "one less thing to get wrong"; it was the more things to get wrong. The HTTP-to-HTTPS check expected a 307. On Render the redirect is a 301 issued by Cloudflare at the edge and never reaches the app - the response carries neither x-render-origin-server nor rndr-id. UseHttpsRedirection's 307 only appears off-Render. Both docs now say what the check is actually for: one hop, no loop. GO_LIVE_CHECKLIST never mentioned the Render deploy hook at all. DEPLOYMENT.md covers it, but anyone working the tick-box version ends up with autoDeploy off and no RENDER_DEPLOY_HOOK_URL - so every push to main migrates the database and then fails on an empty URL, leaving the schema ahead of the code with only a red check mark to show for it. Added, with the gh command and why it is a secret. Also added to the checklist: a redeploy box for the case where the service booted before the migration finished (SubscriptionCatalogSyncService runs only at boot and never retries, so the symptom is an empty catalog behind a healthy /health), and a /subscriptions/resolve check that proves migration-before-boot in one call - tier, providerName and a non-null categoryId each pin a different half of it. The second-registration path is noted as a free SMTP check, since it emails the existing owner while returning an identical 202.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three corrections, all found by working through the runbook against real Render and Neon rather than by reading it.
1. Keep-warm ping →
/health/live, not the root pathRoot
404s. cron-job.org counts any non-2xx as a failed execution: it alerts every five minutes and disables jobs that keep failing. A keep-warm ping that switches itself off is the worst failure mode available here — the app silently returns to 40–60s cold starts and nothing says so./health/liveis registered withPredicate = _ => false, so it runs no checks at all and still never wakes Neon's compute — which was the entire reason the doc steered away from/health. With a 200 expected, failure notifications become a real uptime alert instead of noise worth muting.The doc already conceded this in a parenthetical and chose root anyway as "one less thing to get wrong". It was the more things to get wrong.
2. The HTTP→HTTPS check expects the wrong status
Documented as
307. Actual, against the deployed service:No
x-render-origin-server: Kestrel, norndr-id— headers present on every real response from the container. The redirect is issued by Cloudflare at Render's edge and never reaches the app, soUseHttpsRedirection's307only appears off-Render.Both docs now state what the check is actually for: one hop, no loop. The loop is the failure it catches, and that still means
UseForwardedHeadersis missing or mis-ordered.3.
GO_LIVE_CHECKLIST.mdnever mentioned the Render deploy hookDEPLOYMENT.md§4 covers it. The tick-box version does not — so anyone working the checklist finishes withautoDeploy: falseand noRENDER_DEPLOY_HOOK_URL, and every push tomainmigrates the database and then fails on an empty URL. Schema ahead of code, one red check mark to show for it.This is not hypothetical — it is what happened on this deploy, and it was only caught by inspecting
gh secret list. Added as a step-5 box with thegh secret setcommand and why the URL is a secret rather than a plain setting.Also added to the checklist
SubscriptionCatalogSyncServiceruns only at boot and never retries, so the symptom is an empty provider catalog behind a perfectly healthy/health— silent, and confusing to diagnose later./subscriptions/resolvecheck, which proves migration-before-boot in a single call. Each field pins a different half:Faster and more specific than "add a subscription, then check the burn-rate breakdown", which is kept as the end-to-end confirmation.
Code change
One comment in
Program.cs, at the/health/liveregistration, so the reason the cron points there lives next to the endpoint rather than only in the runbook. Build verified: 0 warnings, 0 errors. No behaviour change.