Gate Cloud Run traffic on an HTTP readiness startup probe#3765
Merged
Conversation
Cloud Run's default TCP startup probe passes the moment gunicorn's master binds the port, ~161s before a worker finishes importing (the import runs post-fork; --preload is deliberately unset). Cloud Run therefore marked booting instances 'started' and routed live traffic onto them, where requests queued until the 300s timeout — the 'no available instance' 500s and 504s seen on every scale-out burst. Probe /readiness-check over HTTP instead so Cloud Run withholds traffic until the app can serve. The probe window (failureThreshold x periodSeconds) is 240s, the platform maximum; measured import is ~161s, leaving ~79s headroom. Past the window Cloud Run kills the container, so a slow import now fails the candidate deploy in CI on --no-traffic rather than degrading production. Also corrects the warm-capacity runbook, which documented the revision-level --min-instances flag for a service-level setting: --min is service-level warm capacity, --min-instances is the per-tag cost bomb. Records the new value of 2 and its cost. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Measured 48 real boots over 7 days (Starting gunicorn -> last Application startup complete): p50 201s, p90 371s, p95 417s, max 503s, with 11/48 (23%) over 240s. Cloud Run hard-caps startup-probe time at 240s and shuts the container down past it, so an HTTP readiness probe would kill ~23% of boots — failing that share of deploys and thrashing scale-out instances, worst under the CPU contention that triggers scale-out in the first place. Stage 2's '~161s import' measures the import step on an uncontended instance, not boot-to-ready; sizing the probe from it was the error. Documents the distribution and the prerequisite (cut p95 boot-to-ready well under 240s via lazy country loading or cached tax-benefit systems) so this is not re-attempted from config alone. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cloud Run's default TCP probe passes when gunicorn's master binds the port, long before a worker finishes importing, so live traffic was routed onto instances that could not answer and queued to the 300s timeout. The platform caps failureThreshold x periodSeconds at 240s, which alone covers only 77% of our measured boots. initialDelaySeconds is additive (no probe runs during it, so no failures accumulate) and separately capped at 240s, giving a real ceiling of 480s. Using 120 + 24x10 = 360s covers ~90% of boots while keeping initialDelaySeconds below the p50 boot (201s), since it also delays availability and would otherwise slow every scale-out. ~10% of boots still exceed this and will be killed and retried: a deliberate trade, since the TCP probe served real users 500s and 504s from instances that were never ready. Cutting boot time removes the residual risk. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
failureThreshold x periodSeconds is already at its 240s platform ceiling, so initialDelaySeconds is the only lever left. Raising it 120 -> 180 halves the share of boots killed by the window (12.5% -> 6.2%); the instances it newly delays are mostly booting at 155-179s, so they wait ~25s. 240 was rejected because it would hold 77% of instances to a full 240s, slowing every scale-out, for four further points. The costs are asymmetric: a delayed instance loses tens of seconds, a killed one loses its whole boot plus a retry (400s+) and fails the deploy if it happens in CI. Also replaces the stale docs section that declared the HTTP probe unviable — that conclusion assumed the 240s cap was the whole window, before initialDelaySeconds was found to be additive. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Records that services replace needs the Cloud Resource Manager API enabled, and that service-level warm capacity was first set on 2026-07-08 before being raised on 2026-07-22 — both via the YAML-replace fallback. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The sizing analysis belongs in cloud-run-operations.md, which already carries it; the env file just needs the gist and a pointer. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
anth-volk
marked this pull request as ready for review
July 22, 2026 17:21
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #3765 +/- ##
=======================================
Coverage 80.03% 80.03%
=======================================
Files 70 70
Lines 4342 4342
Branches 809 809
=======================================
Hits 3475 3475
Misses 648 648
Partials 219 219 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3764
Summary
--startup-probe httpGet.path=/readiness-checkon every Cloud Run deploy, replacing the default TCP probe that passes the instant gunicorn's master binds the port — minutes before a worker finishes importing. Cloud Run now withholds traffic until the app can actually serve.initialDelaySeconds 180 + failureThreshold 24 × periodSeconds 10 = 420s. The threshold/period half is already at its 240s platform ceiling;initialDelaySeconds(also capped at 240s) is additive and the only remaining lever.tcpSocket), that both platform caps hold independently, that the total window covers the measured p90 boot, and thatinitialDelaySecondsstays below p50 so it does not needlessly delay scale-out.Sizing evidence
Boot-to-ready across 48 boots over 7 days: p50 201s, p90 371s, p95 417s, max 503s.
240 was rejected: it holds 77% of instances to a full 240s, slowing every scale-out, for four further points.
Qualification
Deployed to a
--no-trafficrevision on 2026-07-22. Cloud Run stored the config exactly as specified (confirminginitialDelaySecondsis accepted alongside a full 240s threshold window) and the revision reachedReady/ContainerHealthyon a 181.6s boot. Production traffic was never touched, and the service template has since been reverted to the default TCP probe so nothing goes live until this merges.Caveat: that qualification boot was 181.6s, i.e. under 240s — it validated the config and the readiness gating, but did not exercise a boot longer than 240s.
Accepted risk
~6% of boots exceed 420s and will be killed and retried, which can fail a deploy (recoverable on retry). The alternative is the status quo: real users served 500s and 504s from instances that were never ready.
Also included
--min-instancesflag for a service-level setting — the revision-level minimum keeps a warm 16Gi instance per accumulated tag. Records the current value of 2 and thegcloud run services replaceprocedure for older gcloud.Why draft
Merging triggers the deploy chain (version-bump commit → staging Cloud Run deploy + integration tests → production candidate → promote to 100%), so this should land when someone can watch it. Staging acts as a real canary — it deploys and is tested before production.
🤖 Generated with Claude Code