Serialize local sqlite initialization under a file lock#3747
Merged
Conversation
With WEB_CONCURRENCY=2, both gunicorn workers import the app concurrently on a fresh instance; both could pass the sqlite exists-check and race initialize() — the losing worker died at boot on 'UNIQUE constraint failed: policy.id' (observed as 503 bursts during the first real Cloud Run scale-out, Stage 5 LB validation), and a worker could also observe a created-but-unseeded database and skip initialization entirely, serving with empty reference tables. An exclusive fcntl lock around the exists-check + initialize means exactly one process initializes while the others wait, then see the complete file and skip. Empirical verification (6 concurrent fresh boots x 3 rounds): unfixed code failed every round (boot crashes and row counts of 0/2/5 across workers); fixed code passes with all workers seeing exactly 5 seed rows. Fixes #3746 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #3747 +/- ##
==========================================
+ Coverage 79.89% 79.92% +0.02%
==========================================
Files 70 70
Lines 4332 4338 +6
Branches 807 808 +1
==========================================
+ Hits 3461 3467 +6
Misses 652 652
Partials 219 219 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
anth-volk
added a commit
that referenced
this pull request
Jul 9, 2026
Both runs: 2026-07-08 (all gates; caught the sqlite cold-boot race, fixed in #3747) and the 2026-07-09 zero-5xx re-run (race confirmed fixed across a 1->4 scale-out; three 504s dissected and recorded as the known cold-scale-out saturation artifact). Stage 5 closed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3746
Summary
Stage 5's 50/50 LB test triggered the Cloud Run service's first real scale-out, and cold instances served brief 503 bursts: with
WEB_CONCURRENCY=2, both gunicorn workers import the app concurrently on a fresh instance, both can pass the sqliteexists()check, and both runinitialize()against the same file — the loser dies at boot withsqlite3.IntegrityError: UNIQUE constraint failed: policy.id(revision logs, 2026-07-08 20:07). A worse silent variant also exists: a worker can observe the file created but not yet seeded, skip initialization, and serve with empty reference tables.Change
One block in
data.py: the exists-check +initialize()now runs under an exclusivefcntlfile lock — exactly one process initializes; the others block until it finishes, then see the complete database and skip. ~10 lines, local-sqlite path only; the remote (Cloud SQL) path is untouched.Verification
Empirical 6-way concurrent fresh-boot test (fork model, mirroring gunicorn):
Unit suite: 658 passed (one pre-existing local-env error in
ai_promptsreproduces identically on unfixed master; CI arbitrates).Context
This is the last known blocker class before ramp stages: any scale-out under real traffic would otherwise risk 503 bursts and silently mis-initialized workers. Stage 5's zero-5xx gate re-runs after this deploys.
🤖 Generated with Claude Code