Skip to content

Serialize local sqlite initialization under a file lock#3747

Merged
anth-volk merged 2 commits into
masterfrom
fix/local-db-init-race
Jul 8, 2026
Merged

Serialize local sqlite initialization under a file lock#3747
anth-volk merged 2 commits into
masterfrom
fix/local-db-init-race

Conversation

@anth-volk

Copy link
Copy Markdown
Collaborator

Fixes #3746

Summary

Stage 5's 50/50 LB test triggered the Cloud Run service's first real scale-out, and cold instances served brief 503 bursts: with WEB_CONCURRENCY=2, both gunicorn workers import the app concurrently on a fresh instance, both can pass the sqlite exists() check, and both run initialize() against the same file — the loser dies at boot with sqlite3.IntegrityError: UNIQUE constraint failed: policy.id (revision logs, 2026-07-08 20:07). A worse silent variant also exists: a worker can observe the file created but not yet seeded, skip initialization, and serve with empty reference tables.

Change

One block in data.py: the exists-check + initialize() now runs under an exclusive fcntl file lock — exactly one process initializes; the others block until it finishes, then see the complete database and skip. ~10 lines, local-sqlite path only; the remote (Cloud SQL) path is untouched.

Verification

Empirical 6-way concurrent fresh-boot test (fork model, mirroring gunicorn):

Unfixed (3 rounds) Fixed
All workers boot ❌ every round
Policy rows seen across workers {0, 2, 5} — including the silent empty-table variant {5} exactly

Unit suite: 658 passed (one pre-existing local-env error in ai_prompts reproduces identically on unfixed master; CI arbitrates).

Context

This is the last known blocker class before ramp stages: any scale-out under real traffic would otherwise risk 503 bursts and silently mis-initialized workers. Stage 5's zero-5xx gate re-runs after this deploys.


🤖 Generated with Claude Code

anth-volk and others added 2 commits July 8, 2026 22:29
With WEB_CONCURRENCY=2, both gunicorn workers import the app
concurrently on a fresh instance; both could pass the sqlite
exists-check and race initialize() — the losing worker died at boot on
'UNIQUE constraint failed: policy.id' (observed as 503 bursts during
the first real Cloud Run scale-out, Stage 5 LB validation), and a
worker could also observe a created-but-unseeded database and skip
initialization entirely, serving with empty reference tables. An
exclusive fcntl lock around the exists-check + initialize means exactly
one process initializes while the others wait, then see the complete
file and skip.

Empirical verification (6 concurrent fresh boots x 3 rounds): unfixed
code failed every round (boot crashes and row counts of 0/2/5 across
workers); fixed code passes with all workers seeing exactly 5 seed
rows.

Fixes #3746

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@codecov

codecov Bot commented Jul 8, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.92%. Comparing base (ab7ee0e) to head (05801c1).
⚠️ Report is 6 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #3747      +/-   ##
==========================================
+ Coverage   79.89%   79.92%   +0.02%     
==========================================
  Files          70       70              
  Lines        4332     4338       +6     
  Branches      807      808       +1     
==========================================
+ Hits         3461     3467       +6     
  Misses        652      652              
  Partials      219      219              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@anth-volk anth-volk marked this pull request as ready for review July 8, 2026 20:41
@anth-volk anth-volk merged commit 48a7bdb into master Jul 8, 2026
10 checks passed
@anth-volk anth-volk deleted the fix/local-db-init-race branch July 8, 2026 20:43
anth-volk added a commit that referenced this pull request Jul 9, 2026
Both runs: 2026-07-08 (all gates; caught the sqlite cold-boot race,
fixed in #3747) and the 2026-07-09 zero-5xx re-run (race confirmed
fixed across a 1->4 scale-out; three 504s dissected and recorded as
the known cold-scale-out saturation artifact). Stage 5 closed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Two-worker cold boot races local sqlite initialization, causing 503 bursts on Cloud Run scale-out

1 participant