File Cloudflare support ticket: per-database D1 request stalls on uploads-production

[zachdunn]

Context

2026-08-23: intermittent prod slowness traced to Cloudflare-side stalls on the uploads-production D1 database — not app code. App-side hardening shipped in #805 / #806 / #807 (bounded auth reads, DO-backed rate limiting, resilient session gate), so the app now degrades gracefully, but the underlying stalls are Cloudflare's to fix. This issue tracks filing the support ticket and holds the proposed copy.

Proposed ticket copy

---

Subject: Periodic request-round-trip stalls on one D1 database (query execution stays sub-ms)

Our D1 database uploads-production (id 5f73946b-edd5-414c-856c-8f9bd3c644ba, ENAM primary, ~3.7 MB, low query volume, replication disabled) is experiencing periodic stalls where any access to it — Workers binding queries and independent REST-API queries alike — takes 2–25 seconds instead of the normal sub-second round trip.

Important framing: the dashboard's query-latency metrics look completely healthy, and d1 insights shows all queries executing in under 1 ms. The stall is not query execution — it is the request round-trip to the database (queueing/routing). Please don't close this on the basis of the latency graphs.

Reproduction (from our measurements on 2026-08-23, ~18:50–18:55 PT):

• Timed wrangler d1 execute uploads-production --remote --command "SELECT 1" in a loop: baseline ~1.1 s, with spikes of 12.0 s, 7.7 s, and 8.2 s arriving roughly every 55–60 seconds.
• A control database on the same account (uploads-preview) probed in the same runs stayed flat at ~1.0–1.2 s throughout — so this is per-database, not account- or colo-wide.
• Simultaneous wrangler tail on the Workers bound to this database shows requests with wallTime ≈ 4000+ ms and cpuTime ≈ 0–5 ms during the same windows — the Workers are purely waiting on D1.
• First observed 2026-08-23 (new behavior); recurring across the day in ~5–25 s windows.

Ask: please investigate the serving path for this specific database — routing, placement, or storage-layer contention — since query execution itself is demonstrably fine.

---

Checklist

• File the ticket with Cloudflare (dashboard → Support), attaching fresh probe timings if the stalls are still reproducible
• Link the ticket number back here
• Optional zone-level backstop while waiting: per-IP WAF rate-limiting rule on uploads.sh/api/auth/*

Related: #805, #806, #807.

[zachdunn]

Prior-art check (2026-08-23): searched cloudflare/workers-sdk, cloudflare/workerd, cloudflare/cloudflare-docs, and better-auth/better-auth (open + closed, ~18 months) for this symptom pattern. No existing issue matches — nothing describes periodic per-database stall windows (~55–60 s cadence, 2–25 s round trips) with clean execution-latency metrics and an unaffected sibling database. Closest tangential hits, all different mechanisms: workers-sdk #7376 (dev-proxy idle disconnect), workers-sdk #10801 (D1_ERROR on idle remote bindings — hard error, not variable latency), workerd #1618 (monotonic local-sqlite growth, not periodic), better-auth #10188 (per-request router-rebuild CPU cost on Workers, not DB round-trip). So the support ticket would be a novel report; worth stating that no public issue tracks this.

[zachdunn]

Ticket filed with Cloudflare: Case #02293891 (2026-08-23).

[zachdunn]

Question to add to Case #02293891: would enabling D1 read replication route read queries around whatever is stalling this database? i.e., is the stall specific to the primary's placement/storage path (replicas would dodge it for Sessions-API reads), or in shared routing infrastructure (replicas would inherit it)? The answer determines whether migrating our read paths to the Sessions API is a useful mitigation. Note for us: enabling replication alone does nothing — only queries made through db.withSession(...) are eligible for replica serving; all current binding calls go to the primary regardless.

[zachdunn]

WAF backstop is live: rate-limiting rule auth-endpoints-per-ip on uploads.sh/api/auth/* — 100 req/min per IP (per colo), block 60s with JSON 429. Verified via ruleset read-back and a normal get-session still passing through. That closes the optional checklist item; remaining open item is the Cloudflare response on Case #02293891.

[zachdunn]

Additional evidence for Case #02293891 (2026-08-23 ~20:32–20:36 ET, correlated 3-way probe — cookieless get-session vs static web fetch vs REST SELECT 1):

• 20:33:15: get-session 503 (auth;dur=4000, our bound) concurrent with a 17.8 s D1 REST probe.
• 20:34:29 and 20:34:37: two get-session 503s while the D1 probe was flat (~1.1 s) — the auth worker's invocation path stalls independently of the database.
• The static-fetch control through the web worker stayed ~0.1 s through every event; cold start is ruled out (construction ≈1.8 s max, events 8 s apart).
• A cookieless get-session performs no database read, yet stalls >4 s with ~0 ms CPU (wall≈4000ms, cpu≈3ms in tails) — the runtime is waiting on something platform-side we cannot observe.

Suggested ticket addendum: the stall is not limited to the D1 request path — one specific Worker (uploads-auth) exhibits the same multi-second, zero-CPU invocation stalls, sometimes concurrent with the D1 stalls and sometimes alone, suggesting shared underlying placement/storage infrastructure. Baseline for the same request between windows: ~8 ms.

[zachdunn]

Timeline finding worth adding to Case #02293891: the auth worker cut over from its dedicated D1 database to the shared uploads-production database on 2026-08-22 (#762, the #754 item-1 consolidation — table merge + import). The stalls began 2026-08-23, one day later. App-side mechanisms remain ruled out (REST-path stalls with no Workers involved; cookieless requests stall with no D1 read; ~0 CPU during stalls; no app cron at the ~60 s cadence), but the migration is a plausible trigger: post-import compaction / storage-layer rebalancing / placement churn on the database's serving path would produce exactly this signature, and would explain why the uploads-auth Worker shares the stall pattern. Suggested addition to the ticket: "this database absorbed a table merge/import on 2026-08-22; stalls began 2026-08-23 — please check storage-layer/placement activity for it since that date."