Skip to content

fix(analytics-platform): disable INSERT quorum in local dev so lazy precompute does not hang#59250

Merged
lricoy merged 1 commit into
masterfrom
lricoy/lazy-computation-disable-quorum-in-debug
May 20, 2026
Merged

fix(analytics-platform): disable INSERT quorum in local dev so lazy precompute does not hang#59250
lricoy merged 1 commit into
masterfrom
lricoy/lazy-computation-disable-quorum-in-debug

Conversation

@lricoy
Copy link
Copy Markdown
Member

@lricoy lricoy commented May 20, 2026

Problem

Local dev runs against a single-replica ClickHouse, but the lazy computation framework hardcodes insert_quorum: \"auto\" on every INSERT. On a single-node setup, ClickHouse writes immediately but still blocks on the quorum protocol — the INSERT then hits the 600s max_execution_time and dies with:

Code: 319. Unknown quorum status. The data was inserted in the local replica
  but we could not verify quorum. Reason: Timeout while waiting for quorum:
  Insertion status: Wrote 1 blocks and N rows on shard 0 replica 0

Every fresh lazy precompute request hangs for 10 minutes locally, fails, and falls through to the live path. Tests already had a special case for this (PREAGGREGATION_INSERT_QUORUM = 0 if TEST else "auto"), but DEBUG mode didn't.

Changes

Extend the existing TEST exemption to also cover DEBUG=True. Prod (DEBUG=False, TEST=False) behavior is unchanged — auto still waits for majority quorum on multi-replica clusters.

How did you test this code?

I'm an agent — no manual UI testing claimed.

  • Locally: WebOverview lazy precompute request went from hanging 600s → completing in ~1.1s. INSERT succeeds with usedLazyPrecompute=True in the response. Surfaced while debugging #59075.
  • hogli test products/analytics_platform/backend/lazy_computation/tests/test_lazy_computation_executor.py → 110 passed.

Publish to changelog?

no

🤖 Agent context

Agent-authored. Found by following the chain "INSERT hangs" → log says "Unknown quorum status" → single-node CH can't satisfy quorum → existing exemption already exists for TEST, just needs to also apply to DEBUG. Same shape and justification as the original TEST guard.

@lricoy lricoy marked this pull request as ready for review May 20, 2026 18:07
@assign-reviewers-posthog assign-reviewers-posthog Bot requested a review from a team May 20, 2026 18:07
@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps Bot commented May 20, 2026

Reviews (1): Last reviewed commit: "fix(analytics-platform): disable INSERT ..." | Re-trigger Greptile

@lricoy lricoy enabled auto-merge (squash) May 20, 2026 18:08
@lricoy lricoy merged commit fe2a77b into master May 20, 2026
220 checks passed
@lricoy lricoy deleted the lricoy/lazy-computation-disable-quorum-in-debug branch May 20, 2026 18:26
@deployment-status-posthog
Copy link
Copy Markdown

deployment-status-posthog Bot commented May 20, 2026

Deploy status

Environment Status Deployed At Workflow
dev ✅ Deployed 2026-05-20 20:18 UTC Run
prod-us ✅ Deployed 2026-05-20 20:33 UTC Run
prod-eu ✅ Deployed 2026-05-20 20:36 UTC Run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants