Skip to content

fix(modeling): fix generic delta errors and s3 fnf errors from empty queryable tables#60688

Merged
andrewjmcgehee merged 3 commits into
masterfrom
andrew/delta-fnf-errors
Jun 1, 2026
Merged

fix(modeling): fix generic delta errors and s3 fnf errors from empty queryable tables#60688
andrewjmcgehee merged 3 commits into
masterfrom
andrew/delta-fnf-errors

Conversation

@andrewjmcgehee
Copy link
Copy Markdown
Contributor

@andrewjmcgehee andrewjmcgehee commented May 29, 2026

Problem

we get generic delta errors from renamed (lowercased) columns names; we also get file not found errors in s3

Changes

avoid the delta data fusion write path by using always using mode overwrite. we delete the s3 folder anyway.
to be clear, this is a temporary fix. if we want to support incremental matviews in the future we will HAVE to
find a better way around this. probably some setting somewhere in deltalake. but if we are moving to iceberg
/ ducklake and away from delta format anyway this might not be worth digging into.

How did you test this code?

tests

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

Publish to changelog?

no

Docs update

no

🤖 Agent context

gg stack

@assign-reviewers-posthog assign-reviewers-posthog Bot requested a review from a team May 29, 2026 15:06
@andrewjmcgehee andrewjmcgehee added the stamphog Request AI review from stamphog label May 29, 2026
@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps Bot commented May 29, 2026

Prompt To Fix All With AI
Fix the following 1 code review issue. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 1
posthog/temporal/data_modeling/activities/materialize_view.py:579-588
**Drain loop not cancellation-safe under activity cancellation**

If Temporal cancels the activity while the drain loop is sleeping, `await asyncio.sleep(0.05)` raises `CancelledError`, which propagates immediately out of the `finally` block — `await producer_task` is never reached. If the producer is simultaneously blocked on `await asyncio.to_thread(batch_queue.put, SENTINEL)` (queue still full because nothing drained it), the thread pool thread holding that `put` is orphaned until the GC eventually drops the queue, and the producer task itself is left un-awaited. Wrapping the drain section in `try/except asyncio.CancelledError` (re-raising after awaiting the task) or guarding with `asyncio.shield(producer_task)` would close this gap.

Reviews (1): Last reviewed commit: "fix delta generic errors and s3 fnf erro..." | Re-trigger Greptile

Comment thread posthog/temporal/data_modeling/activities/materialize_view.py
@andrewjmcgehee andrewjmcgehee force-pushed the andrew/delta-fnf-errors branch from 9319162 to ecc661a Compare May 29, 2026 16:31
@andrewjmcgehee andrewjmcgehee removed the stamphog Request AI review from stamphog label May 29, 2026
@andrewjmcgehee andrewjmcgehee merged commit 8d5cf28 into master Jun 1, 2026
200 of 204 checks passed
@andrewjmcgehee andrewjmcgehee deleted the andrew/delta-fnf-errors branch June 1, 2026 02:21
@deployment-status-posthog
Copy link
Copy Markdown

deployment-status-posthog Bot commented Jun 1, 2026

Deploy status

Environment Status Deployed At Workflow
dev ✅ Deployed 2026-06-01 02:43 UTC Run
prod-us ✅ Deployed 2026-06-01 02:54 UTC Run
prod-eu ✅ Deployed 2026-06-01 02:56 UTC Run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants