fix(bigquery): format incremental cursor literal against the live column type - #68012
Merged
Gilbert09 merged 1 commit intoJul 3, 2026
Merged
Conversation
…umn type
Incremental syncs built the cursor bound purely from the incremental_field_type
recorded at source setup. When a column was retyped to DATE in BigQuery after
schema discovery, that stored type still said DateTime/Timestamp, so we emitted a
datetime-shaped literal ("1970-01-01T00:00:00") that BigQuery refuses to cast to
DATE, failing every sync with a BadRequest.
Drive the literal off the column's live BigQuery type from bq_table.schema
(falling back to the stored type when the column isn't found), mirroring how
row-filter conditions already resolve column types. DATE columns now get a
date-only literal; DATETIME stays tz-naive; TIMESTAMP keeps its offset.
Generated-By: PostHog Code
Task-Id: cd4fe323-d03a-4406-8ae1-4bbd8a8c9d28
Contributor
|
Hey @Gilbert09! 👋 It looks like your git author email on this PR isn't your
You can fix it for this repo with: git config user.email "you@posthog.com"Or set it globally with |
Contributor
|
Reviews (1): Last reviewed commit: "fix(bigquery): format incremental cursor..." | Re-trigger Greptile |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
BigQuery imports on incremental syncs are crashing with a
BadRequest:Error tracking issue: https://us.posthog.com/project/2/error_tracking/019f244c-a6df-7d21-8b68-76deada789b3
The stack trace is entirely inside our code:
_run→_with_job_not_found_retry→_query_result_with_job_retry→_get_rows_to_syncinproducts/warehouse_sources/backend/temporal/data_imports/sources/bigquery/bigquery.py.The incremental query builder (
_get_query) formats the cursor bound purely fromincremental_field_type, the type we recorded when the source was set up. When a column gets retyped toDATEin BigQuery after schema discovery (dbt-managed datasets do this), that stored type still saysDateTime/Timestamp, so we emit a datetime-shaped literal (1970-01-01T00:00:00). BigQuery won't implicitly cast that to aDATEcolumn, so the COUNT query — and the data-read query, which builds the same clause — fail on every attempt.Changes
_get_querynow formats the incremental cursor literal from the column's live BigQuery type (read offbq_table.schema), falling back to the storedincremental_field_typewhen the column isn't in the schema. This mirrors how_bq_row_filter_conditionsalready resolves column types.DATEcolumn: date-only literal (1970-01-01), which casts cleanly.DATETIMEcolumn: timezone-naive literal (unchanged behavior).TIMESTAMPcolumn: keeps the offset (unchanged behavior).This is a fixable bug, not a user/upstream error: our generated SQL was wrong for the column, and we already have the correct type in hand at query-build time.
How did you test this code?
Automated only (I, Claude, cannot run a live BigQuery sync). Added a parameterized regression test
test_bigquery_get_query_date_column_with_datetime_cursorthat builds_get_queryagainst aDATEcolumn with aDateTime/Timestampcursor and asserts the literal is date-only (noT00:00:00, no+00:00) — this fails on the old code path and catches the exact cast crash. Updated the existingtest_bigquery_get_query_datetime_cursor_timezone_offsetto attach a realistic column schema so it now exercises the live-type lookup for matchingDATETIME/TIMESTAMPcolumns.Ran the bigquery source suite locally: 115 passed.
ruff check/ruff formatclean.Automatic notifications
🤖 Agent context
Autonomy: Fully autonomous
Triaged from an error-tracking webhook by Claude (Claude Code). Confirmed the issue in PostHog error tracking (9 occurrences across 9 sources, all first seen the same day), read the full stack trace, and traced the failing query construction to
_get_query. Skills invoked:/writing-tests.Considered two fixes: (1) special-case only the datetime→DATE mismatch, or (2) drive all datetime-literal formatting off the live column type. Chose (2) — it's one coherent rule, matches what row filters already do, and also closes the symmetric latent crash (a
Timestamp-stored cursor against a now-DATETIMEcolumn would fail the same way with an offset literal). Kept it scoped to literal formatting; did not touch the retry policy or the incremental operator selection.