[source-bigquery] BigQuery Incremental Sync with non-granular cursor - expected behavior?

[Emon Datta (emdat)]

Hello,

Question: I'm syncing a BigQuery table to a snowflake table. We have a primary key and cursor in the BigQuery table, which is quite large, so we are using the Incremental - Append + Deduped field. If we have new rows that are added (or modified) after a sync runs that have the same cursor value as the previous max cursor value, will they still be processed on the next sync run?

Example: For example, let's say we have 1 row where the cursor indicates the updated_at is "June 3 2026 at 1 am", and the sync runs with that as the max cursor values. After the sync runs, 6 additional rows (perhaps some net-new rows, and some updated rows of existing data) now have the same updated_at as "June 3 2026 at 1 am". Would those 6 additional rows still be processed on the next sync run?

Context: I'm having trouble understanding what the expectation is for this scenario is, since the documentation mentions the cursor query logic being select * from table where cursor_field > 'last_sync_max_cursor_field_value' which implies that those 6 rows wouldn't be processed since their cursor value is NOT > 'last_sync_max_cursor_field_value'. However, the Inclusive Cursors section of the documentation seems to suggest that these rows would indeed be processed. Which is correct?

Thank you!

[Metehan Kaygısız (metehankaygsz)]

For the current source-bigquery connector, the six late rows will not be picked up.

The distinction is source extraction versus destination deduplication:

• BigQuery incremental extraction currently executes WHERE cursor > ?.
• The BigQuery connector docs also describe the cursor as exclusive and monotonically increasing.
• Append + Deduped only deduplicates records that the source emitted. It cannot recover records excluded by the source query.

So after state advances to 2026-06-03 01:00, rows inserted later with that exact same cursor value fail the > predicate. The general "inclusive cursors" section describes the convention sources should follow when a cursor is ambiguous, but this connector's implementation is not inclusive.

The reliable fix is to make the cursor strictly increasing at the granularity of your writes, for example:

1. Store a full-precision TIMESTAMP rather than an hour/day-rounded value.
2. Ensure every insert/update assigns an updated_at later than the previous saved cursor.
3. Use a monotonically increasing ingestion/sequence column. If you need (updated_at, primary_key) as a tie-breaker, expose a single synthetic sortable cursor through a BigQuery view, since the connector selects one cursor field.

If late rows with equal cursor values are unavoidable, use a periodic state reset/full refresh or a custom extraction with a lookback window. A primary key helps deduplication after extraction, but does not prevent this gap.