Airbyte fetching duplicate rows from Potgres into SNowflake

[akhilkumar123-eng]

While using airbyte with incrmental | append to load data from postgres to Snowflake, airbyte loading duplicate rows even thoughg the cursor column dateupdaetd is same for all rows, is it expecyed airbte behaviour wiht incremntal |append?

[Airbyte Support Bot (airbyte-support-bot)]

Hi akhilkumar123-eng, thanks for reaching out and for the detailed screenshot!

What you're seeing is expected with Incremental | Append and is worth understanding before you change anything:

• Append does not deduplicate. In Incremental | Append, Airbyte only adds records to the destination — it never updates or removes existing rows. So the same primary key can appear multiple times across syncs.
• The cursor comparison is inclusive. At the start of each sync, Airbyte reads records where cursor >= last_synced_cursor_value (not strictly greater than). This is intentional: it guarantees Airbyte doesn't miss records that were written with the same cursor timestamp as the last one it saw. The trade-off is that records sitting exactly on the cursor boundary get re-read on the next sync.
• When the cursor doesn't advance, everything re-syncs. In your case dateupdated is the same for every row, so the cursor value never moves past those rows. Every sync re-reads all of them and appends them again, which is why you see growing duplicates.

How to avoid the duplicates

• Use Incremental | Append + Deduped with a primary key. This keeps only the latest version of each record (deduplicated on the primary key using the cursor to pick the most recent), which is usually what people want for CDC-style upserts. See the sync modes docs.
• Pick a cursor column that actually advances — e.g. a real updated_at/modified_at timestamp that changes on every row update, or a monotonically increasing ID. A cursor whose value is identical across all rows can't reliably track progress.
• If you want the raw append history but need to query the latest state, you can deduplicate downstream in Snowflake (e.g. QUALIFY ROW_NUMBER() OVER (PARTITION BY <pk> ORDER BY <cursor> DESC) = 1).

Docs that go deeper:

• Incremental Sync - Append
• Incremental Sync - Append + Deduped

If after switching to a properly-advancing cursor and/or Append + Deduped you still see unexpected duplication, let us know your Airbyte version (OSS/Cloud), the Postgres source and Snowflake destination connector versions, and a redacted sync log and we'll take a closer look. (Please mask or remove any sensitive information — passwords, tokens, connection strings — before sharing logs.)

---

Need more help? Join Airbyte Community Slack for peer support, or if you're a Cloud customer, open a support ticket referencing this URL.