Some rows are not replicated onto final table (pg source and bigquery destination)

[rezmuh]

Enviroment

• Airbyte version: 0.30.23-alpha
• OS Version / Instance: GKE 1.19
• Deployment: Kubernetes
• Source Connector and version: postgres 0.3.11
• Destination Connector and version: Bigquery 0.5.0
• Severity: High
• Step where error happened: Sync job

Current Behavior

I'm replicating Postgres using Logical replication with pgoutput from CloudSQL to BigQuery. The sync mode is using incremental + dedup + history. And I sync every 30 mins.

Everyday, there would be some rows are not updated to the final table. In my example, I have a final order_items table. If I search from order_items_scd table for a specific order_id, I see there are 12 rows. But If I query from the final order_items table, there would only be 6 rows returned.

What I found was that those rows which were not carried over to the final order_items table, have _airbyte_end_at se to a UTC timestamp and its _airbyte_active_row field set to 0. However, these rows are still active and appear in the source database.

If I do a full refresh, then those missing rows would appear in the final order_items table. This is happening everyday (we would be missing rows from 20+ order ids in one day).

Expected Behavior

Those rows should appear.

Steps to Reproduce

1. Set pg logical replication with pgoutput
2. sync mode: incremental + dedup + history
3. sync every 15 mins
4. Within 24 hours, there will be some missing rows.

Are you willing to submit a PR?

no

Reference

Initially raised on: https://airbytehq.slack.com/archives/C01MFR03D5W/p1635771024396100

[rezmuh]

@sherifnada any additional info you guys need that I need to provide so that we can fix this?

[sashaNeshcheret]

Could not reproduce the issue, number of records in postgres are equal to the number in bigquery. @rezmuh can you check if it is still reproducible and please, share the logs.

[sashaNeshcheret]

Closed the issue as not reproducible.