New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Destination BigQuery: add row ids for dummy inserts in check #34021
Conversation
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
Before Merging a Connector Pull RequestWow! What a great pull request you have here! 🎉 To merge this PR, ensure the following has been done/considered for each connector added or updated:
If the checklist is complete, but the CI check is failing,
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🔥 (lgtm assuming tests pass)
do we (should we) add row ids in the real inserts? I would guess we saw some 502s during syncs as well - presumably those eventually retried the entire sync, but retrying within a single attempt would be nicer. (this is scope creep, feel free to make that a separate ticket)
@edgao for standard inserts we use local batch load mechanism https://cloud.google.com/bigquery/docs/batch-loading-data#loading_data_from_a_local_data_source with a random JobID generated. This is the specific method invoked https://github.com/googleapis/java-bigquery/blob/main/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/TableDataWriteChannel.java#L52 which has retries by default using uploadID as idempotency |
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
…hq#34021) Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
…hq#34021) Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
…hq#34021) Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
What
Fixes: #33982
How
Adding row-ids for BQ client to use ids for idempotency in insertAll rpc.