Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Destination BigQuery: add row ids for dummy inserts in check #34021

Merged
merged 4 commits into from Jan 8, 2024

Conversation

gisripa
Copy link
Contributor

@gisripa gisripa commented Jan 8, 2024

What

Fixes: #33982

How

Adding row-ids for BQ client to use ids for idempotency in insertAll rpc.

Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
@gisripa gisripa requested a review from a team as a code owner January 8, 2024 20:48
Copy link

vercel bot commented Jan 8, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
airbyte-docs ✅ Ready (Inspect) Visit Preview 💬 Add feedback Jan 8, 2024 10:12pm

Copy link
Contributor

github-actions bot commented Jan 8, 2024

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

  • PR name follows PR naming conventions
  • Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
  • Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
  • You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
  • Secrets in the connector's spec are annotated with airbyte_secret
  • All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
  • Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
  • Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
  • If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

  1. Check for hidden checklists in your PR description

  2. Toggle the github label checklist-action-run on/off to re-run the checklist CI.

@gisripa gisripa changed the title destination-bigquery: add row ids for dummy inserts in check Destination BigQuery: add row ids for dummy inserts in check Jan 8, 2024
Copy link
Contributor

@edgao edgao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔥 (lgtm assuming tests pass)

do we (should we) add row ids in the real inserts? I would guess we saw some 502s during syncs as well - presumably those eventually retried the entire sync, but retrying within a single attempt would be nicer. (this is scope creep, feel free to make that a separate ticket)

@gisripa
Copy link
Contributor Author

gisripa commented Jan 8, 2024

@edgao for standard inserts we use local batch load mechanism https://cloud.google.com/bigquery/docs/batch-loading-data#loading_data_from_a_local_data_source with a random JobID generated. This is the specific method invoked https://github.com/googleapis/java-bigquery/blob/main/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/TableDataWriteChannel.java#L52 which has retries by default using uploadID as idempotency

Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
@octavia-squidington-iii octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Jan 8, 2024
@gisripa gisripa enabled auto-merge (squash) January 8, 2024 22:10
@gisripa gisripa merged commit 0e4ffb5 into master Jan 8, 2024
23 of 25 checks passed
@gisripa gisripa deleted the gireesh/bigquery/check-fix branch January 8, 2024 23:13
jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 26, 2024
jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 26, 2024
jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/connectors Connector related issues area/documentation Improvements or additions to documentation checklist-action-run connectors/destination/bigquery
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Bigquery's check method doesn't do retries for insertAll
3 participants