You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository has been archived by the owner on Nov 11, 2022. It is now read-only.
If we do a streaming write, the write connector will generate unique ids and tag rows transparently to avoid duplication. If the user knows something about their data like "the transaction-id column is unique across the table", it would be nice to be able to use that instead of the generated ids.
The text was updated successfully, but these errors were encountered:
dhalperi
changed the title
Allow user for user controlled unique id in BigQuery Write
Allow for user controlled unique id in BigQuery Write
Oct 16, 2015
My understanding of this FR is that users typically think this will help them de-dupe across jobs or long periods of time. These row-IDs are very short-lived (1 minute). So it does not make sense to let users control them.
If you have some other ID you can de-dupe with, they can use GroupByKey or RemoveDuplicates within your pipeline to do so.
If we do a streaming write, the write connector will generate unique ids and tag rows transparently to avoid duplication. If the user knows something about their data like "the transaction-id column is unique across the table", it would be nice to be able to use that instead of the generated ids.
The text was updated successfully, but these errors were encountered: