-
Notifications
You must be signed in to change notification settings - Fork 4.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bug]: Bigquery Load jobs with WRITE_TRUNCATE disposition may truncate valid records. #24535
Comments
…together. This fixes apache#24535.
Is this a regression? It sounds severe and worth a cherry pick most likely. |
I don't know at what point this regression was introduced, but it is pretty
severe and I think worth a cherry-pick.
…On Tue, Dec 6, 2022 at 7:24 AM Kenn Knowles ***@***.***> wrote:
Is this a regression? It sounds severe and worth a cherry pick most likely.
—
Reply to this email directly, view it on GitHub
<#24535 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AADWVAO6XBEP3HZUHCHILYTWL5LDLANCNFSM6AAAAAASU3MJEM>
.
You are receiving this because you were assigned.Message ID:
***@***.***>
|
I see it is first reported by a user in #23306. I think keeping the earlier bug is worthwhile so I will dupe this one. |
The bug can still occur when the number of files are greater than 10,000 then bigquery load jobs will be conducted in multiple partitions:
Did a test that decrease this number and force the write happens in multiple bigquery load jobs, still see dropping elements |
reopen for cherry pick |
What happened?
TriggerCopyJobs
was written to assume all jobs belonging to the same table are processed by the same bundle, but this precondition appears to have been invalidated with future changes to the code. Though this still seems to hold most of the time, it may be violated for large writes, and seems to trigger more on runner v2 than runner v1.Issue Priority
Priority: 1
Issue Component
Component: io-py-gcp
The text was updated successfully, but these errors were encountered: