Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bigquery: Big Query Managed Writer flow controller can dead lock when connections error out #9540

Closed
benjaminRomano opened this issue Mar 9, 2024 · 1 comment · Fixed by #9649
Closed

bigquery: Big Query Managed Writer flow controller can dead lock when connections error out #9540

benjaminRomano opened this issue Mar 9, 2024 · 1 comment · Fixed by #9649
Assignees
@shollyman @alvarowolfx
Labels
api: bigquery Issues related to the BigQuery API. priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@benjaminRomano
Copy link

Background

We have a service that reads a message from a pub sub topic, performs some computations and then uploads data to a Bigquery table using the managedwriter Big query package.

We are periodically getting io.EOF errors from the AppendRows API which eventually causes the API to dead lock. We have an open Google Cloud support ticket to root cause why we are receiving EOF errors from Bigquery backend.

Reproduction

I was able to reproduce the issue in the following PR by simulating the connection always returning EOFs with a small in-flight request limit:
#9539

Screenshot 2024-03-08 at 9 39 50 PM

It is likely that there are other scenarios where the connection handling logic can error and lead to in flight requests not being properly released.

Workaround

The deadlock can be avoided by disabling the in-flight request limit.
@benjaminRomano benjaminRomano added the triage me I really want to be triaged. label Mar 9, 2024
@product-auto-label product-auto-label bot added the api: pubsub Issues related to the Pub/Sub API. label Mar 9, 2024
@hongalex hongalex changed the title pubsub: Big Query Managed Writer flow controller can dead lock when connections error out bigquery: Big Query Managed Writer flow controller can dead lock when connections error out Mar 14, 2024
@hongalex hongalex added api: bigquery Issues related to the BigQuery API. and removed api: pubsub Issues related to the Pub/Sub API. labels Mar 14, 2024
@hongalex hongalex assigned alvarowolfx and unassigned hongalex Mar 14, 2024
@alvarowolfx alvarowolfx added priority: p2 Moderately-important priority. Fix may not be included in next release. and removed triage me I really want to be triaged. labels Mar 14, 2024
@shollyman shollyman added the type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. label Mar 26, 2024
@shollyman
Copy link
Contributor

Thanks for the report here, sorry for the delay.

The correct behavior is to ensure that connection's lockingAppend refunds the flow controller on errored send. I'm working on a PR for this now.

shollyman added a commit to shollyman/google-cloud-go that referenced this issue Mar 26, 2024
@shollyman
fix(bigquery/storage/managedwriter): fix flowcontrol refund on error
007fd73 
Previously, connection's `lockingAppend` did not properly refund the
connection's flow controller if the send response errored.  This PR
addresses that issue, and includes a test to ensure the correct
behavior.

Fixes: googleapis#9540
@shollyman shollyman mentioned this issue Mar 26, 2024
Merged
shollyman added a commit that referenced this issue Mar 26, 2024
@shollyman
fix(bigquery/storage/managedwriter): fix flowcontrol refund on error (#…
a07bf1d 
…9649)

Previously, connection's `lockingAppend` did not properly refund the
connection's flow controller if the send response errored.  This PR
addresses that issue, and includes a test to ensure the correct
behavior.

Fixes: #9540
@benjaminRomano benjaminRomano mentioned this issue Mar 28, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@shollyman shollyman

@alvarowolfx alvarowolfx

Labels
api: bigquery Issues related to the BigQuery API. priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(bigquery/storage/managedwriter): fix flowcontrol refund on error shollyman/google-cloud-go
4 participants
@shollyman @alvarowolfx @benjaminRomano @hongalex