Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: don't fetch rows when waiting for query to finish #400

Merged

Conversation

tswast
Copy link
Contributor

@tswast tswast commented Nov 24, 2020

When there are large result sets, fetching rows while waiting for the
query to finish can cause the API to hang indefinitely. (This may be due
to an interaction between connection timeout and API timeout.)

This reverts commit 86f6a51 (#374).

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

  • Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
  • Ensure the tests and linter pass
  • Code coverage does not decrease (if any source code was changed)
  • Appropriate docs were updated (if necessary)

Fixes googleapis/python-bigquery-pandas#343
Fixes #394 馃

When there are large result sets, fetching rows while waiting for the
query to finish can cause the API to hang indefinitely. (This may be due
to an interaction between connection timeout and API timeout.)

This reverts commit 86f6a51 (googleapis#374).
@tswast tswast requested review from a team and pmakani November 24, 2020 19:33
@google-cla google-cla bot added the cla: yes This human has signed the Contributor License Agreement. label Nov 24, 2020
@product-auto-label product-auto-label bot added the api: bigquery Issues related to the googleapis/python-bigquery API. label Nov 24, 2020
@tswast tswast requested review from shollyman and removed request for pmakani November 24, 2020 19:33
@@ -1630,6 +1630,40 @@ def test_iterate(self):

api_request.assert_called_once_with(method="GET", path=path, query_params={})

def test_iterate_with_cached_first_page(self):
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I add these tests to fix coverage.

Perhaps instead I should also be reverting the changes from #384 ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The alternative I'd consider is doing a flag-based feature, so users can turn on the latency changes (or flip default, and provide an opt-out while we diagnose these other cases).

@@ -1630,6 +1630,40 @@ def test_iterate(self):

api_request.assert_called_once_with(method="GET", path=path, query_params={})

def test_iterate_with_cached_first_page(self):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The alternative I'd consider is doing a flag-based feature, so users can turn on the latency changes (or flip default, and provide an opt-out while we diagnose these other cases).

@tswast tswast added the automerge Merge the pull request once unit tests and other checks pass. label Nov 24, 2020
@tswast tswast mentioned this pull request Nov 24, 2020
7 tasks
@gcf-merge-on-green gcf-merge-on-green bot merged commit 730df17 into googleapis:master Nov 24, 2020
9 checks passed
@gcf-merge-on-green gcf-merge-on-green bot removed the automerge Merge the pull request once unit tests and other checks pass. label Nov 24, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the googleapis/python-bigquery API. cla: yes This human has signed the Contributor License Agreement.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

simple query hangs in 0.14.1, works in 0.13.3 increased memory usage in 2.4.0
2 participants