Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: QueryJob.exception() *returns* the errors, not raises them #467

Merged
merged 8 commits into from Feb 25, 2021

Conversation

@plamut
Copy link
Contributor

@plamut plamut commented Jan 12, 2021

Fixes #451.

This PR makes sure that QueryJob.exception() actually returns exceptions as documented, not raises them.

Checklist:

  • Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
  • Ensure the tests and linter pass
  • Code coverage does not decrease (if any source code was changed)
  • Appropriate docs were updated (if necessary)
@plamut plamut requested a review from tswast Jan 12, 2021
@plamut plamut requested a review from as a code owner Jan 12, 2021
@google-cla google-cla bot added the cla: yes label Jan 12, 2021
google/cloud/bigquery/job/query.py Outdated Show resolved Hide resolved
Loading
google/cloud/bigquery/job/query.py Outdated Show resolved Hide resolved
Loading
google/cloud/bigquery/job/query.py Outdated Show resolved Hide resolved
Loading
google/cloud/bigquery/job/query.py Outdated Show resolved Hide resolved
Loading
tswast
tswast approved these changes Jan 25, 2021
Copy link
Contributor

@tswast tswast left a comment

Thanks!

Loading

Copy link
Contributor

@tswast tswast left a comment

Retracting approval until

Or can we call set_exception and return False in this case? We'd want a unit test that confirms result() doesn't get into an infinite loop.

is handled.

Loading

@plamut plamut requested a review from tswast Jan 27, 2021
@plamut
Copy link
Contributor Author

@plamut plamut commented Jan 27, 2021

Hmm, TestQueryJob.test_result_w_retry seems to be flaky (failed run).

oogle.api_core.exceptions.RetryError: Deadline of 0.0s exceeded while calling functools.partial(functools.partial(<MagicMock name='mock.api_request' spec='function' id='140372364135632'>, method='GET', path='/projects/project/jobs/JOB_ID', query_params={}, timeout=None))

The same test passed with Python 3.6, 3.8, and also locally - let's see if the results are consistent on a re-run.

Loading

google/cloud/bigquery/job/query.py Outdated Show resolved Hide resolved
Loading
@plamut
Copy link
Contributor Author

@plamut plamut commented Jan 28, 2021

I was able to reproduce flakiness on my machine by reducing the retry deadline in affected tests by ~25 %, which means the latter was indeed the culprit. Because done() now does a bit of extra work, it appears that was just enough of a slowdown to start causing the failures (at least under Python 3.7).

Although not ideal, increasing the deadline should be a solution sufficient with high enough probability.

Loading

google/cloud/bigquery/job/query.py Outdated Show resolved Hide resolved
Loading
is_done = job.done()

assert is_done
assert isinstance(job._exception, exceptions.BadRequest)
Copy link
Contributor

@tswast tswast Feb 2, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why use the private property in this and the other tests? any objections to calling job.exception() here and the other tests?

Loading

Copy link
Contributor Author

@plamut plamut Feb 23, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reasoning was that job.exception() can execute additional logic, and errors in that method would make the tests for done() fail, too, even if there was nothing wrong with the done() method itself.

One could argue that the chosen unit of test is too small, and that the class itself should represent a unit as opposed to its individual methods, but addressing that would require quite some refactoring (we already tinker with internal _properties , for instance).

Here, practicality beats purity IMHO, thus the "cheating" by examining the internal state of the class. Or do you have a strong opinion on this?

Loading

Copy link
Contributor

@tswast tswast Feb 25, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works for me. I agree that ideally we'd have higher-level tests than this, but makes sense to stay with existing conventions, especially given our 100% coverage requirement.

Loading

@plamut plamut requested a review from tswast Feb 23, 2021
tswast
tswast approved these changes Feb 25, 2021
is_done = job.done()

assert is_done
assert isinstance(job._exception, exceptions.BadRequest)
Copy link
Contributor

@tswast tswast Feb 25, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works for me. I agree that ideally we'd have higher-level tests than this, but makes sense to stay with existing conventions, especially given our 100% coverage requirement.

Loading

@tswast tswast merged commit d763279 into googleapis:master Feb 25, 2021
10 of 11 checks passed
Loading
gcf-merge-on-green bot pushed a commit that referenced this issue Feb 25, 2021
🤖 I have created a release \*beep\* \*boop\* 
---
## [2.10.0](https://www.github.com/googleapis/python-bigquery/compare/v2.9.0...v2.10.0) (2021-02-25)


### Features

* add BIGNUMERIC support ([#527](https://www.github.com/googleapis/python-bigquery/issues/527)) ([cc3394f](https://www.github.com/googleapis/python-bigquery/commit/cc3394f80934419eb00c2029bb81c92a696e7d88))


### Bug Fixes

* error using empty array of structs parameter ([#474](https://www.github.com/googleapis/python-bigquery/issues/474)) ([c1d15f4](https://www.github.com/googleapis/python-bigquery/commit/c1d15f4e5da4b7e10c00afffd59a5c7f3ded027a))
* QueryJob.exception() *returns* the errors, not raises them ([#467](https://www.github.com/googleapis/python-bigquery/issues/467)) ([d763279](https://www.github.com/googleapis/python-bigquery/commit/d7632799769248b09a8558ba18f5025ebdd9675a))


### Documentation

* **bigquery:** Add alternative approach to setting credentials ([#517](https://www.github.com/googleapis/python-bigquery/issues/517)) ([60fbf28](https://www.github.com/googleapis/python-bigquery/commit/60fbf287b0d34d5db2e61cce7a5b42735ed43d0e))
* explain retry behavior for DONE jobs ([#532](https://www.github.com/googleapis/python-bigquery/issues/532)) ([696c443](https://www.github.com/googleapis/python-bigquery/commit/696c443f0a6740be0767e12b706a7771bc1460c3))
---


This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
@plamut plamut deleted the iss-451 branch Feb 25, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

3 participants