-
Notifications
You must be signed in to change notification settings - Fork 298
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
BigQuery: raise a custom exception if 400 BadRequest
is encountered due to "internal error during execution"
#23
Comments
@tswast I'm pretty sure that this is a back-end issue: the API shouldn't be returning '400 Bad Request' for internal server errors. Can you confirm? |
@shollyman Is this related to another BigQuery backend rollout? |
I agree that 400 Bad Request is the wrong response code for this error. |
I've filed bug 124319762 internally to track this issue. I see several similar reports internally, so this is likely not new behavior. @bencaine1 If you have a support plan, I recommend filing a ticket with them to raise the priority of this issue on the BigQuery backend. |
@tswast Do you want to leave this issue open (i.e., do you imagine we will be making changes here to work around the 400?). |
Let's close this. The client workaround would be to look for certain text in the response body and ignore the response code, which I'd prefer not to do if we can avoid it. |
I saw this error again today. Has the underlying BigQuery issue been fixed? Is there another new issue with the same symptom? If this bug continues to recur with various BigQuery bugs, I think there is (sadly) a case for having the client retry, because otherwise the non-Google customer application becomes responsible for retrying. That seems ... worse. |
Backend issue was closed as infeasible. Backend engineers say:
Since the job ID cannot be reused, this error is one that requires the whole job to be retried from the beginning. I think it's reasonable to do this, though will likely a bit difficult to do, as the failure won't be discovered until |
400 BadRequest
is encountered due to "internal error during execution"
If the error reason is "jobBackendError", it should be definitely be included in the BigQuery error table, so that developers can deal with it appropriately. |
If the job ID is not reusable, and it is not possible to retry the job from within the library, |
@pietrodn It is possible to retry the job from within the library, for that need to use python-bigquery/google/cloud/bigquery/client.py Line 1594 in 5a422eb
|
I think this is a reasonable feature request. |
400 BadRequest
is encountered due to "internal error during execution"400 BadRequest
is encountered due to "internal error during execution"
Some requirements for a custom exception:
|
At this point, going to close this item as "Will not fix" due to competing priorities. |
OS: Linux dc32b7e8763a 4.9.0-6-amd64 googleapis/google-cloud-python#1 SMP Debian 4.9.82-1+deb9u3 (2018-03-02) x86_64 x86_64 x86_64 GNU/Linux
Python version: Python 2.7.6
google-cloud-bigquery: 1.8.0
We're getting flaky 400 BadRequest errors on our query jobs. We've been seeing this issue for a while on and off, but last night starting at around 7pm we saw a spike in these failures.
These errors are not caught by the default Retry objects because 400 usually signifies a malformed query or a missing table, rather than a transient error.
A fix might be to add a clause catching 400s with this exact error message to _should_retry at https://github.com/googleapis/google-cloud-python/blob/master/bigquery/google/cloud/bigquery/retry.py#L30 and/or RETRY_PREDICATE at https://github.com/googleapis/google-cloud-python/blob/master/api_core/google/api_core/future/polling.py#L32.
Code example
Stack trace
One example:
The text was updated successfully, but these errors were encountered: