[Spanner] authentication issues should result in a user error

[jeanbza]

1. Copy paste code from https://cloud.google.com/spanner/docs/getting-started/python/
2. Run

... it hangs forever

3. Waste Spend hour struggling to figure out what is happening.

4. Figure out how to turn on http logging with,

   import logging
   
   logging.basicConfig(level=logging.DEBUG)
   

5. See:

   DEBUG:google.auth.transport.requests:Making request: POST https://oauth2.googleapis.com/token
   DEBUG:urllib3.connectionpool:https://oauth2.googleapis.com:443 "POST /token HTTP/1.1" 400 None
   ERROR:grpc._plugin_wrapping:AuthMetadataPluginCallback "<google.auth.transport.grpc.AuthMetadataPlugin object at 0x10d79d4a8>" raised exception!
   Traceback (most recent call last):
   File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/grpc/_plugin_wrapping.py", line 79, in __call__
       callback_state, callback))
   File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/google/auth/transport/grpc.py", line 77, in __call__
       callback(self._get_authorization_headers(context), None)
   File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/google/auth/transport/grpc.py", line 65, in _get_authorization_headers
       headers)
   File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/google/auth/credentials.py", line 122, in before_request
       self.refresh(request)
   File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/google/oauth2/service_account.py", line 322, in refresh
       request, self._token_uri, assertion)
   File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/google/oauth2/_client.py", line 145, in jwt_grant
       response_data = _token_endpoint_request(request, token_uri, body)
   File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/google/oauth2/_client.py", line 111, in _token_endpoint_request
       _handle_error_response(response_body)
   File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/google/oauth2/_client.py", line 61, in _handle_error_response
       error_details, response_body)
   google.auth.exceptions.RefreshError: ('invalid_grant: Not a valid email or user ID.', '{\n  "error": "invalid_grant",\n  "error_description": "Not a valid email or user ID."\n}')
   DEBUG:google.api_core.retry:Retrying due to 503 Getting metadata from plugin failed with error: ('invalid_grant: Not a valid email or user ID.', '{\n  "error": "invalid_grant",\n  "error_description": "Not a valid email or user ID."\n}'), sleeping 1.3s ...
   

This is an error that will never resolve. We should surface it to the user immediately.

Also: I have no idea why this is a 503 UNAVAILABLE. Why would it not be a 400 BAD REQUEST or 401 UNAUTHORIZED??

[jeanbza]

cc @broady

[HemangChothani]

found the related issue in java also 3573

[HemangChothani]

@jadekler I tried to figure out why it throws 503, as per my research urllib3 throws 400 but the grpc convert into the INTERNAL error code and that is 13, may be the suspected line of code and then rest is done in google-api-core.

[jeanbza]

@HemangChothani Is INTERNAL a retried code?

[HemangChothani]

@jadekler I think yes, due to that i have got the error multiple times and by default deadline is 3600s. So due to this foor loop which is in api_core/retry.py and long deadline it looks like goes into the infinite loop or it hangs.

[broady]

Looks like it's a 400 at one point, which indicates some broken plumbing where it didn't surface properly? I'm confused how it seems to be multiple codes at once - is it because the 400 wasn't treated as an error?

…

On Tue, Dec 3, 2019, 10:52 PM HemangChothani ***@***.***> wrote: @jadekler <https://github.com/jadekler> I think yes, due to that i have got the error multiple times and by default deadline is 3600s. So due to this foor loop <https://github.com/googleapis/google-cloud-python/blob/master/api_core/google/api_core/retry.py#L180> which is in api_core/retry.py and long deadline it looks like goes into the infinite loop or it hangs. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/googleapis/google-cloud-python/issues/9896?email_source=notifications&email_token=AAAGDFWULRQYQTRKA676C33QW5HSLA5CNFSM4JSA4QV2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEF356II#issuecomment-561504033>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAGDFUVMGYDPYDDYWDHYALQW5HSLANCNFSM4JSA4QVQ> .

[jeanbza]

@jadekler I think yes, due to that i have got the error multiple times and by default deadline is 3600s. So due to this foor loop which is in api_core/retry.py and long deadline it looks like goes into the infinite loop or it hangs.

Gotcha. We probably shouldn't retry INTERNAL (https://aip.dev/194).

Looks like it's a 400 at one point, which indicates some broken plumbing
where it didn't surface properly?

Sounds right.

[HemangChothani]

Another similar issue 7199 has been found, so can we close one of them? Because it is a kind of duplicate issue.

[larkee]

Not sure how to progress here. Seems like an issue in another library. Need to investigate further.

[larkee]

Looks like it's a 400 at one point, which indicates some broken plumbing where it didn't surface properly?

The auth library does not pass through the status error code through to gRPC. However, gRPC would replace it with INTERNAL regardless. See status conversion.

It is still unclear as to why the client receives an UNAVAILABLE error.

[larkee]

This is not an issue with the client library, but with its dependencies. I have filed an issue with grpc and will monitor its progress. Since there is nothing to be done in the client library, I am closing this issue.