Intermittent DefaultCredentialsError on GCE

[dhermes]

Original issue: googleapis/google-cloud-python#4358

After successful use of credentials, _ = google.auth.default(), an application crashes when credentials cannot be detected:

...
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/client.py", line 212, in __init__
    Client.__init__(self, credentials=credentials, _http=_http)
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/client.py", line 125, in __init__
    credentials, _ = google.auth.default()
  File "/usr/local/lib/python2.7/dist-packages/google/auth/_default.py", line 286, in default
    raise exceptions.DefaultCredentialsError(_HELP_MESSAGE)
DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or
explicitly create credential and re-run the application. For more
information, please see
https://developers.google.com/accounts/docs/application-default-credentials.

/cc @dmho418

[dhermes]

Also from @dmho418 (may be unrelated to this issue, but is probably the root cause):

...
  File "/usr/local/lib/python2.7/dist-packages/google/resumable_media/requests/upload.py", line 97, in transmit
    retry_strategy=self._retry_strategy)
  File "/usr/local/lib/python2.7/dist-packages/google/resumable_media/requests/_helpers.py", line 101, in http_request
    func, RequestsMixin._get_status_code, retry_strategy)
  File "/usr/local/lib/python2.7/dist-packages/google/resumable_media/_helpers.py", line 146, in wait_and_retry
    response = func()
  File "/usr/local/lib/python2.7/dist-packages/google/auth/transport/requests.py", line 176, in request
    self._auth_request, method, url, request_headers)
  File "/usr/local/lib/python2.7/dist-packages/google/auth/credentials.py", line 121, in before_request
    self.refresh(request)
  File "/usr/local/lib/python2.7/dist-packages/google/auth/compute_engine/credentials.py", line 93, in refresh
    raise exceptions.RefreshError(exc)
RuntimeError: RefreshError: HTTPConnectionPool(host='metadata.google.internal', port=80): Max retries exceeded with url: /computeMetadata/v1/instance/service-accounts/default/?recursive=true

[dhermes]

@jonparrott Would there be a way to access the number of retries that were exceeded?

[theacodes]

It'll be the requests / urllib3 default

[dhermes]

• Failing method (invoked by / raised from refresh())
• Function throwing exception (actual line where thrown)
• Transport used
• Exception being raised (from urllib3)

Default retry strategy for the requests transport:

>>> import requests
>>> session = requests.Session()
>>> adapter1, adapter2 = session.adapters.values()
>>> adapter1.max_retries
Retry(total=0, connect=None, read=False, redirect=None, status=None)
>>> adapter2.max_retries
Retry(total=0, connect=None, read=False, redirect=None, status=None)

From the docstring:

By default, Requests does not retry failed connections.

[dhermes]

I'm trying to reproduce on GCE with:

$ sudo apt-get update
$ sudo apt-get upgrade
$ sudo apt-get install -y python-dev python-pip
$ sudo -H pip install --upgrade google-auth requests
$ cat << EOF > repro.py
> import pdb
> import sys
> import time
>
> import google.auth
>
>
> def main():
>     if len(sys.argv) < 2:
>         num_attempts = 100
>     else:
>         num_attempts = int(sys.argv[1])
>     for n in range(num_attempts):
>         print(n)
>         try:
>             credentials, project = google.auth.default()
>         except:
>             pdb.set_trace()
>             raise
>         time.sleep(0.5)
>
>
> if __name__ == '__main__':
>     main()
> EOF
$
$ python repro.py 10
$ python repro.py 100
$ python repro.py 1000

and am not having any luck.

If I had to guess, I'd say the only way to reproduce would be to stress out the instance (e.g. high CPU usage) so that the process handling the metadata server fails.

[antoineazar]

We see this issue in our setup as well. Python script connecting to BQ and running hundreds of queries in rapid fire sequence. We'll occasionally see:
A INFO:google.auth.compute_engine._metadata:Compute Engine Metadata server unavailable.

followed by a crash:
google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credential and re-run the application.

[dhermes]

Awesome data @antoineazar! Thanks for the confirmation.

Any code you could share so we could try to reproduce / strategize?

[antoineazar]

@dhermes any code that runs multiple (dozens usually suffice) queries on BQ in rapid sequence. Some simple sample code (pre-0.28 library, didn't test with 0.28), you can wrap this in a loop:

client = bigquery.Client()
query_job = client.run_async_query(str(uuid.uuid4()), query)

# Use standard SQL syntax.
query_job.use_legacy_sql = False

# Set a destination table.
dest_dataset = client.dataset(dest_dataset_id)
dest_table = dest_dataset.table(dest_table_id)
query_job.destination = dest_table

# Allow the results table to be overwritten.
query_job.write_disposition = 'WRITE_TRUNCATE'

query_job.begin()
query_job.result()  # Wait for query to finish.

[GEverding]

I'm seeing this issue in production. Is there a workaround/fix?

[theacodes]

@GEverding the recommendation right now is to use a service account keyfile instead of relying on the GCE metadata service.

It's possible that we could make retry failed connections to the metadata service, but I'm unsure on that at the moment.

[GEverding]

Thanks. Is that documented somewhere?

On Thu, Mar 1, 2018 at 12:25 PM Jon Wayne Parrott ***@***.***> wrote: @GEverding <https://github.com/geverding> the recommendation right now is to use a service account keyfile instead of relying on the GCE metadata service. It's possible that we could make retry failed connections to the metadata service, but I'm unsure on that at the moment. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#211 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABUFzCiFYhfaVHVD87TTf7cDedV8XLULks5taC6ogaJpZM4QYcXk> .

-- Garrett

[theacodes]

https://cloud.google.com/docs/authentication/getting-started

[vanpelt]

I'm getting "[Errno 111] Connection refused" triggering the same exceptions @dhermes mentioned fairly regularly on my appengine flex deployment. Is the best solution getting a keyfile into my flex deployment? I'm going to try passing in a requests adapter that has retry logic configured, but it's frustrating I've already spent this much time on such a fundamental feature of this library.

[theacodes]

@vanpelt yep, that is a completely acceptable approach.

[KevinTydlacka]

I'm also seeing these errors. Stack trace below:

DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://developers.google.com/accounts/docs/application-default-credentials.
at default (/env/local/lib/python2.7/site-packages/google/auth/_default.py:306)
at create_channel (/env/local/lib/python2.7/site-packages/google/api_core/grpc_helpers.py:170)
at __init__ (/env/local/lib/python2.7/site-packages/google/cloud/pubsub_v1/publisher/client.py:81)
at handle_asc_sub_notif (/home/vmagent/app/main.py:101)
at dispatch_request (/env/local/lib/python2.7/site-packages/flask/app.py:1598)
at full_dispatch_request (/env/local/lib/python2.7/site-packages/flask/app.py:1612)
at handle_user_exception (/env/local/lib/python2.7/site-packages/flask/app.py:1517)
at full_dispatch_request (/env/local/lib/python2.7/site-packages/flask/app.py:1614)
at wsgi_app (/env/local/lib/python2.7/site-packages/flask/app.py:1982)

I'm running a very, very simple App Engine app in the Flexible Python 2.7 environment. It really doesn't do anything more than the sample found here, using the exact code: https://github.com/GoogleCloudPlatform/python-docs-samples/tree/master/appengine/flexible/pubsub

I ingest post data within a flask app, then publish it to a topic using the pubsub_v1 client. No other errors in the logs before or after, and the app/method successfully processes other requests within seconds before the failed attempt, and within a minute after.

This is a little frustrating/concerning since I'm basically seeing the issue using the provided GAE sample code. I'm updating my app and pushing a service account secrets json with my code, and will see what happens if I manually create the credentials for the client instead, but would love to see this resolved.

[ocervell]

Still seeing this today. Any workaround if storing the credentials file on the server is not possible (riskier than default service account credentials) ?

[joshnewlinatclearobject]

Is this something that I'll need to push up a keyfile with my flex deployment (as @vanpelt mentioned), or will it be fixed in the PR that was just linked?

[dhendry]

I would love to see a fix for this issue

[cpavon]

Is this still open 1 year later?

[mike-seekwell]

@JustinBeckwith @theacodes I see #323 was merged, should this issue be fixed now? A lot of people will likely land here when they run into this issue, you should confirm the fix here and, if it's really a fix, tell people to upgrade (google-auth==1.6.3).

[busunkim96]

@mike-seekwell Thanks for the call out! #323 merged a fix to retry the ping to the metadata server.

If you're seeing this error, please upgrade to version 1.6.3 or greater.

[ghost]

I'm still seeing this error fairly regularly running on GAE flex for Python 3.6 with google-auth==1.6.3. Here's the full stack trace:

  ...
  File "/env/lib/python3.6/site-packages/google/auth/transport/requests.py", line 205, in request
    self._auth_request, method, url, request_headers)
  File "/env/lib/python3.6/site-packages/google/auth/credentials.py", line 122, in before_request
    self.refresh(request)
  File "/env/lib/python3.6/site-packages/google/auth/compute_engine/credentials.py", line 102, in refresh
    six.raise_from(new_exc, caught_exc)
  File "<string>", line 3, in raise_from
google.auth.exceptions.RefreshError: HTTPConnectionPool(host='metadata.google.internal', port=80): Max retries exceeded with url: /computeMetadata/v1/instance/service-accounts/myst-ai-crystal@appspot.gserviceaccount.com/?recursive=true (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9964d74080>: Failed to establish a new connection: [Errno 111] Connection refused',))

[JustinBeckwith]

Greetings! Would you mind opening a new issue? It makes tracking these discussions much easier.