Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DefaultCredentialsProvider caches failure for flaky Compute Engine credential lookup #109

Closed
lukecwik opened this issue Jun 15, 2017 · 1 comment
Assignees
Labels
🚨 This issue needs some love. triage me I really want to be triaged.

Comments

@lukecwik
Copy link

Looking up application default credentials on a GCE VM can fail due to VM metadata server being unavailable during VM launch. This is a rare event but Google Cloud Dataflow customers hit this rare case one or two times a month due to the sheer number of VMs. GCE attempted to mitigate VM metadata server unavailability but were only able to reduce it be an order of magnitude thus we need support from the client to retry. Additionally, when contacting the GCE VM metadata server, we should be using the fixed IP address avoiding the nameserver lookup (another potential point of failure).

Problem area in the code:
https://github.com/google/google-auth-library-java/blob/b94f8e4d02bf6917af2e2f7ef8d7114a51dbcfa8/oauth2_http/java/com/google/auth/oauth2/DefaultCredentialsProvider.java#L261

Note that the code in this library and the Apiary auth support code are very similar. The fix was done within the Apiary auth code (note the use of the static IP address and also the presence of a fixed number of retries):
https://github.com/google/google-api-java-client/blob/4fc8c099d9db5646770868cc1bc9a33c9225b3c7/google-api-client/src/main/java/com/google/api/client/googleapis/auth/oauth2/OAuth2Utils.java#L74

It turned out that the fixes resulted in zero future customer contacts about this issue.

tcoffee-google added a commit to tcoffee-google/google-auth-library-java that referenced this issue Jun 26, 2017
Adapt fix from Apiary client libraries. Currently untested.

Addresses googleapis#109.
tcoffee-google added a commit that referenced this issue Jun 30, 2017
Adapt fix from Apiary client libraries. Addresses #109.
@JustinBeckwith JustinBeckwith added triage me I really want to be triaged. 🚨 This issue needs some love. labels Jun 8, 2018
@chingor13
Copy link
Contributor

Original issue fixed in #110

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🚨 This issue needs some love. triage me I really want to be triaged.
Projects
None yet
Development

No branches or pull requests

3 participants