Invalid use of service credentials return via 'google_default'

[asford]

Opening as a tracking issue for follow-up discussed #58. I will be able to follow-up on this issue the week of 2018-01-28. From that thread:

[Testing #58] on a GCE instance I use as a dev box. On this machine I've the compute sdk installed and I'm I'm using the service account for application default credentials:

fordas@salish:~/distributed_dev/gcsfs$ gcloud auth list
                  Credentialed Accounts
ACTIVE  ACCOUNT
*       353972334481-compute@developer.gserviceaccount.com

In this context auth2client.client.GoogleCredentials.get_application_default returns an token component that manages access to the service credentials and does not include a refresh_token, client_secret. It instead provides access tokens via the get_access_token interface, which handles token expiration and refresh. It may also be used to retrieve auth headers for requests.

This currently breaks the current master implementation, which expects a refresh token to be available. This will also break this implementation.

After setting application default credentials to a user account via gcloud auth application-default login auth2client.client.GoogleCredentials.get_application_default returns a GoogleCredentials object exposing refresh_token et al. This credentials object is capable of handling access token and auth header generation.

Given that behavior, my current recommendation would be to hold the credential object returned by the get_application_default and use the interface credential.get_access_token().access_token to manage access. gcsfs would then not be responsible for managing auth token management and refresh. It would also be desirable to wrap the device credentials generated via the browser auth mode in a the same oauth handler and remove the token management logic currently implemented in gcsfs.

In a broader sense, I would suggest emphasizing use of the default app credentials instead of obtaining and directly caching a credential object for gcsfs. In "off cloud" contexts, the user can then manage authorization via the gcloud auth application-default command, which provides login, revoke. In "on cloud" this then provides easy access to the default service account, with the option for a user to override this setting by gcloud auth login.

[martindurant]

I think this should be easier to implement now, as gcs.method stored the manner of auth that was originally performed, gcs.token stores an arbitrary object - in this case the google credential - and refresh() can call the appropriate action depending on the type of auth. Only the headers attribute is actually used in any of the calling machinery.

As I mentioned elsewhere, the expiry of credentials should be checked on every _call(), which could be a pass-through to the google code, or check the expiry date if it is a dict like previously. Both are no-ops in the case of non-expiry.

[asford]

My recommendation would be that any further development focus on moving to google-auth (see #60) and the google.auth.transport.requests support module. This module provides support for all credential types requires for gcsfs including Application Default, Service Account, User, and Anonymous. This will then provide a standard implementation handling authorization token retrieval, token refresh and request authorization.

[martindurant]

Yes, I see what you mean, it seems that Google have upped their game somewhat recently. I suggest that you do the minimum required to work with service account credentials for now, and then concentrate on the prefix/delimiter object listings, before returning to this.

[asford]

Primarily as a note to any interested parties, using the token="cloud" authorization mode is not affected by this issue. Using this authorization mode for all on-cloud deployments, potentially using the default workaround outlined in #54, is an acceptable short-term workaround.