Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Discussion] Storage and locking #33

Open
theacodes opened this issue Oct 18, 2016 · 13 comments
Open

[Discussion] Storage and locking #33

theacodes opened this issue Oct 18, 2016 · 13 comments
Assignees
Milestone

Comments

@theacodes
Copy link
Member

@theacodes theacodes commented Oct 18, 2016

(post-1.0.0 discussion of implementing credential storage and locking)

/cc @bshaffer

@theacodes theacodes added this to the 2.0.0 milestone Oct 18, 2016
@theacodes theacodes self-assigned this Oct 18, 2016
@theacodes theacodes mentioned this issue Oct 21, 2016
12 of 12 tasks complete
@Mackey22

This comment has been minimized.

Copy link

@Mackey22 Mackey22 commented Aug 4, 2017

@jonparrott I need storage capability for a project I'm working on that needs to be code-complete by the end of next week (8/11/2017). https://github.com/GoogleCloudPlatform/google-auth-library-python/pull/165/files says that oauth2client will still be functional for the foreseeable future - I just want to make sure that's still the case before going ahead and using that. Thanks

@theacodes

This comment has been minimized.

Copy link
Member Author

@theacodes theacodes commented Aug 4, 2017

that is indeed the case. oauth2client is deprecated but we have no plans to remove it.

@amit-am

This comment has been minimized.

Copy link

@amit-am amit-am commented Oct 4, 2017

Does this mean, "google-api-python-client" would be replaced by something else ?
Is "google-api-python-client" library under development ? Will there be support as and when Google-Cloud has more features adding to its services & that incorporating into "google-api-python-client" library ?

@theacodes

This comment has been minimized.

Copy link
Member Author

@theacodes theacodes commented Oct 6, 2017

Does this mean, "google-api-python-client" would be replaced by something else ?

Eventually yes, but in the near-term no.

Is "google-api-python-client" library under development ?

It is in maintenance mode - we will fix bugs but we will not be adding any new features.

@amit-am

This comment has been minimized.

Copy link

@amit-am amit-am commented Oct 9, 2017

Eventually yes, but in the near-term no.

My current code infra depends on this library. Should i consider migrating to another library which would be there long term ? Which is the other library that i should think of moving to ?

It is in maintenance mode - we will fix bugs but we will not be adding any new features.

I've been using google-api-python-client library & it's not thread-safe. So the library that you'll mention in answer to above question will be thread-safe ?

@theacodes

This comment has been minimized.

Copy link
Member Author

@theacodes theacodes commented Oct 9, 2017

My current code infra depends on this library. Should i consider migrating to another library which would be there long term ? Which is the other library that i should think of moving to ?

We will continue to address issues in google-api-python-client until we formally deprecate it and then a year after that. We do not have any plans to do right now or any timeline. For cloud APIs, many of the Cloud Client Libraries are now GA and we suggest using those for new projects.

I've been using google-api-python-client library & it's not thread-safe. So the library that you'll mention in answer to above question will be thread-safe ?

The Cloud Client Libraries are thread-safe.

While I appreciate these questions and I'm happy to answer, this is the wrong project and issue to ask. I'd suggest filing issues on https://github.com/google/google-api-python-client.

@LambdaDriver

This comment has been minimized.

Copy link

@LambdaDriver LambdaDriver commented Jul 8, 2018

I have begun using this library on a "serverless" project where I'm making use of Google services and APIs in programs that exist within ephemeral containers. Storage would be helpful as I'm concerned there may be thousands of access tokens for a refresh token at any given time, eventually hitting a limit.

I'm not super familiar with oauth flows and architecture, but here are a few things that come to my mind:

  • If all clients try to refresh the token at the same time we run into a thundering herd situation
  • Ideally would only work with the central store when the token is near expiration or invalid
  • High-level storage approach should work with anything from a relational DB with row locking to an eventually-consistant key-value store

Perhaps something along these lines?

if expiry - random(5-10 minutes) < Now()
	db = get info from database

	if db.expiry < random expiry
		assume someone else got there first and update credentials from this database data
	else
		open transaction/aquire lock (if applicable)
		refresh token
		update database and release lock/commit

I think this is a happy compromise between collisions, load on the database, and load on the authentication server. The 5 to 10 minutes I pulled out of the air, with the lower bound being your choice of 5 minutes clock skew (PS I agree with that choice as it's what MIT kerberos and Microsoft AD Kerberos by default... a lot of people are used to that number). I'm sure we could put a bit more rigor into finding the upper bound.

I do like the idea of a more generic interface for storage, however, if this is going to be part of the first-run experience working with Google APIs on one of the most popular beginner languages, perhaps there should be some batteries included for redis and certainly for cloud memory store. Let people like me contribute the AWS plugins.

Thoughts?

@sduskis

This comment has been minimized.

Copy link

@sduskis sduskis commented Dec 6, 2018

@theacodes, is this issue still relevant for the google-auth-library-python repo?

@theacodes

This comment has been minimized.

Copy link
Member Author

@theacodes theacodes commented Dec 6, 2018

@ei-grad

This comment has been minimized.

Copy link

@ei-grad ei-grad commented May 22, 2019

It doesn't look like a big deal to implement the storage outside of google.auth, but it is a bit unclear how to store the credentials object (which fields to store? there is no universal interface to serialize/deserialaze it... one can use the pickle, or hardcode the token/expiry fields and use json, but...) and how to integrate the storage with their refresh cycle.

@theacodes

This comment has been minimized.

Copy link
Member Author

@theacodes theacodes commented May 22, 2019

@ei-grad

This comment has been minimized.

Copy link

@ei-grad ei-grad commented May 22, 2019

Thanks! The evil Flow.credentials property wrapper dissembled them from me in the IPython shell, ha-ha :-D.

I'll just put it here for clearness :-).

import datetime
import google.auth
import google.oauth2
import google_auth_oauthlib.flow

flow = google_auth_oauthlib.flow.InstalledAppFlow.from_client_secrets_file( 
    'client_secret.json',
    scopes=["https://www.googleapis.com/auth/userinfo.email",
            "https://www.googleapis.com/auth/userinfo.profile"],
) 
flow.run_local_server()  # or flow.run_console()
session = flow.authorized_session()
print(session.get('https://www.googleapis.com/userinfo/v2/me').json())

creds = flow.credentials

info = { 
    'token': creds.token,
    'refresh_token': creds.refresh_token,
    'token_uri': creds.token_uri,
    'client_id': creds.client_id,
    'client_secret': creds.client_secret,
    'scopes': creds.scopes,
    'expiry': creds.expiry.isoformat(),
}

# implement your storage logic here, e.g. just good old json.dump() / json.load()

creds = google.oauth2.credentials.Credentials( 
    token=info['token'],
    refresh_token=info['refresh_token'],
    token_uri=info['token_uri'],
    client_id=info['client_id'],
    client_secret=info['client_secret'],
    scopes=info['scopes'],
)
creds.expiry = datetime.datetime.fromisoformat(info['expiry'])

if creds.expired:
    # don't forget to dump one more time after the refresh
    # also, some file-locking routines wouldn't be needless
    creds.refresh(google.auth.transport.requests.Request())

Any remarks?

@jmdobry

This comment has been minimized.

Copy link

@jmdobry jmdobry commented Jun 7, 2019

Test comment (plz ignore, will delete in a bit)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
9 participants
You can’t perform that action at this time.