Cache token / transport confusion

[billmccord]

Hi, I'm having a really hard time figuring out how to accomplish what I want to do. Essentially, I would like to cache the access and refresh tokens so that I don't have to ask the user to authenticate every time I want to create a new client. It seems like the Transport may have been added for this purpose, but an example would make things much clearer. Thanks for the awesome library and any help you can provide!

[billmccord]

After digging into the code a bit more, it seems like perhaps when I'm finished using an OAuth2 Client and want to cache the token away for later reuse I could:

1. Get the Transport from the http.Client
2. Cast it to an oauth2.Transport
3. Get the TokenSource from the oauth2.Transport
4. Get the latest Token from the TokenSource
5. Save the Token
   Is this a recommended approach or is there an easier way to do this by injecting some kind of callback function that gets notified when a Token is refreshed?

[jfcote87]

You could also just cache the Token from created from the Config.Exchange() func so that you only cache Token once because the refresh token never expires.

[billmccord]

Every time I refresh tokens on the OAuth library I'm using I get a new
refresh token. Is that not standard?
On Jan 31, 2015 1:48 AM, "Jim Cote" notifications@github.com wrote:

You could also just cache the Token from created from the
Config.Exchange() func so that you only cache Token once because the
refresh token never expires.

—
Reply to this email directly or view it on GitHub
#84 (comment).

[jfcote87]

Token.RefreshToken stays the same. Token.AccessToken changes.

See if the following gist explains it any better.

https://gist.github.com/jfcote87/89eca3032cd5f9705ba3

[billmccord]

Jim, thanks for posting the gist. That was helpful, but unless I'm misunderstanding something I think there are still cases where the token could be lost. Consider the following and let me know if this is a possible scenario:

1. I retrieve the token from my datastore and, while it is valid when I check it, it only has a few milliseconds left before expiration.
2. I create the new Client with the token.
3. As I'm using the Client, the token expires.
4. The RoundTrip method on the Client's Transport attempt to get the Token.
5. The reuseTokenSource determines that the token is invalid.
6. Now, the tokenRefresher's Token method is invoked.
7. retrieveToken is called and fetches a new token.
8. The Token is completely replaced (new AccessToken, RefreshToken, and Expiry.)

In this scenario, I would have no way of knowing the token was modified unless I followed my process above of recaching the token or, at the very least, verifying that the token didn't change while I was using the Client. This is because the Client automatically refreshes the Token for you as a convenience and, unfortunately, doesn't tell you that it did it via a callback.

Does that make sense or am I still missing something?

[jfcote87]

The RefreshToken does not change. so you don't have to worry about the background refresh. The tokenRefresher struct stores the RefreshToken in the oldToken field and this field is not updated again.
https://github.com/golang/oauth2/blob/master/oauth2.go#L221-L227

*note that the RefreshToken field of the returned Token pointer is updated by the passed refresh token. The tokenRefresher is not updated by the returned Token.
https://github.com/golang/oauth2/blob/master/oauth2.go#L345-L347

[billmccord]

Jim, unfortunately, this does not seem to be the case. I built a test based on the tests included with oauth2 here:
https://gist.github.com/billmccord/4247b0c4d2a6b5a4d09f

You can see from my gist that if the token expires after being used to create an oauth2.Client, then another request will be made in the background to obtain a new token (new access_token and new refresh_token) before proceeding with the request.

Unfortunately, at this point the refresh_token is lost because the ORIGINAL_REFRESH_TOKEN is no longer valid after the NEW_REFRESH_TOKEN is generated according to the OAuth2 spec.

Therefore, it is possible that the token that I stored per your suggestion will become invalid and the only way I can see to ensure that I have the right tokens is to always get the Token again from the Client after use and ensure that a) it hasn't changed from what I have stored or b) update what I have stored with the changed Token I got from the Client.

[jfcote87]

I think I see where we might be talking past each other and that an actual bug exists. I tested code using Google's client apis and their implementation of oauth2. According to Google's oauth documentation , refresh tokens do not expire but must be revoked. In my tests, a new refresh token is only generated during a code exchange (i.e. Config.Exchange() ). In a token refresh call (tokenRefresher.Token() ) the refresh_token field is omitted/left blank. This is allowed in the oauth2 spec as sending a new refresh_token during a refresh operation is optional (oauth2 doc section 1.5), not required.

You have found a bug. If a new refresh_token is generated during a refresh, the token source refresh token is not updated. See test code for an example.

Back to your original question of how to cache a Token each time it is refreshed. A hook would need to be added to Config and ReuseTokenSource().

[billmccord]

Thanks for validating this. The OAuth provider I'm calling uses a library that always generates a new refresh_token and there isn't an option to not do this yet. Since it is optional, I agree, that a hook would be necessary to get the updated Token in cases where it is renewed. Since that hook doesn't exist yet, I suppose that my workaround of getting the Token from the client when I'm finished using it is probably the only option?

e.g.
// --- Obtain token from storage. ---
// --- Create and use client with stored token. ---
// Get latest token from client.
newToken, err := client.Transport.(*oauth2.Transport).Source.Token()
// --- Store revised token (if changed.) ---

It isn't pretty, but it seems like a reasonable workaround for now.

[rakyll]

Does the provider return a different refresh token each time you ask for a new access token?

[billmccord]

Yes. You can see here that issue_refresh_token is hard-coded to true for the refresh_token grant_type:
https://github.com/FriendsOfSymfony/oauth2-php/blob/b0e57e17c84175a51af01cef7bbb2961261c84ad/lib/OAuth2.php#L840-L842

[rakyll]

Their API is not complaint with the OAuth 2.0 spec, and we have no intention to support provider-specific non-spec features.

[billmccord]

Two points:

1. I'm not clear on how it isn't compliant with the OAuth 2.0 spec when the spec specifically says in section 1.5 (emphasis, mine):
   " (H) The authorization server authenticates the client and validates
   the refresh token, and if valid issues a new access token (and
   optionally, a new refresh token)."
2. Haven't you already?
   https://github.com/golang/oauth2/blob/master/oauth2.go#L387-L395

[jfcote87]

https://tools.ietf.org/html/rfc6749#section-6
Section 6. Refreshing an Access Token
(last paragraph, emphasis in original)

The authorization server MAY issue a new refresh token, in which case
the client MUST discard the old refresh token and replace it with the
new refresh token. The authorization server MAY revoke the old
refresh token after issuing a new refresh token to the client. If a
new refresh token is issued, the refresh token scope MUST be
identical to that of the refresh token included by the client in the
request.

https://tools.ietf.org/html/rfc6749#section-10.4
Section 10.4. Refresh Tokens
4th paragraph

For example, the authorization server could employ refresh token
rotation in which a new refresh token is issued with every access
token refresh response. The previous refresh token is invalidated
but retained by the authorization server. If a refresh token is
compromised and subsequently used by both the attacker and the
legitimate client, one of them will present an invalidated refresh
token, which will inform the authorization server of the breach.

[billmccord]

Jim, thanks, that seems to validate that the bug lies with this project because it doesn't provide a mechanism for obtaining the new refresh_token in situations where the refresh_token is replaced during the Refreshing an Access Token operation, correct?

If so, I can file a close this out and create a more specific issue OR we can just use this to track.

[jfcote87]

Bill,
I have a CL ready that fixes the refresh token problem and will upload later today.

@rakyll are you ok with calling this a bug? Should we start a new issue to discuss caching difficulties?

[adg]

@jfcote87 It looks like a bug to me. I think we should start a new issue to discuss caching strategies.

[mgenov]

@adg is there open issue about caching strategies?

[adg]

Not that I know of.

On 4 May 2015 at 04:53, Miroslav Genov notifications@github.com wrote:

@adg https://github.com/adg is there open issue about caching
strategies?

—
Reply to this email directly or view it on GitHub
#84 (comment).

[johnl]

For reference, there used to be support for token caching but it was removed for some reason: 93ad3f4

[abourget]

So is this fixed, then ?

[rakyll]

@johnl, the trivial cache implementation is removed because we were not able to come up with a generic and useful cacher interface. I think a wrapper cacher RoundTripper is the perfect solution although it is not documented anywhere.

[gebv]

If the token has been restored, i can access a resource. But token is not auto-refreshed. Why?

sourceToken := oauth2.ReuseTokenSource(nil, &fileTokenSource{accessToken, refreshToken, expiry})

client := &http.Client{
    Transport: &oauth2.Transport{
        Base:   ContextTransport(oauth2.NoContext), // from internal/transport.go
        Source: oauth2.ReuseTokenSource(nil, sourceToken),
    },
}

client.Get("...")
// access the resource
// long time
// token expired
client.Get("...")
// and not auto-refreshed

[gebv]

When your mind is fresh... no problems
Working code

sourceToken := oauth2.ReuseTokenSource(nil, &fileTokenSource{accessToken, refreshToken, expiry})
t, _ := sourceToken.Token()
client = conf.Client(oauth2.NoContext, t)
client.Get("...")
// token is auto-updated

[lewgun]

@rakyll I think a wrapper cacher RoundTripper is the perfect solution although it is not documented anywhere. is it here now ?

[arvenil]

Is there currently any workaround for this problem?

[arvenil]

Ok, I see bug with refreshing token got fixed: cc2494a right?

However, original question remains - what's currently proper way to save and restore token? Is there any kind of event when token get's changed? Any example how to achieve that?

[lpar]

@ibudiallo:

Before I attempt to make a web request and let the client automatically update the token (which I cannot retrieve), I check the expiry date myself:

While this is admirably simple, I realized it unfortunately has a failure mode:

1. Your code to get a client is called.
2. It fetches the cached token from the DB, and notes that the token hasn't expired and can be reused.
3. Your code passes back a client to the caller, set to re-use the cached token.
4. The caller makes a sequence of OAuth2 requests.
5. Between two of those requests, the token expires.
6. The client fetches a new token, and the server chooses to respond with a new refresh token as well and invalidate the previous one (which it's allowed to do, as per the spec).
7. Your code never finds out about the new tokens, so they don't get written to cache.
8. Next time your code is called to get a client, it returns a client that tries to use an invalid access token with an invalid refresh token, and fails.

[gouthampc]

@lpar in the same solution mentioned by @ibudiallo, how about saving the token with a TTL with little less time (say 5 minutes) than the expiry time, this way we're refreshing the token ahead of time.

ctx := context.Background() // reuse your context
token := fetchedFromDB()
conf := &oauth2.Config{}
if (token == nil) { // TTL has passed, so let's fetch the token update 
   src := conf.TokenSource(ctx, token)
   newToken, err := src.Token() // this actually goes and renews the tokens
   if err != nil {
     panic(err)
   }
   if newToken.AccessToken != token.AccessToken {
     ttl := token.Expiry.Sub(time.Now()) - 5 * time.Minute // TTL 5 minutes before actual token expiry.
     SaveTokenWithTTL(newToken, ttl) // back to the database with new access and refresh token
     token = newToken
   }
}
client := config.Client(ctx, token)

[blueforesticarus]

I came here to say, yes, this is very confusing.

It took a whole hour of searching just to figure out that CacheFile used to be supported but was removed. It is strange this isnt a solved issue since I expect it is a common usecase, I'd think that most of the time application developers would want to cache the token. Maybe there is something I don't understand about oauth2, but if so it is not obvious from this thread or this repo.

My understanding is:

1. the token issued in the authentication flow expires, after which it can be used to get a new token
2. this library does the refresh transparently under the hood. When the server say the token expired, the library gets a new one, and completes the request
3. once this happens, ie. the expired token is used to get a new one, the old token becomes invalid. When the new token expires, IT will be used to get a new one. You cannot use the same token for a refresh more than once.
4. Therefore if you write code to save the token, and the library refreshes the token later on, your saved token becomes useless. You need to save the new token.

Most of the solutions here seem to ignore the 4th point, or like @gouthampc above, require extra time based plumbing.
The best solutions here seems to be from @j0hnsmith and @bgentry, although those are far from self explanatory.

I think either something needs to be added to the library to facilitate having a CacheFile that updates when the token is refreshed, or at least an example needs to be written for how to do it with existing plumbing.

[himby]

What about utilizing "golang.org/x/sync/singleflight" to prevent unnecessary renewal of access tokens?
In my case I'm using a memory based cache to store tokens. keyed with a sessionId. Then I use the sessionId combined with existingToken.Expiry.Unix() as key in the singleflight query. It is important that the requestGroup is a pointer set from a global scope. Seems promising, but not very well tested.

...
if existingToken.Expiry.Before(time.Now().UTC()) {
    src := o.Config.TokenSource(o.ctx, existingToken)
    key := sessionId + strconv.FormatInt(existingToken.Expiry.Unix(), 10)
    res, err, shared := requestGroup.Do(key, func() (interface{}, error) {
        newToken, err := src.Token() // this actually goes and renews the tokens
        if err != nil {
            log.Debug(err)
        }
        
        if newToken != nil {
            if newToken.AccessToken != existingToken.AccessToken {
	        existingToken = newToken
            }
        }
        ..
})
...

[ellulpatrick]

@lpar , @ibudiallo - You could set the Token.Expiry to 0 to disable the "auto-refresh", as per the documentation here: https://pkg.go.dev/golang.org/x/oauth2#Token

This means that then you can manage refreshing manually yourself. It defeats the purpose of auto-refresh functionality though. It really should be simpler than this, to save a refreshed token.

[KoduIsGreat]

I've read this thread, multiple times and am still confused about what, if any of these solutions actually work, or if there is a more documented way to solve this problem.

[kke]

Agree with @KoduIsGreat and @blueforesticarus - I've been trying to understand and implement this or to find a project using this package that would have a clean token cache implementation but haven't found one.

[sonmaximum]

I also wound up here looking for documentation on how to solve this problem (caching/saving tokens when refresh token is invalidated/updated upon use) and was surprised to find there doesn't seem to be a clear answer.

It would be helpful for users to have some clarity on this.

[manojmalik20]

I tried using the method suggested by @ibudiallo above and I'm getting this error from the Token method -

Post "": unsupported protocol scheme ""

Does anyone know the reason behind this and how to fix this?

[j0hnsmith]

@manojmalik20 usually that happens when you have a URI without http/https

[Blasikov]

@ibudiallo - You saved my bacon. Thank you.

My workflow's target system refresh token is always unlimited, so this is perfect.

Would have taken me ages, if ever, to understand this gem:

newToken, err := src.Token() // this actually goes and renews the tokens

[tonimelisma]

Am I to understand correctly that for 7 years no one has fixed this bug? Would a PR be accepted, or would it be better to fork the library? EDIT: I just realized there's over 70 open pull requests and over 110 open issues. So it looks like Google is no longer maintaining the official oauth2 library? Would you be open to handing over maintenance to volunteers who have more time?

[tonimelisma]

A simple way to cache refreshed token (for very simple use cases):

tokenSource := conf.TokenSource(oauth2.NoContext, token)
newToken, err := tokenSource.Token()
if err != nil {
    log.Fatalln(err)
}

if newToken.AccessToken != token.AccessToken {
    SaveToken(newToken)
    log.Println("Saved new token:", newToken.AccessToken)
}

client := oauth2.NewClient(oauth2.NoContext, tokenSource)
resp, err := client.Get(...)

This only works for the initial loading of the token. As I understand it, each time you call the client to make an API call it might update the token silently. So you have to make another check after each API call to see if the token changed.

[dnesting]

Something like this solution works for me:

type cachingTokenSource struct {
  base oauth2.TokenSource
  filename string
}

func (c *cachingTokenSource) saveToken(tok *oauth2.Token) error // save tok as JSON to c.filename
func (c *cachingTokenSource) loadToken() (*oauth2.Token, error) // load JSON token from c.filename

func (c *cachingTokenSource) Token() (tok *oauth2.Token, err error) {
  tok, _ = t.loadToken()
  if tok != nil && tok.IsValid() {
    return tok, nil
  }

  if tok, err = c.base.Token(); err != nil {
    return nil, err
  }

  err2 := c.saveToken(tok)
  if err2 != nil {
    log.Error("cache token:", err2)  // or return it
  }

  return tok, err
}

func NewCachingTokenSource(filename string, config *oauth2.Config, tok *oauth2.Token) oauth2.TokenSource {
  orig := config.TokenSource(context.Background(), tok)
  return oauth2.ReuseTokenSource(nil, &cachingTokenSource{
    filename: filename,
    base: orig,
  })
}

So long as this is the TokenSource you use everywhere, it's guaranteed to get the cache updated when a token refresh occurs, so it seems to address all of the requirements discussed in this thread.

[pmrt]

Something like this solution works for me:

type cachingTokenSource struct {
  base oauth2.TokenSource
  filename string
}

func (c *cachingTokenSource) saveToken(tok *oauth2.Token) error // save tok as JSON to c.filename
func (c *cachingTokenSource) loadToken() (*oauth2.Token, error) // load JSON token from c.filename

func (c *cachingTokenSource) Token() (tok *oauth2.Token, err error) {
  tok, _ = t.loadToken()
  if tok != nil && tok.IsValid() {
    return tok, nil
  }

  if tok, err = c.base.Token(); err != nil {
    return nil, err
  }

  err2 := c.saveToken(tok)
  if err2 != nil {
    log.Error("cache token:", err2)  // or return it
  }

  return tok, err
}

func NewCachingTokenSource(filename string, config *oauth2.Config, tok *oauth2.Token) oauth2.TokenSource {
  orig := config.TokenSource(context.Background(), tok)
  return oauth2.ReuseTokenSource(nil, &cachingTokenSource{
    filename: filename,
    base: orig,
  })
}

So long as this is the TokenSource you use everywhere, it's guaranteed to get the cache updated when a token refresh occurs, so it seems to address all of the requirements discussed in this thread.

Just to clarify, your workaround is to never use oauth2.Config.Client() (ie. use your own http client) and handle the refresh by your own, retrieving the token from cachingTokenSource.Token() before each request until the oauth2 http client library implements a token cache interface, right? Because if you later use oauth2.Config.Client() to make requests your tokens could be silently refreshed during the requests.

On the other hand, this is not thread safe so beware of race conditions in your files, if your storage layer is a cookie or a database this may not be a problem (the scope of cookies is per user-agent and databases generally handle race conditions well) but if you're writing to files as in your example, two different requests could try to write to the same file at the same time, so use singleflight or mutexes.

[dnesting]

your workaround is to never use oauth2.Config.Client() (ie. use your own http client)

Sort of. The implementation of oauth2/Config.Client() is trivial:

func (c *Config) Client(ctx context.Context, t *Token) *http.Client {
	return NewClient(ctx, c.TokenSource(ctx, t))
}

So if you wish to insert yourself in the process somewhere in order to cache the last token for reuse later, the TokenSource is the logical place to do that. To obtain an http.Client you then just need to oauth2.NewClient(ctx, myCachingTokenSource).

handle the refresh by your own

Sort of.

  orig := config.TokenSource(context.Background(), tok)
  return oauth2.ReuseTokenSource(nil, &cachingTokenSource{
    filename: filename,
    base: orig,
  })

What this code does is instantiate a TokenSource that will handle the refreshes automatically, and the logic implemented above will call it when needed if the cached token isn't valid. What I'm doing here is just inserting myself in between the existing TokenSource and the http.Client produced by oauth2.NewClient.

Probably one difference worth noting is that I'm using context.Background() instead of passing the original context around, so if you were relying on that context being preserved for the token refreshes you'd need to modify accordingly.

this is not thread safe so beware of race conditions in your files

It should be concurrency-safe. Because I wrap my caching token source with ReuseTokenSource, this ensures that calls to Token are serialized with a mutex. You would only have concurrency issues if you instantiate multiple caching token sources pointing to the same file.

[tonimelisma]

Thank you very much for these workarounds. However, they're so complex it seems to me like this library has significant architectural issues. It would be great to get maintainer views on whether they foresee anyone ever fixing things or even approving PRs.

[pmrt]

Noted, thank you so much @dnesting That was the part of the puzzle I was missing.

In my use case apart from updating the token pair in the database I also want to update the pair in an encrypted cookie, which are obviously per user-agent so my idea was instead of using a cacheTokenSource, to use something more like a notifyTokenSource that takes a callback that gets executed when the token is refreshed. Implementing it would be very similar to your workaround but I have doubts about the usage, take this pseudocode as an example where we would use it within an http handler:

type SomeHandler struct {
	db     *sql.DB
	config *oauth2.Config
}

func (h *SomeHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
	// recreate *oauth.Token from cookie
	tok := tokenFromCookie(r)
        // create a new tokenSource for our current request
	ts := NewNotifyTokenSource(h.config, tok, func(newTok *oauth2.Token) {
		// save new token in the db
		saveToken(newTok, db)
		// update the encrypted cookie
		saveCookie(newTok, w)
	})
	c := oauth2.NewClient(context.Background(), ts)
	
	// make requests with my client
	c.Do(...)
	// etc
}

Notice how here we would create an instance of tokenSource and http.Client per request because we have 1 token per 1 request in this case (also we use the same instance of oauth2.Config across requests).

My guess is that it is concurrent safe because we're using 1 instance per 1 request so it shouldn't even need mutex. And even if within a request we spawn multiple goroutines that try to use the same client with that token source, both the original config.TokingSource() and our custom tokenSource are returning a tokenRefresher wrapped with a reuseTokenSource which is guarded with mutex. From what you've mentioned this should be okay and safe to use, but I'm not sure, can someone confirm this please?

And yeah, I agree with @tonimelisma. We need an official and clean way to hook into this client which doesn't feel so hacky.

[tonimelisma]

Am I to understand correctly that for 7 years no one has fixed this bug? Would a PR be accepted, or would it be better to fork the library? EDIT: I just realized there's over 70 open pull requests and over 110 open issues. So it looks like Google is no longer maintaining the official oauth2 library? Would you be open to handing over maintenance to volunteers who have more time?

Bumping this. Looks like there isn't a popular fork of this library and creating one would be difficult as this one is maintained by Google, would the maintainers be willing to move ownership of this library to more active community maintainers who could approve PRs and fix some of the bugs?

Not sure who the right people are, but perhaps @quartzmo @codyoss @BigTailWolf @rolandshoemaker @dmitshur @bcmills could comment?

[andrewhowdencom]

(similar to @dnesting's approach): A caching token provider:

• Add storage backed token source andrewhowdencom/x40.link#63

I struggled with seeding the cache, which is why the cache instantiation is so long. The seed is also in the same project.