Expose ability to refresh access token

[myitcv]

I am using omniauth-oauth2 with the Google strategy

My use case is roughly as follows:

• I have an initialiser with client id and secret etc, requesting offline access - entirely standard
• I store the returned access_token, refresh_token along with the uid etc in a user model
• Whenever I request data on behalf of said user, I pull the access_token from the corresponding attribute on the user model
• But this access token can (and does) expire. At a later stage (possibly when the user themselves is offline) I need to refresh their access token because it has expired

As far as I can tell, there is no omniauth* provided means of achieving a refresh.

What's the best way of achieving this? Is there something within omniauth that can be exposed to make this easier? Something along the lines of this

At the moment I have a rather unattractive helper method that utilises the same ENV variables to extract the client id, secret etc, takes a user object and performs a refresh using an approach not dissimilar to this

Any thoughts?

[myitcv]

Copying in @zquestz

[zquestz]

This is built into the Google API Client for Ruby. Check out https://github.com/google/google-api-ruby-client

The methods are documented in this file: https://github.com/google/google-api-ruby-client/blob/master/lib/google/api_client.rb

[ajsharp]

+1

[myitcv]

@zquestz thanks - I'm now exploring the best ways to incorporate omniauth and the Google API. Looks like there isn't a plug-and-play Signet (Google API's underlying OAuth implementation) integration with Rails.... unless I've missed something?

@ajsharp was that a +1 for my question or the answer?

[ajsharp]

@myitcv the question :)

[pauldacus]

I am having the same problem, and have long been looking for an answer. The solution suggested by zquestz in this file: https://github.com/google/google-api-ruby-client/blob/master/lib/google/api_client.rb

look like this is the relevant section in "def execute":

result = request.send(connection)
if result.status == 401 && request.authorization.respond_to?(:refresh_token) && auto_refresh_token
  begin
    logger.debug("Attempting refresh of access token & retry of request")
    request.authorization.fetch_access_token!
    result = request.send(connection, true)
  rescue Signet::AuthorizationError
    # Ignore since we want the original error
  end

It appears that if there is a 401, that the Signet client is supposed to execute "fetch_access_token!". If you look at the Signet docs (https://code.google.com/p/oauth-signet/wiki/SignetOAuth2Client#Method%3A_Signet%3A%3AOAuth2%3A%3AClient%23initialize), this doesn't seem to be the right method call.

There is an "update_token!" call that seems more appropriate. But either way, when my access token expires in the project I am working on, the above code to auto-refresh the access token doesn't work.

Does anyone have working code snippets on making this refresh of the access token actually work? It would be way cool if it could somehow be documented somewhere definitively. This issue has precious little docs out in the World.

[zquestz]

I am going to work on this a bit over the next few days and add a solution to the gem. Stay tuned.

[pauldacus]

I wondered how this is going, zquestz?

I have been investigating this problem, trying to figure out some of the above code. What I am really confused about is the fact that there is a check to see if the Signet client will "respond_to?(:refresh_token)", but then in the body of the method, instead of actually calling "refresh_token", it calls "fetch_access_token!". This doesn't make a lot of sense.

The reason I see it as important that there be a method that will actually issue a new valid access token via the "refresh token" method, instead of issuing access tokens via calls to "fetch_access_token!", is in the google docs (https://developers.google.com/accounts/docs/OAuth2WebServer):

"Note that there are limits on the number of refresh tokens that will be issued; one limit per client/user combination, and another per user across all clients. You should save refresh tokens in long-term storage and continue to use them as long as they remain valid. If your application requests too many refresh tokens, it may run into these limits, in which case older refresh tokens will stop working."

So the "fetch_access_token!" method issues new refresh tokens repeatedly, something Google is trying to discourage, and may take action to limit, if you cross their vaguely defined limits.

But what I have found is that when I can get a command line version of the Google client working (https://code.google.com/p/google-api-ruby-client/ - bottom of the page), if I issue "refresh_token!" calls, the access token appears unchanged. I include params for :client_secret, :client_id, :refresh_type, and :refresh_token. These appear to be the variables needed by Google, which you can see if you use the Oauth2 playground to get a refresh token, REST-style:

POST /o/oauth2/token HTTP/1.1
Host: accounts.google.com
Content-length: 163
content-type: application/x-www-form-urlencoded
user-agent: google-oauth-playground
client_secret=************&grant_type=refresh_token&refresh_token=XXXXXXXXXXXXXXXXXXX_DfMkPbUfeB_8eIefO5-7MGs3PU&client_id=XXXXXXXXXXXXXXX2.apps.googleusercontent.com

The docs on update_token! are confusing at best: https://code.google.com/p/oauth-signet/wiki/SignetOAuth2Client#Method%3A_Signet%3A%3AOAuth2%3A%3AClient%23update_token%21

If you look at the update_token! method, you can see there are params in the docs that are ignored in the "example":

client.update!(
  :refresh_token => 'n4E9O119d',
  :access_token => 'FJQbwq9',
  :expires_in => 3600
)

DOCS:

• :id_token —
  The current ID token for this client.
• :expires_in —
  The time in seconds until access token expiration.
• :issued_at —
  The timestamp that the token was issued at.

It is unclear whether "id_token" or "issued_at" are required or not.

Now... I can also say that when I issue a call to "fetch_access_token!" on the command line, that the access token IS refreshed. Again, this seems really suboptimal given the covert threats by the Googler to punish excess API calls. There seems to be a real need to get the ability to refresh access tokens working.

Sorry if this is too much detail. But I have been trying to figure out this whole Google Oauth2 Refresh Token Issue forever. The docs and StackOverflow info is scarce & vague at best.

Anyway, thanks zquestz for your gem, and any work you can do on this thread/issue.

[pauldacus]

Hmm... now I am a little confused. I waited until my command like Google::APIClient had expired (1 hr), and tried to issue "execute" commands, which failed as expected. So I attempted to run "update_token!", which appears to run just find (no errors), but the access token in the Signet client appears to be unchanged, so it seems as if it doesn't actually do anything to update the access token. However, I then tried to execute method calls (listing my Google cal events), and the call succeeded, to my surprise. So I looked at the access token again, and it had changed.

So just a heads up, at least in the command line, there is some sort of caching going on that requires a API call to somehow expire.

I wish there was a way of manually "expiring" the Signet client, this doing 1 test per hour business is tedious. Is there?

[myitcv]

As an update on where I got to with this.

My solution sadly does not involve ominauth but rather builds on top of signet

My solution is signet-rails:

Signet::Rails - a basic Rails wrapper around the Signet gem that handles persistence of a user's credentials on top of handling the auth flow within Rails applications

test-signet-rails gives an example of then how to utilise the credentials retrieved by signet-rails. Here is a snippet of the code one would write in the controller to then use these credentials via google-api-client (this will incidentally soon be wrapped up in a much neater google-api-client-rails with a factory for APIClient instances):

class HomeController < ApplicationController
  def index
    if logged_in?
      auth = Signet::Rails::Factory.create_from_env :google, request.env
      client = Google::APIClient.new
      client.authorization = auth
      service = client.discovered_api('calendar', 'v3')
      @result = client.execute(
        :api_method => service.calendar_list.list,
        :parameters => {},
        :headers => {'Content-Type' => 'application/json'}
      )
    end
  end
end

Benefits being:

• Persistence of credentials automatically handled
• Refresh of tokens is also automatically handled, and updated access tokens are automatically persisted
• Lightweight-omniauth-esque integration into a Rails app
• Credentials management is entirely separated from API calls - for example, omniauth calls userinfo.profile. signet-rails persists only the bare essentials, leaving you to do the rest. For example within your sessions controller you might well do something like this:

class SessionsController < ApplicationController
  def create
    user = request.env['signet.google.persistence_obj']
    session[:user_id] = user.id
    flash[:notice] = 'Signed In!'

    # update the user details
    auth = Signet::Rails::Factory.create_from_env :google, request.env
    client = Google::APIClient.new
    client.authorization = auth
    result = auth.fetch_protected_resource({uri: 'https://www.googleapis.com/oauth2/v1/userinfo?alt=json'})
    userinfo = JSON.parse(result.body)
    user.name = userinfo['name']
    user.email = userinfo['email']
    user.save

    # redirect to the homepage
    redirect_to root_url
  end

  def destroy
    session[:user_id] = nil
    flash[:notice] = 'Signed Out'
    redirect_to root_url
  end
end

Of course the pattern could well be adapted for use in omniauth - but then maybe persistence support belongs in another package?

Would welcome thoughts on the approach.

[pauldacus]

Not to muddy the waters...

But I am experiencing some intermittently odd behavior. When I use the ruby gem from the command line, I get regular, predictable behavior; with a fresh access_token my API calls succeed. As I noted above, if it has been just past the 1 hour expiration, and I make a call to the Google API, this also works. But, after leaving my terminal open for the night, I came back to work the next day, and made API calls, and these failed due to "Invalid Credentials".

Needless to say, this is frustrating to develop with. I don't think this is the fault of the gem. I see this on the OAuth2 Playground page as well, which I am under the impression makes direct REST calls to google. I have experienced this type of failure very reliably on very stale access tokens, and have found that the attempts to refresh the access tokens fails as well.

FYI Summary: Fresh access tokens, and slightly stale (1+ hrs) access tokens work, but my terminal has an access token about 27 hrs old, and I get errors. Maybe something to consider when coding a workaround. I wish google had more information on this sort of thing.

[pauldacus]

I have to say that trying to get the Google OAuth2 authentication mechanism to work reliably has been one of the most frustrating and confusing coding tasks I have ever undertaken.

The behavior of command line tools ("google-api irb" on command line), has different behavior from tools such as the OAuth2 playground. When the google-api command line tool issues API calls for "slightly" expired access tokens, they will succeed. I have issued many successful API calls for clients whose $client.authorization.expired? response was true (eg: it was expired), and I could also plainly see that the token expiration had past. These same calls in the Oauth2 Playground will always fail if the token is expired. I understand there have been "convenience" workarounds done in the code so that if the tokens have expired, it will attempt to fetch another token (not refresh, which is also strange). Finally, I have made API calls with "very" expired tokens (~27 hrs past expiration), and these do no succeed at all. I must reinstantiate a client with client_id & client_secret, and so forth.

I am using omniauth in rails, and am finding that intermittently (of course) that I will be returned an auth hash that does Not have a refresh token. It is just not there. So I go to the Google auth revocation page (https://accounts.google.com/IssuedAuthSubTokens), and revoke access for my app, and... viola, on the next request, I get a refresh token... but only for 1 request!!! Then on the next request after that... no refresh token again.

This is a coding domain (full scale rails app integrating with Google web services) with just enough confusion vectors (OAuth2 itself, Omniauth, omniauth-google-oauth2, Signet, Google REST web services, devise,...) combined with the intermittent (and delayed) nature of failures that makes coding in this area extremely difficult.

If I EVER figure it out completely, I'll write a book. Or put a comprehensive example online. All I ask is that if someone reading this has somehow gotten a fairly comprehensive & reliable example of a rails app that uses google web services that you PLEASE post a link to it. The only example of an "app" in ruby that uses google web services is actually what looks like Sinatra, and there is almost no explanation of what is going on or why.

I feel like sending this to Ryan at railscasts.... he seems to have some burnout and hints it may be from lack of good material. This problem is a Goldmine. Someone... PLEASE document all this. I would, but I am still profoundly confused.

[yellowman]

I'm going to document this all in my new book about Edward Snowden and the NSA's infiltration of the Ruby community.

[ajsharp]

Hi @pauldacus. Man, I know where you're at, and it sucks.

Let me say that I'm slightly behind on this thread, so if some of my comments are behind, outdated or whatever, take it with a grain of salt.

So, I also find the google ruby sdk extremely awkward, and dare I say, over-engineered. Also, let me be clear that the app I'm testing this with is in production yet; that is to say, I haven't encountered anything unexpected in my testing.

So basically, Google issues pretty short-lived tokens. Kind of a PITA, but either way you have to deal with expiration / refresh. I've written some code that handles it, and it seems to be working really well in my testing (again, pre-production). Sure, sucks that I have to deal with this in app code, and it'd be great if it eventually became a non-concern due to library code handling it, but this is what I'm working with currently to refresh tokens:

class Authorization
  # ...
  def refresh!
    return true if !expired?
    conn = Faraday.new('https://accounts.google.com') do |conn|
      conn.request  :url_encoded
      conn.response :json
      conn.response :raise_error
      conn.response :logger unless Rails.env.production?
      conn.adapter  Faraday.default_adapter
    end
    response = conn.post('/o/oauth2/token', {
      grant_type:    'refresh_token',
      refresh_token: refresh_token,
      client_id:     AppConfig::Google.client_id,
      client_secret: AppConfig::Google.client_secret
    })

    body = response.body

    self.access_token = body['access_token'] if body['access_token']
    self.expires_in   = body['expires_in']
    save!
  end
end

Then, I have a (fairly poorly designed) class that serves as a wrapper for the google ruby sdk. Here's the method that all outbound requests run through, thus ensuring the access token is valid

class GoogleClient
  DEFAULT_OPTIONS = {
    application_name: 'Octocall',
    application_version: '1',
    auto_refresh_token: true,
    authorization: :oauth_2
  }

  def initialize(authorization)
    @auth     = authorization
    @retrying = false

    @client = Google::APIClient.new(DEFAULT_OPTIONS)
    setup_client_auth
  end

  # I included this method in this code snippet to provide an example of how the `do_request` method is used.
  def get_calendar(id)
    do_request do
      @client.execute(
        api_method: resources[:cal].calendars.get,
        parameters: {calendarId: id}
      )
    end
  end

  private
  # Request wrapper. Returns data when successful. Raises exceptions when appropriate.
  #
  # Handles automatically refreshing the access token when a 401 is returned.
  # Raises an exception on 404.
  # Returns the payload on success
  def do_request(&block)
    if auth.expired?
      refresh_token
    end

    res = yield
    case res.status
    when 200, 201 # success
      res.data
    when 400
      raise BadRequestError, res
    when 401
      if !@retrying
        @retrying = true
        refresh_token
        do_request(&block) # retry the request
      else
        raise "Already attempted retry. Auth token=#{auth.id} is invalid."
      end
    when 404
      raise ResourceNotFoundError, res
    when 500
      raise UnexpectedAPIError, res
    else
      raise UnexpectedAPIError, res
    end
  end

  def setup_client_auth
    @client.authorization.access_token  = auth.access_token
    @client.authorization.refresh_token = auth.refresh_token
  end

  def refresh_token
    auth.refresh!
    setup_client_auth
  end
end

[pauldacus]

@ajsharp I am actually disappointed in my own inability to understand what is going on than anything else. I am really appreciative of what people like @zquestz & others have done to write awesome stuff that people like me can use... I certainly could not have done it myself.

If I have a rant, it is about the lack of documentation about getting working code going when there is such a myriad number of causes. I think people who have been doing OAuth coding for awhile, especially in ruby, may have forgotten how difficult it really is to code when there are so many interacting pieces, and failures could be coming from anywhere in the stack, and if you aren't very well acquainted with all of it, debugging is just hell.

I have just spent months trying to get reliable access to the Google API, and I feel like I am close to a solution. I may well try to write some docs that may help someone along, and turn a multi-month frustrating experience, into something much shorter & more enjoyable.

I will have to look at your code (above) tomorrow... it's Fri afternoon on what has been a particularly challenging week. I need to take a little break! :-)

[myitcv]

@pauldacus @ajsharp - I feel your pain, because I've been their in the past. Reading back through your various comments, and sprinkling a few of my own comments/thoughts:

• Firstly it's worth pointing out that there are different scenarios where one can use OAuth2. I'm assuming from the discussion above that we're all talking about the scenario for web server applications which is documented here. Note the OAuth2 Playground is an example of a web server application - command line applications I think fall under the installed applications scenario. Again the discussion that follows assumes the web server application scenario only
• As you have pointed out, access_token's and refresh_token's are different beasts. A refresh_token is issued if and only if the access_type parameter is set to offline
• A refresh_token will only be issued once, at the time the user authorises the web application for offline access. It is the responsibility of the web application to store this refresh_token against said user for use at a future date
• You can force a refresh_token to be issued by setting the approval_prompt parameter to force - or (as you point out @pauldacus) by revoking access to the application and reauthorising
• At the time a refresh_token is issued an access_token is also issued. An access_token is your means of using APIs but is short lived. It will expire. The refresh_token is used to get a fresh access_token
• access_token's should also be stored, along with expiration information to avoid making unnecessary calls to get new access_token's (your point above @pauldacus)
• When an API call via google-api-ruby-client is made with an expired access_token, the google-api-ruby-client gem will attempt to refresh the access_token using the refresh_token if auto_refresh_token is set. Note this all presupposes that the request.authorization is correctly set (this is a type of Signet::OAuth2::Client for OAuth2). It is then the responsibility of the application to store the updated access_token
• The complication comes when you try to link omniauth and google-api-ruby-client (and by implication signet). Using the refresh_token, access_token etc returned by omniauth one then has to instantiate and populate a Signet::OAuth2::Client instance before you can start accessing APIs. And then you have to handle the storing of credentials (both types of tokens, user IDs etc). It starts to get really messy
• Given the previous point I, and as I mention above, I stopped trying to integrate omniauth with google-api-ruby-client and instead wrapped signet to create signet-rails. signet-rails is a basic Rails wrapper around the Signet gem that handles persistence of a user's credentials on top of handling the auth flow within Rails applications, as well as seamlessly handling the refresh and persistence of access_token's
• test-signet-rails gives an example of how to use signet-rails - it's designed to be super simple so that the only code required is one setup and the following (example where API call is made from a controller):

# app/controllers/home_controller.rb

class HomeController < ApplicationController
  def index
    if logged_in?
      auth = Signet::Rails::Factory.create_from_env :google, request.env
      client = Google::APIClient.new
      client.authorization = auth
      service = client.discovered_api('calendar', 'v3')
      @result = client.execute(
        :api_method => service.calendar_list.list,
        :parameters => {},
        :headers => {'Content-Type' => 'application/json'}
      )
    end
  end
end

I actually plan to make this simpler still by creating google-api-ruby-client-rails but this seems like a pretty good start.

Sorry, this turned out to be quite a long response. But hopefully some of the points are useful.

[zquestz]

Thanks for the detailed info @myitcv!

[pauldacus]

Very cool @myitcv , and thanks for the explanation.

I'm curious how you are persisting the access_token & refresh_token?

I have the problem of storing multiple access/refresh tokens from multiple users, from multiple providers. This has necessitated some pretty convoluted code.

" It starts to get really messy"

Yes, the understatement of the day :-)

I have a before_filter action that first makes sure the user has any authorization at all for the provider, whether expired or not (auth_google). If not, they are redirected to the omniauth authorize path for the provider. I put the API client builder code that you have above in "index" into a application controller method with a before filter also (set_google_api_client). This code looks like this:

    def auth_google
        if current_user
            redirect_to user_omniauth_authorize_path(:google_oauth2) unless current_user.google_authorization? || request.path =~ /^\/users\/auth\/google_oauth2/
        end
    end

    def set_google_api_client
        if current_user
            client = Google::APIClient.new
            goog_auth = current_user.google_authorization
            client.authorization.access_token = goog_auth.access_token
            client.authorization.refresh_token = goog_auth.refresh_token
            client.authorization.scope = CALENDAR_SCOPE
            client.authorization.client_id = CLIENT_ID
            client.authorization.client_secret = CLIENT_SECRET
            client.authorization.redirect_uri = OAUTH2_REDIRECT
            client.authorization.code = goog_auth.auth_code
            @api_client = client
            redirect_to user_omniauth_authorize_path(:google_oauth2) if @api_client.authorization.expired?
            @calendar = @api_client.discovered_api('calendar', 'v3')
        end
    end 

What I have found universally to be true, is that if the access_token goes "extremely" stale (over 1 day old), the attempt to generate another with the refresh token will not work. I have found this to be the case whether on my web app, in the command line, the OAuth2 Playground, or googles own "Reference" area (https://developers.google.com/google-apps/calendar/v3/reference/events).

As Jerry Seinfeld would say, "That's what was so vexing."

We are supposed to keep the refresh token, which is supposed to "Never" expire, but it certainly does expire. I have found that there is only one solution, and that is to go back to square one, and just re-authorize from scratch, in all instances. I have never found a way to make this work.

Yes, very vexing.

[pauldacus]

As an illustration of what I'm talking about, I have left a page on the OAuth2 Playground open for quite awhile now, and attempting a request yields this:

GET /calendar/v3/calendars/primary/events HTTP/1.1
Host: www.googleapis.com
Content-length: 0
Authorization: Bearer ya29.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
HTTP/1.1 401 Unauthorized
Content-length: 249
X-xss-protection: 1; mode=block
X-content-type-options: nosniff
Expires: Tue, 06 Aug 2013 18:27:20 GMT
Server: GSE
Cache-control: private, max-age=0
Date: Tue, 06 Aug 2013 18:27:20 GMT
X-frame-options: SAMEORIGIN
Content-type: application/json; charset=UTF-8
Www-authenticate: Bearer realm="https://www.google.com/accounts/AuthSubRequest", error=invalid_token
{
 "error": {
  "errors": [
   {
    "domain": "global",
    "reason": "authError",
    "message": "Invalid Credentials",
    "locationType": "header",
    "location": "Authorization"
   }
  ],
  "code": 401,
  "message": "Invalid Credentials"
 }
}

Attempting to refresh the access token gets this:

POST /o/oauth2/token HTTP/1.1
Host: accounts.google.com
Content-length: 163
content-type: application/x-www-form-urlencoded
user-agent: google-oauth-playground
client_secret=************&grant_type=refresh_token&refresh_token=xxxxxxxxxxxxltec0fZ7JMa1SHA2as2MIciGX6MWHhk&client_id=xxxxxxxxs.googleusercontent.com
HTTP/1.1 400 Bad Request
Content-length: 31
X-xss-protection: 1; mode=block
X-content-type-options: nosniff
X-google-cache-control: remote-fetch
-content-encoding: gzip
Server: GSE
Via: HTTP/1.1 GWA
Pragma: no-cache
Cache-control: no-cache, no-store, max-age=0, must-revalidate
Date: Tue, 06 Aug 2013 18:31:05 GMT
X-frame-options: SAMEORIGIN
Content-type: application/json
Expires: Fri, 01 Jan 1990 00:00:00 GMT
{
  "error" : "invalid_grant"
}

Trying to "Exchange Authorization Code for Token", gets this:

Request / Response
POST /o/oauth2/token HTTP/1.1
Host: accounts.google.com
Content-length: 250
content-type: application/x-www-form-urlencoded
user-agent: google-oauth-playground
code=xxxxxxxxxxxxxxxzJEgCf.wu8r19ujMwQcEnp6UAPFm0FGZHmggAI&redirect_uri=https%3A%2F%2Fdevelopers.google.com%2Foauthplayground&client_id=xxxxxxxxxxxxxxxxxxxgleusercontent.com&scope=&client_secret=************&grant_type=authorization_code
HTTP/1.1 400 Bad Request
Content-length: 31
X-xss-protection: 1; mode=block
X-content-type-options: nosniff
X-google-cache-control: remote-fetch
-content-encoding: gzip
Server: GSE
Via: HTTP/1.1 GWA
Pragma: no-cache
Cache-control: no-cache, no-store, max-age=0, must-revalidate
Date: Tue, 06 Aug 2013 18:32:10 GMT
X-frame-options: SAMEORIGIN
Content-type: application/json
Expires: Fri, 01 Jan 1990 00:00:00 GMT
{
  "error" : "invalid_grant"
}

These are the only real avenues to trying to refresh the access token, without going back to the beginning & just re-authorizing from scratch & getting brand new access & refresh tokens. This makes it essentially impossible to comply with Google's request that you rarely if ever request new refresh tokens.

[myitcv]

@pauldacus - strange, because I just used the OAuth2 Playground and was able to successfully refresh an access token using the 'refresh access token' button provided. Does that not work for you?

Indeed in my experience using signet-rails and google-api-ruby-client I have been successfully using refresh_token's

So it seems like there is something not quite right in your setup, or at least your mapping from omniauth to google-api-ruby-client (incidentally the mapping code you included above is exactly what I expected to see)?

How am I storing my tokens? I'm using a simple strategy pattern to abstract from how the tokens are stored - for now I am using a very basic ActiveRecord strategy. With signet-rails the storage however happens under the hood, down in the rack layers that sit below Rails.

What is universally true in my experience is that the error messages returned by Google are not at all helpful when something goes wrong. This has always been my biggest pain. Indeed it often requires a painstaking comparison between my HTTP requests and those of the OAuth2 Playground to figure out where I am/was going wrong.

[pauldacus]

@myitcv Oh, it absolutely DOES work when the access token is valid, or "slightly" expired ( I can't really define that...). But leave your browser open to the OAuth2 Playground overnight... then try to refresh the token. It almost certainly will not work.

Herein lies the difficulty: Google WILL refresh a "slightly" expired access token (minutes or hours, maybe)... but not a "very" expired token (days). What does this mean exactly? I don't know. Due to the nature of the problem, it is extremely difficult to define/reproduce/etc.

I just know that after a weekend of having the OAuth2 Playground page open, the "Refresh Token" function has never worked for me.

[myitcv]

@pauldacus - hmm, this doesn't seem to add up. Because the attempt to get a 'fresh' access_token relies only on the refresh_token and not at all on pre-existing (some which may be expired) access_token's

Are you sure the overnight issue is nothing to do with your session to OAuth2 Playground timing out?

[pauldacus]

@myitcv It may well be that the OAuth2 Playground site sets a cookie that expires, not sure about that. But the behavior happens it all contexts for me; web app, command line, Oauth2 Playground. It is pretty universal.

"Because the attempt to get a 'fresh' access_token relies only on the refresh_token and not at all on pre-existing (some which may be expired) access_token's"

Correct. And when I try this with refresh_tokens older than about 1 day, it doesn't work; the refresh attempt to get a valid access token does not work.

My 2nd copy/pastefrom the OAuth2 Playground above show me only using a refresh token. You can see from the response that it doesn't work.

And what actually has to happen, is I actually have to go back to square one, and "Select & Authorize API's". I have to start over completely, and wipe out any existing access or refresh tokens.

Maybe my choice of words is confusing, "fresh" vs "new". All attempts to get a "fresh" access token fail once the access & refresh tokens are more than about 1 day old, and I have to start over totally and get "new" access & refresh tokens.

Hopefully that is clearer.

[pauldacus]

"Are you sure the overnight issue is nothing to do with your session to OAuth2 Playground timing out?"

After thinking about this a little more, I'm not sure this would be the problem. The request to authorize the API actually does work fine, which you'd think would also break if it were some sort of session problem. And Google does return valid error codes, and it appears the requests are being issued just fine.

So I think it seems to be a Google problem. And as I said before, I experience this issue in all contexts, not just OAuth2 Playground.

[myitcv]

@pauldacus - if Google is only refreshing "slight expired" tokens as you suggest, the implication is that everyone would be experiencing OAuth2 issues. This aspect of Google is heavily in use and so I struggle to think that this lowly thread would be the first place to have picked it up!

I just tried to get a fresh access_token using a refresh_token that itself is almost 24 hours old. And this worked fine.

So it would seem to suggest that something in your setup/code is not as it should be. The OAuth2 Playground issue is I think a red herring.

[pauldacus]

Well, I agree that Google cal is in heavy use. But I have seen quite a bit of confusion over OAuth2 & Googles implementation of it.

I'm not posting doctored screenshots of OAuth2 failures. I'm not sure what you mean by "red herring". These are Googles own tools for posting REST calls to their service, and they are the results that I am seeing.

I'm not posting here for the purposes of misdirection. I am simply curious if anyone else is experiencing the problems I am having, and am wondering how they solved them, if so.

[myitcv]

@pauldacus - don't get me wrong, I'm not for one second suggesting you're doctoring your results! Or indeed trying to send us off on a wild goose chase! I totally sympathise with your situation, because I've had issues in the past.

Can you reproduce the problem with OAuth2 Playground? If so, can you list the steps here? (http requests/headers not needed)

[pauldacus]

"Can you reproduce the problem with OAuth2 Playground?"

Yes, it's on my screen right now, and it won't work. The browser tab has been open for a long time, over a week. But I have found that it really only matters how long it's been since the most recent access token refresh has taken place to replicate the result. Reproducing it simply involves that: Open the OAuth2 Playground, get it working.... then let it sit for 3 days, and try retrieving your calendar list. This should fail, of course, and you should be able to click "Refresh Access Token" under Step 2, and be good to go. But this does not happen, at least not for me.

Is the time period needed 3 days? I don't know. It's really impossible to sit & do tests at a browser that involve waiting for days to then test for failure or success... I do know I was running OAuth2 calendar list functions yesterday before I left work, and I returned this morning, went to this same browser tab, and refreshing the access token did not work.

What I am seeing is exactly what I pasted above. And the only way to access Google calendar again in the OAuth2 Playground, is to go to Step 1 (Select & Authorize APIs), select the calendar as the scope, and click the Authorize API's button to get some brand new access & refresh tokens.

I should say that I am not having "crippling" problems with Google OAuth2, it simply nagging problems with getting new access tokens when I know I should not have to. I have coded a workaround where the user simply re-authorizes & gets new access & refresh tokens when the Signet client is expired. It's just that this seems unnecessary and could possibly hit Googles rate limiting for refresh tokens.

@ajsharp - I would be interested in seeing or hearing about how you have coded how you refresh access tokens.

[myitcv]

@pauldacus - for what it's worth, I'm not doing any special coding in order to refresh my access_token's. I'm entirely relying on google-api-client-ruby for that, specifically this code (relevant snippet pasted):

module Google
  class APIClient
  # ...
  def execute(*params)
    # ...
    if result.status == 401 && request.authorization.respond_to?(:refresh_token) && auto_refresh_token
      begin
        logger.debug("Attempting refresh of access token & retry of request")
        request.authorization.fetch_access_token!
        result = request.send(connection, true)
      rescue Signet::AuthorizationError
         # Ignore since we want the original error
      end
    end
#...

So the behaviour I see when using an expired access_token is as follows:

1. I attempt to make my API call via google-api-client-ruby using an expired access_token
2. During this call (i.e. before control is returned to my code) a 401 error results, meaning the above code kicks in
3. Because I have auto_refresh_token set, google-api-client-ruby (via signet) tries to get a fresh access_token
4. If the attempt to get a fresh access_token succeeds, the original API call is retried
5. Control is then returned to my code either way, success or failure

The key here is that the Signet::OAuth2::Client instance that is set on your APIClient instance needs to be properly initialised.

My suggestion would be that if you're not seeing google-api-client-ruby auto-refresh your access tokens, debug whether this code above is getting hit.

The README I have created for test-signet-rails might also help as it gives a ground up example.

[pauldacus]

"This aspect of Google is heavily in use..."

What's funny, is I thought this to be the case as well, especially for ruby. But given the dearth of information about these issues (I feel I have looked just about everywhere), I'm starting to question this.

Most of the Google calendar "integrations" offered by apps, are not bi-directional. Usually, you just paste the Google cal iCal feed URL to read Google calendar data. They don't offer full read, write capability. Most will either push or pull from Google calendar. And the Google Calendar gem (https://github.com/northworld/google_calendar) appears to use Client Login authentication, which I think is deprecated or Google is discouraging its use.

I'm starting to wonder exactly how many people have actually built full read/write OAuth2 capable apps which authenticate with Google? If it is "a lot", it certainly doesn't seem so, given the amount of usable documentation.

[pruzicka]

Guys, thanks a lot for all above info, it's golden. I'm building web app for a accessing Google Spreadsheet and this all helped a lot (to at least understand things under the hood). Not that I have a solution however :(

[deeTEEcee]

while randomly searching how to setup google calendar connections, i found this code in someone else's repository. there seems to be many ways but this one seemed fairly understandable...

# client = some obj i created that stores the authentication-related info. 
  def refresh_current_token(client)
    oauth_client = OAuth2::Client.new(
      Rails.application.secrets.omniauth['google_calendar']['client_id'], Rails.application.secrets.omniauth['google_calendar']['secret'],
      :site => "https://accounts.google.com",
      :token_url => "/o/oauth2/token",
      :authorize_url => "/o/oauth2/auth")
    access_token = OAuth2::AccessToken.from_hash(oauth_client, {:refresh_token => client.refresh_token})
    access_token = access_token.refresh!
    client.access_token = access_token.token
    client.expires_at = Time.now + access_token.expires_in
    client.save
  end

[technic-tec]

Got this page through search, a bit confused. Link an answer here about the odd oauth2 playground issue for future reference: http://stackoverflow.com/questions/33771197/fail-to-refresh-access-token-in-oauth-2-0-playground

[thbar]

Sharing what I learned, and building up on @deeTEEcee answer (and others): if you have an existing strategy (like Freckle below which I'm building at the moment), you can leverage a bit more the strategy to build the client like this:

strategy = OmniAuth::Strategies::Freckle.new(nil, key, secret)
client = strategy.client

oauth2_token = data_integration.oauth2_token
oauth2_refresh_token = data_integration.oauth2_refresh_token

token = OAuth2::AccessToken.new client, oauth2_token, {refresh_token: oauth2_refresh_token}
new_token = token.refresh!

data_integration.oauth2_token = new_token.token unless new_token.token.blank?
data_integration.oauth2_refresh_token = new_token.refresh_token unless new_token.refresh_token.blank?
data_integration.oauth2_token_expires_at = Time.at(new_token.expires_at) unless new_token.expires_at.blank?
data_integration.save!

Hope this helps!

[rob-race]

Quick example on how to work with dynamic strategies:

provider = :bitbucket
strategy = OmniAuth::Strategies.const_get(provider.to_s.capitalize).new(
      nil,
      Figaro.env.send("#{provider.to_s}_key"),
      Figaro.env.send("#{provider.to_s}_secret")
    )

[tayloredwebsites]

I found this article helpful for refreshing access tokens from google:
https://stackoverflow.com/questions/12792326/how-do-i-refresh-my-google-oauth2-access-token-using-my-refresh-token#14491560