Getting invalid_grant after about a week

[sandorvasas]

Version 4.4.3

We deployed our pre-tested Oauth2 app to prod (pre-tested for a couple of days), and about a week later, we started getting invalid_grants when trying to refresh the token. Not sure where and why it goes wrong. For the sake of simplicity, before every xero request, an ensureAccessTokenValidity function is ran and a CRON job 2 times a week to re-new the refresh token. I'm aware refresh tokens are valid for 30 days, but to be secure and account for any CRON job outages, we try to renew it 8-9 times a month.

This is the function which is called before every Xero request:

  async ensureAccessTokenValidity(region: InstanceType<RegionSchema>) {
    const checkAndTryToRefresh = async () => {
      const xa = region.xeroAccount;
      try {
        this.Log.debug("ACCESS TOKEN SET:", JSON.stringify(xa.getXeroAccessTokenSet(), null, 4));
        const tokenSet = xa.getXeroAccessTokenSet();
        if (!tokenSet) throw new Error("Xero is not connected.");

        if (xa.getXeroAccessTokenSet().expired()) {
          this.Log.info("tokenset expired");
        }
        await xero.setTokenSet(xa.getXeroAccessTokenSet())
      
        await xero.accountingApi.getAccounts(region.xeroAccount.xeroTenantId)

        return [true, xa.getXeroAccessTokenSet()]
      }
      catch (e) {
        if (e.response && e.response.statusCode && e.response.body) {
          const {body} = e.response;
          const tokenExpiredError = body.Status == 401 && body.Detail.includes("TokenExpired");
          const authUnsuccessfulError = body.Status == 403 && body.Detail.includes("AuthenticationUnsuccessful")

          if (tokenExpiredError || authUnsuccessfulError) {
            const tokenSet = await xero.refreshToken();
            xa.setXeroAccessTokenSet(tokenSet)
            await RegionModel.updateOne({ _id: region._id }, {
              'xeroAccount.xeroAccessToken': xa.xeroAccessToken
            })
            await xero.setTokenSet(tokenSet);
            return [true, tokenSet]
          }
        } 
        return [false, e]
      }
    }

    const maxTries = 3;
    let tries = 0;
    do {
      const result = await checkAndTryToRefresh();
      if (result[0] == true) {
        return result[0];
      } else 
      if (tries == maxTries - 1) {
        throw result[1];
      } else {
        await new Promise(resolve => setTimeout(resolve, 2000));
      }
      tries++;
    } while (1)
  }

I just added the retry logic, so the invalid_grants happened without it. Now, since the invalid_grant is quite indescriptive, I'm not sure what's going on. This same function is ran by the CRON too, so basically I try to xero.accountingApi.getAccounts(region.xeroAccount.xeroTenantId) and if it fails, refresh the token.

Can it be because of Xero & our server's time mismatch? Haven't checked for that, but I read it happens with Google OAuth2. What possible reasons can there be?

[SerKnight]

Thanks for sharing this code back with us. It is helpful in understanding how our community is using xero-node in the wild!

Question - is this for every token refresh or just a subset?

As for the invalid_grants, we've yet to be able to reproduce in a consistent format. If you can do so in a shareable environment ( either in https://github.com/XeroAPI/xero-node-oauth2-ts-starter or https://github.com/XeroAPI/xero-node-oauth2-app ? ) then I can put some time into diagnosing.

But from other peoples feedback and the error return it could be a few things.. Most common response i'd seen is that server times mismatch but that could be a red-herring rabbit hole

Here is a great post around possible answers to the issue ( https://blog.timekit.io/google-oauth-invalid-grant-nightmare-and-how-to-fix-it-9f4efaf1da35 )

In their case it was because the user had removed access manually ( this would be from the Xero settings page ) Is this happening with all tokens or a select few orgs? Might be worth looking for a pattern in that someone could have manually revoked access.

[sandorvasas]

Thanks for the tips. I'll try to give you a reproducible example if everything else fails.

Manual revocation is out of the picture. Most definitely hasn't been revoked.

Re timezones, our server is located in Sydney (Lightsail), and just now I configured NTP to use the NZ servers. ntpstat shows a ~200ms delay. Can a delay of 70-200ms be causing it? What NTP servers do your API use?

[sandorvasas]

I see you use openid-client in the lib. May I ask what openid lib do you use on server side? I might go and surf through its code to find possible reasons for invalid_grants.

[sandorvasas]

Can this be because I'm making the request from the server and the Origin header of the request is not set to be the callback url's domain?
i.e. our callback url is app.xyz.com/callback whereas the API server tries to renew the token from api.xyz.com

[SerKnight]

Hmm thats a good thought.. However if you have an expired access_token and valid refresh_token ( in your tokenSet ) it should not matter what domain you are refreshing it from. I will have to validate this though. Were the refresh tokens you are using still valid ( 60 days https://developer.xero.com/faq/oauth2/oauth-refresh-expiry - though this was 30 days last month )

[sandorvasas]

Yes, they were valid. Started happening only after a few days, then clients now almost daily have to reconnect. Created a standalone app and set it up on its own server (planned to share its code with you when we start receiving invalid_grants but that hasn't happened yet). It uses the same code, stripped down to just periodically calling getAccounts() and trying to renew the token, and that's been working fine for almost 2 weeks now. There are a couple more areas which I need to check tho

[sandorvasas]

Well, we could not solve this issue, but -maybe it's early to say after a couple of days- we found a workaround. Instead of letting the access token expire and trying to renew it when the user makes a request, I applied a brute-force method: CRON job that runs every 5 mins updates the token whether it is expired or not. Been connected for 3-4 days.

[marcschregardus]

We've also experienced the same issue (and have a similar brute force method to refresh the token - although it's initiated by the user when they view this particular problem). It would be good to resolve though (even though it's difficult to reproduce), as we've been experiencing this issue in prod for the last 6 weeks or so (it's not a great look for the users of the system).

[RettBehrens]

Closing this since we're unable to replicate and there hasn't been further communication since April. Please reopen if this remains an issue.

[gurubobnz]

Hi there, early days but I suspect we're running into this also ... I will see what I can find on the situation but we definitely had a valid token this morning and then later in the day we got back an invalid_grant.