Intermittent error: request had invalid authentication credentials

[vieira]

If you are still having issues, please be sure to include as much information as possible:

Environment details

• OS: docker image: node:10-alpine / on gke cos
• Node.js version: v10.15.3
• yarn version: 1.16.0
• @google-cloud/common version: 1.0.0

Steps to reproduce

We are long polling a Pub/Sub REST endpoint and from time to time (about once a week or so) we get an error that seems to indicate that something went wrong with the refresh token for OAuth? We handle the exception by logging and returning an empty list, without any interaction the problem goes away on the next iteration.

Failed to pull messages from chats-rated-webhook { Error: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.

request({
      body: { returnImmediately, maxMessages },
      method: 'POST',
      uri: `subscriptions/${this.name}:pull`,
      json: true,
    })
      .catch((err) => {
        console.error(`Failed to pull messages from ${this.name}`, err);
        return { receivedMessages: [] };
      });

This is happening for some time now, before 1.0.0 but I cannot easily determine in which version it started to happen.

[vieira]

I am still experiencing this issue, any news? It usually happens a few consecutive times and then goes away on its own.

[JustinBeckwith]

Greetings! We get reports of these from time to time, and they are hard to track down. Are you directly using this library? Or the @google-cloud/pubsub npm module?

We suspect this has something to do with instability in the metadata server or networking stack inside of GKE used to serve up credentials. One way around that is to create your own service account, and put those secrets in the cluster. I'd really love to know if this happens when you're using a downloaded service account key file as opposed to the default metadata credentials.

[vieira]

Hello @JustinBeckwith,

Thanks for your response. I am using this library directly to implement a REST client to PubSub (@google-cloud/pubsub uses grpc, that we cannot use atm).

Although we are running on GKE, we use our own service account and don't rely on GKE to serve up the credentials.

However I have just noticed that in the last couple of weeks the only service where this keeps happening is using an older version of this library (i.e., before the "update dependency google-auth-library to v5"). I have just deployed a new version that includes the latest version of this library (and consequently the google-auth-library v5) and will monitor if it happens again.

Thanks again for taking the time to reply.

Have a great weekend.

[vieira]

unfortunately it is still happening even on the latest release. Let me know if there is anything I can do to help with pinpointing the problem

[aaleksandrov]

hey guys, I was investigating the similar issue for BigQuery googleapis/nodejs-bigquery#473 and my comment to it googleapis/nodejs-bigquery#473 (comment)
I think it's problem on Google OAuth2 api because when google-auth-library tries to refresh access token Google Oauth API keeps returning the same access token until it's really gets expired. So these 401's happen only as edge case when Google Oauth Api returns token which is valid for less than a second, then client makes a request to pubsub (or Bigquery in my case) and fais with 401 because token has just expired.

I added some logs in google-auth-library and what I got:

2019-08-13T23:48:03.111Z Current access token = "xxx", expires at 2019-08-13 23:49
2019-08-13T23:48:12.111Z Gooole auth library attempts refreshing a token ...
2019-08-13T23:48:91.111Z New token "xxx", expires at 2019-08-13 23:49         <---(SAME ONE, expiring in less than one second)
....
2019-08-13T23:49:14.111Z Making request to BigQuery with "xxx"
2019-08-13T23:49:89.111Z Error 401
....

Next request: 
....
2019-08-13T23:53:67.111Z Current access token = "xxx", expires at 2019-08-13 23:49
2019-08-13T23:53:99.111Z Gooole auth library attempts refreshing a token ...
2019-08-13T23:54:44.111Z New token "yyy", expires at 2019-08-14 00:54
....
2019-08-13T23:55:03.111Z Making request to BigQuery with "yyy"
2019-08-13T23:56:17.111Z Success

[JustinBeckwith]

Greetings folks! We're mostly sure we've resolved all the issues here. We found a bug in the underlying auth library (shout out to @bcoe) that dealt with token refresh issues. I'm going to close this out for now. If you're still running into this problem, please do let us know!

[ravi-ssi]

I'm receiving a similar error when posting a google chat message via a webhook. It works most of the time, but will start failing with the following error:

"error": {
"code": 401,
"message": "Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.",
"status": "UNAUTHENTICATED"
}

I'm not sure how an authentication failure would occur with a webhook. Are there limits on the number of messages?

[ravi-ssi]

Is there any

I'm receiving a similar error when posting a google chat message via a webhook. It works most of the time, but will start failing with the following error:

"error": {
"code": 401,
"message": "Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.",
"status": "UNAUTHENTICATED"
}

I'm not sure how an authentication failure would occur with a webhook. Are there limits on the number of messages?

Any update on this issue? It's causing major issues with our automated bot notifications and having a production impact.

[JustinBeckwith]

Greetings @ravi-ssi - this is entirely unrelated to this module. If you're having specific issues with auth and the chat API, could I trouble you to open an issue here, with the questions in the issue template?
https://github.com/googleapis/google-api-nodejs-client