Can't use with Application Default Credentials

[lostpebble]

Environment details

• OS: Windows
• Node.js version: 13.6.0
• npm version: 6.13.4
• @google-cloud/vision version: 1.9.0

Steps to reproduce

1. Have my gcloud shell all set up correctly for ADC
2. Try and use this library as I'm using all my other Google API Node.js Client libraries
3. Get an error:

7 PERMISSION_DENIED: Your application has authenticated using end user credentials from the Google Cloud SDK or Google Cloud Shell which are not supported by the vision.googleapis.com. We recommend configuring the billing/quota_project setting in gcloud or using a service account through the auth/impersonate_service_account setting. For more information about service accounts and how to use them in your application, see https://cloud.google.com/docs/authentication/.

Has this always been the case? I thought I'd made use of the library before with ADC.

[bcoe]

@lostpebble I'm not 100% when this became a requirement, I'll see if I can find someone to dig into this for you.

Are you unblocked for the time being did you find these instructions on creating a service account?

[lostpebble]

I have multiple deployment / dev environments - such as Cloud Functions, Kubernetes, and my local windows dev machine... Having to consider and configure environment variables for all these use-cases now is really not a great dev experience. I'm not even sure what the path structures look like in my deployment environments. This was quite a productivity blocker yesterday- I ended up deciding to move on and forget about this part of my app's development for now.

Its been years now that I've relied on simply ADC for all the Google Cloud products I use in my Node.js applications, so I would have hoped this would be the same.

[AlanGasperini]

Can you please list your Cloud SDK version? 'gcloud version'
We had a release of Cloud SDK that changed the ADC behavior and it could produce the behavior described, but if you update to the latest version and run 'gcloud auth application-default login' everything should work as before.

[lostpebble]

@AlanGasperini that's already what I did.

But I'll do it again to make sure:

C:\dev> gcloud version
Google Cloud SDK 282.0.0
alpha 2019.05.17
app-engine-java 1.9.78
app-engine-python 1.9.88
beta 2019.05.17
bq 2.0.54
cloud-datastore-emulator 2.1.0
core 2020.02.21
docker-credential-gcr
gcloud
gsutil 4.47
Updates are available for some Cloud SDK components.  To install them,
please run:
  $ gcloud components update

Then I updated to latest:

Your current Cloud SDK version is: 282.0.0
You will be upgraded to version: 283.0.0
...
Update done!

Did the login auth stuff again:

C:\dev> gcloud auth application-default login
...
Credentials saved to file: [C:\Users\Paul\AppData\Roaming\gcloud\application_default_credentials.json]

These credentials will be used by any library that requests Application Default Credentials (ADC).

And then tried to run the API stuff again, but unfortunately the same error:

7 PERMISSION_DENIED: Your application has authenticated using end user credentials from the Google Cloud SDK or Google Cloud Shell which are not supported by the vision.googleapis.com. We recommend configuring the billing/quota_project setting in gcloud or using a service account through the auth/impersonate_service_account setting. For more information about service accounts and how to use them in your application, see https://cloud.google.com/docs/authentication/.

[AlanGasperini]

Can you please check if application_default_credentials.json has this field?
quota_project_id
That will help determine the nature of the error.

[lostpebble]

It currently has a structure like this:

{
  "client_id": xxx,
  "client_secret": xxx,
  "refresh_token": xxx,
  "type": "authorized_user"
}

So no quota_project_id to be seen

[lostpebble]

On that note, something weird was happening before.. I'm not sure exactly what changed it. I think it was before I re-did the gcloud auth application-default login the last time.

I was getting an error from @google-cloud/vision that basically said I hadn't enabled the API (I definitely had) and it had a link that sent me to the cloud console website to enable it - but the link was incorrect and had a weird "project id" in it that the console didn't know what to do with and just gave me an error.

[AlanGasperini]

There are three flows that are typically used for development:

ADC with the default Cloud SDK client id. That is what happens when you do 'gcloud auth application-default login'. These credentials use a global project for quota and billing. APIs that have usage based billing(such as vision) have been disabled, because the billing needs to be charged to a customer project.

ADC with a different client id. You can set this with --client-id-file flag, documented here: https://cloud.google.com/sdk/gcloud/reference/auth/application-default/login
This will use ADC, but APIs that bill on usage will charge the project associated with the given client id.

Service accounts. This will use service account credentials, and will bill against the project associated with the service account. Documented here: https://cloud.google.com/docs/authentication/getting-started

We have enabled a fourth flow recently, which uses ADC with default credentials, and the ability to specify a project for billing, using the --add-quota-project flag. This requires that you specify a project where you have permission to make API calls, and will use the active gcloud project as the billing project.

It sounds like you've always been using the first method(plain ADC) and perhaps it has broken recently. If you are calling vision APIs that have a cost per API call, then plain ADC will not work because GCP will need a customer project to charge for the calls.

[lostpebble]

@AlanGasperini ahh okay. Thanks for explaining.

I now ran the auth like so:

gcloud auth application-default login --add-quota-project

and my application_default_credentials.json looks like this:

{
  "client_id": xxx,
  "client_secret": xxx,
  "quota_project_id": xxx,
  "refresh_token": xxx,
  "type": "authorized_user"
}

Unfortunately the same error is persisting though.

7 PERMISSION_DENIED: Your application has authenticated using end user credentials from the Google Cloud SDK or Google Cloud Shell which are not supported by the vision.googleapis.com. We recommend configuring the billing/quota_project setting in gcloud or using a service account through the auth/impersonate_service_account setting. For more information about service accounts and how to use them in your application, see https://cloud.google.com/docs/authentication/.

I think something must have broken for @google-cloud/vision, or it isn't communicating properly with the billing stuff. I make use of multiple APIs which require billing and they still continue to run without issue.

[wvvinicius]

Hello devs, has anyone managed to solve these problems? The same thing happens to me. :(

[wvvinicius]

Devs, after generate new file a with a new key. I import the new file to folder of project and it worked.
Tks

[sofisl]

Hi there @lostpebble! After reproducing the issue successfully, I found that I was able to solve it by adding a flag to set the quota project, like so:

gcloud auth application-default set-quota-project QUOTA_PROJECT_ID

Would you mind giving this a try to see if it helps you?

[lostpebble]

Hi @sofisl ,

I did what you suggested and it didn't work on the version which I used when I originally opened this issue. But after updating the package to ^2.1.0, it seems all is working automatically again now.

Thanks!

[lostpebble]

Just thought I'd comment here to say, after trying this out a bit more lately- I noticed that my initial, first request after creating the client works fine. But every request after that fails with the Application Default Credentials error again. I need to create the client every time fresh for the requests to not fail, like so:

  const VisionApiClient = new vision.ImageAnnotatorClient();
  const visionResults = await VisionApiClient.annotateImage({ // ... });

This seems a bit wasteful, as surely a single instance of the client should be able to take care of all future requests?

[ollyde]

We have to implement Firebase for push notifications (extremely annoying and were forced by Google basically) and part of that process is implementing insecure service accounts into the NodeJS environment instead of using the one provided by the environment that is inside GCP, not Firebase.

Service account files should never be used in a production project, big security issue, they really need to fix their docs and Firebase so that it works with the account provided by the environment.

This also applies to some of their libraries that force you to use private keys during init... crazy.

[meredithslota]

Hi folks — generally, it seems like the issues raised in the thread have been resolved. There is updated documentation on using Application Default Credentials, both here: https://cloud.google.com/docs/authentication/application-default-credentials and https://cloud.google.com/docs/authentication/provide-credentials-adc If you're still seeing issues with this, can you please open a new issue describing the problem?

Re: the Firebase issue, I apologize but we're not able to address that in this repo — I recommend filing an issue in the relevant repo for the library you're using.

[ollyde]

@meredithslota it's not resolved as some libraries are still asking for insecure JSON files.
We should never ever be using JSON files to access Google Cloud environments, that's a security problem.

Really, what needs to happen is libraries like 'storage' or 'vision' need to upgrade to use the environments security parameters.

Right now in order to get round this issue we have to use the Secrets API and download the API key into memory on the container, which can also be a security issue.

A few years ago one of our services got hacked and the hackers spun up bitcoin minors because one of our developers used a JSON file to access the environment.

Not good.