Don't use ADCs if an API key is specified

[valenmoore]

I have been trying to use the Palm API and the palm.chat() function with google's new generative api. I've been in a maze of documentation and errors and I can't seem to get past this one. My code is very simple, and the error is coming from a simple request with palm.chat(). I have an API key that works when I test it with curl. I also downloaded credentials. I set up an OAuth consent screen, because I thought that might help me add the scope that I need, but I can't see what the scope requirement would be for palm.chat. Here is my code:

import google.generativeai as palm
import os
palm.configure(api_key='XXXXXXXXXXXXXXXXXXXXX')

os.environ['GOOGLE_APPLICATION_CREDENTIALS']='XXXXXXXXX/.config/gcloud/application_default_credentials.json'

response = palm.chat(messages='Hello')

response.last

The exact error I am getting is:

File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/google/api_core/grpc_helpers.py", line 67, in error_remapped_callable raise exceptions.from_grpc_error(exc) from exc google.api_core.exceptions.PermissionDenied: 403 Request had insufficient authentication scopes. [reason: "ACCESS_TOKEN_SCOPE_INSUFFICIENT" domain: "googleapis.com" metadata { key: "method" value: "google.ai.generativelanguage.v1beta2.TextService.GenerateText" } metadata { key: "service" value: "generativelanguage.googleapis.com" }

I think the problem is that I need to add some kind of scope to oauth but there is no documentation anywhere that I can find that says what that might be. I've posted this on google and stack overflow but no one has had a solution, so any help at all would be greatly appreciated. thank you so much!

[markmcd]

Can you just stick with the API key and ignore the ADCs? API key is the easiest approach if you can use it.

If you must use ADCs, you can use gcloud to add the scopes you need (but it'd be helpful for us if you could reply with some info on why API key isn't sufficient):

gcloud auth application-default login --scopes="https://www.googleapis.com/auth/generative-language,https://www.googleapis.com/auth/cloud-platform"

But don't mix the two - either use API key to auth (and remove the environmental references to the ADCs) or use the ADCs (and don't set api_key=...)

There's some more detail in this comment.

[valenmoore]

Okay I tried not setting ADCs, just using an API key. I followed the documentation letter for letter. My API key works perfectly when I run the following curl script:

curl -H 'Content-Type: application/json'
-d '{"prompt": {text: "Give me five subcategories of jazz"} }'
"https://generativelanguage.googleapis.com/v1beta2/models/text-bison-001:generateText?key=${XXXXXXXXXXXXXX}"

but when I run this code (straight from the documentation):

import google.generativeai as palm
import os
os.environ['API_KEY'] = "xxxxxxxxxxxxxxxxxxxxxxxxxxx"
palm.configure(api_key=os.environ['API_KEY'])

response = palm.generate_text(prompt="The opposite of hot is")
print(response.result) # 'cold.'

I still get that same error message about token scopes. What should I try? Thank you!

[valenmoore]

Also, I tried adding the scopes that you linked in the ADC gcloud code to the project with oauth, but it said they were not valid. I don't know if I am doing something wrong there or what but yeah.

[markmcd]

The client libraries try to use the ADCs up if they are present in the environment - they don't have to be explicitly set. You can remove them using gcloud (log out), delete the files, or just pop the environment var:

import os
# Hide ADCs from Google's API client library
os.environ.pop('GOOGLE_APPLICATION_CREDENTIALS', None)

Maybe there's something we can do better here to avoid the ADCs if an API key is set - I'll leave this open as a FR to "don't use ADCs if an API key is specified".

[valenmoore]

Used this code and it still did not work, so I started fresh with a new project. Just used API key, no ADC stuff. Same story, the API key worked with the curl function but I still got the same error when I ran the code. Seems like I've tried everything here so I'm not sure where to go next. Thanks anyways.

[valenmoore]

Okay I fixed it... somehow. Basically, I just switched to a different computer, did the exact same thing, and it worked first try. I suppose that means it was something wrong with the python version or the pip install? I'm not really sure, but I'm not going to stress too hard about it. Thank you for your help.

[zongsforce]

Okay I fixed it... somehow. Basically, I just switched to a different computer, did the exact same thing, and it worked first try. I suppose that means it was something wrong with the python version or the pip install? I'm not really sure, but I'm not going to stress too hard about it. Thank you for your help.

The situation I encountered is similar to yours. I encountered the same error when using virtualenv, but when I switched to pyenv with the same code, it worked fine. In addition, you don't even need to install Google Cloud CLI since you have the API key. Thank you for your inspiration.

[MarkDaoust]

I think some of the misunderstanding is because GOOGLE_APPLICATION_CREDENTIALS aren't always necessary.

When I've used ADC with the client libraries I've never set GOOGLE_APPLICATION_CREDENTIALS, but I did have to set it when trying to use the Node.js client libraries.

gcloud auth application-default login is probably putting the file in a standard location, and the client library is picking it up from there. So I think it's normal that the env-var makes no difference.

But trying it out:

1. It's easy to trigger the 403 Request had insufficient authentication scopes. error.
2. But if I pass an API-key, that takes precedence, and it works fine.

So I think this is working fine.

From @hankp46's #51 he said he fixed the same error by pinning the versions of a bunch of the required libraries.

I think that's a clue.

API-KEY support is somewhat new, and and it's possible older versions of one of the helpers libs is causing the problem. Since pip is conservative about upgrading packages, you probably had an old version installed, and pip just kept that? That would explain @zongsforce's observations as well since pyenv installs a whole new python, it would have picked up new versions of the packages?

The culprit package is not google-auth, 1.24 fails hard on some other error (too old) and 1.25 passes.

@vayvaychicken @zongsforce : Can either of you post the output of pip freeze from an environment where it's failing like this?

[JosephRivera517]

When I run the app in my local using the API Key, it works perfectly well. But when I run it using github, I got this error: 403 Request had insufficient authentication scopes.

[MarkDaoust]

Hi, I didn't change anything to fix this, but API-keys are rellatively new in google APIs.

I remember when it used to fail like this, but it isn't for me now. Maybe this was fixed lower in the stack.

Do you still have this problem?

[github-actions]

Marking this issue as stale since it has been open for 14 days with no activity. This issue will be closed if no further activity occurs.