pandas_gbq.context.project ignored when using default credentials

[BigMammaPerf]

Setting credentials using google.auth.default() doesn't work as expected.

What I'm doing

Running the following service on Google App Engine

from flask import Flask
import pandas_gbq
import pandas
from google.auth import default as default_creds

credentials, project_id = default_creds()
pandas_gbq.context.credentials = credentials
pandas_gbq.context.project = project_id

app = Flask(__name__)
@app.route("/check_credentials_pandas")
def check_credentials_pandas():
    df = pd.DataFrame(columns = ['check', 'pandas'])
    df.to_gbq("xxx.check_pandas", chunksize=None, if_exists='append')  
    return {"message": 'managed to insert to bigquery with pandas', 'status': 'success'}, 200

if __name__ == "__main__":
    app.run(host='127.0.0.1', port=8080, debug=True)

Logs

Traceback (most recent call last): File "/env/lib/python3.7/site-packages/flask/app.py", line 2446, in wsgi_app response = self.full_dispatch_request() File "/env/lib/python3.7/site-packages/flask/app.py", line 1951, in full_dispatch_request rv = self.handle_user_exception(e) File "/env/lib/python3.7/site-packages/flask/app.py", line 1820, in handle_user_exception reraise(exc_type, exc_value, tb) File "/env/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise raise value File "/env/lib/python3.7/site-packages/flask/app.py", line 1949, in full_dispatch_request rv = self.dispatch_request() File "/env/lib/python3.7/site-packages/flask/app.py", line 1935, in dispatch_request return self.view_functions[rule.endpoint](**req.view_args) File "/srv/main.py", line 101, in check_credentials_pandas df.to_gbq("xxx.check_pandas", chunksize=None, if_exists='append') File "/env/lib/python3.7/site-packages/pandas/core/frame.py", line 1546, in to_gbq credentials=credentials, File "/env/lib/python3.7/site-packages/pandas/io/gbq.py", line 219, in to_gbq private_key=private_key, File "/env/lib/python3.7/site-packages/pandas_gbq/gbq.py", line 1177, in to_gbq credentials=connector.credentials, File "/env/lib/python3.7/site-packages/pandas_gbq/gbq.py", line 1279, in __init__ private_key=private_key, File "/env/lib/python3.7/site-packages/pandas_gbq/gbq.py", line 353, in __init__ "Could not determine project ID and one was not supplied." ValueError: Could not determine project ID and one was not supplied.

What I'm doing v2

However, when I explicitly specify project_id and credentials with df.to_gbq("easilys.check_pandas", chunksize=None, if_exists='append', project_id=project_id, credentials=credentials), everything works fine.

P.S. If I explicitly specify credentials by providing .json key. i.e.

path_to_credentials_key = 'key.json'
pandas_gbq.context.credentials = service_account.Credentials.from_service_account_file(path_to_credentials_key)

df.to_gbq("xxx.check_pandas", chunksize=None, if_exists='append') works fine.

[tswast]

I see that you're getting the project ID at authentication time. Can you please verify that pandas_gbq.context.project is not None before calling to_gbq. The google.auth.default function can return a valid credentials object without a project ID in some cases.

[tswast]

Closing, as I didn't hear back.

Note: the "project" return value in https://google-auth.readthedocs.io/en/latest/reference/google.auth.html#google.auth.default is optional. For example, Cloud SDK credentials don't always return the default project ID.