bigtable.client for CI/CD emulator without credentials

[chapmanjacobd]

maybe related to:
googleapis/java-bigtable-hbase#1070
googleapis/python-datastore#11 (comment)

Environment details

• OS type and version: Fedora 33
• Python version: 3.7
• pip version: 20.2.2
• google-cloud-bigtable version: 1.6.1

Steps to reproduce

1. Start bigtable emulator
2. 🌳 env BIGTABLE_EMULATOR_HOST=localhost:8086 python src/tasks/get_pixel.py

This works locally but not inside of GitHub Actions. Running with BigTable Emulator in a CI/CD context shouldn't require credentials.

Code example

    client = Client(project=PROJECT_ID, admin=True)
    instance = client.instance(INSTANCE_ID, LOCATION_ID)
    connection = happytable.Connection(instance=instance)
...
        batch = happytable.Batch(bigtable_table, transaction=True)
...
            batch.put(key, columns)

Stack trace

Unpacking google-cloud-sdk-bigtable-emulator (319.0.0-0) ...
Setting up google-cloud-sdk-bigtable-emulator (319.0.0-0) ...
Processing triggers for google-cloud-sdk (319.0.0-0) ...
Executing: /usr/lib/google-cloud-sdk/platform/bigtable-emulator/cbtemulator --host=localhost --port=8086
[bigtable] Cloud Bigtable emulator running on 127.0.0.1:8086
Traceback (most recent call last):
  File "src/tasks/get_pixel.py", line 129, in <module>
    pipePixelsToBigTable(1, 1, "tests/data/wealth_roc_value_raster.tif")
  File "src/tasks/get_pixel.py", line 23, in pipePixelsToBigTable
    client = Client(project=PROJECT_ID, admin=True)
  File "/opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/site-packages/google/cloud/bigtable/client.py", line 180, in __init__
    client_options=client_options,
  File "/opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/site-packages/google/cloud/client.py", line 250, in __init__
    Client.__init__(self, credentials=credentials, client_options=client_options, _http=_http)
  File "/opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/site-packages/google/cloud/client.py", line 151, in __init__
    credentials, _ = google.auth.default(scopes=scopes)
  File "/opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/site-packages/google/auth/_default.py", line 356, in default
    raise exceptions.DefaultCredentialsError(_HELP_MESSAGE)
google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see cloud.google.com/docs/authentication/getting-started

[crwilcox]

I wouldn't expect different behavior with credential detection from a GitHub action. Are you sure that the environment variable is being detected from within get_pixel.py?

import os
print(os.getenv("BIGTABLE_EMULATOR_HOST"))

[chapmanjacobd]

Yes. The env var is there. I just double checked

      - name: Test against BigTable emulator
        run: |
          sudo apt-get install google-cloud-sdk google-cloud-sdk-bigtable-emulator
          echo $BIGTABLE_EMULATOR_HOST
          gcloud beta emulators bigtable start --host-port=localhost:8086 --quiet &
          sleep 5
          python src/tasks/get_pixel.py

prints localhost:8086

   File "src/tasks/get_pixel.py", line 23, in pipePixelsToBigTable
    client = Client(project=PROJECT_ID)
  File "/opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/site-packages/google/cloud/bigtable/client.py", line 180, in __init__
    client_options=client_options,
  File "/opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/site-packages/google/cloud/client.py", line 250, in __init__
    Client.__init__(self, credentials=credentials, client_options=client_options, _http=_http)
  File "/opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/site-packages/google/cloud/client.py", line 151, in __init__
    credentials, _ = google.auth.default(scopes=scopes)
  File "/opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/site-packages/google/auth/_default.py", line 356, in default
    raise exceptions.DefaultCredentialsError(_HELP_MESSAGE)
google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application

[crwilcox]

@chapmanjacobd where are you running this in CI? Is it by chance on Google Cloud?

[tseaver]

I ran into something related just recently: the Github Action was setting the env var, but the script it called was actually a wrapper around (several) docker run commands. By default, the caller's environment is not passed through to the docker container: I had to add --env MY_ENV_VAR=value to the docker invocation.

[crwilcox]

I am going to consider this resolved.