Exception "File /etc/gcp/sa_credentials.json was not found" while using Azure Storage Account

[vitorjcorreia]

Hello.

I am using the Helm chart provided in the other repository for installing diffgram in a Kubernetes cluster. The problem is the two containers (diffgram-default and diffgram-walrus) cannot find the following file /etc/gcp/sa_crendetials. If I am using Azure Storage Account to store the static files, this error should not occur. Anyways, I wonder if I am doing this right so I will paste the values.yaml file and the log file of the exception that I obtained.

values.yaml:

# Default values for diffgram.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

# The Diffgram Version. Whenever a new update arrives, this will be changed.
diffgramVersion: latest

# Either 'opencore' or 'enterprise'. Please note that selecting 'enterprise'
# requires that you also set imagePullCredentials.gcrCredentials.
diffgramEdition: opencore

# Set this to your public domain where you want diffgram to be.
# This must be a domain name and not a public IP address.
# The chart will generate TLS certificates for the provided domain if useCertManager is 'true'
diffgramDomain: *HIDDEN*

# Set this to true if you want to use cert manager for TLS certificates generation.
useCertManager: false

dbSettings:
  # Specify How the DB Service should be created
  # - local: use a Postgres Image and Service (No external service) Recommended only for non-production enviroments.
  # - rds: use an ExternalService with an AWS RDS instance. If you set this you need to provide the rdsEndpoint field.
  # - azure: use an ExternalService with an Azure Postgres instance. If you set this you need to provide the azureSqlEndpoint field.
  # - gcsql: use an ExternalService with a Google Cloud SQL instance. If you set this you need to provide the gcSqlEndpoint field.
  dbProvider: local
  rdsEndpoint: none
  azureSqlEndpoint: none
  gcsqlEndpoint: none
  dbUser: *HIDDEN*
  dbName: *HIDDEN*
  dbPassword: *HIDDEN*
  # For the local postgres DB. Does not have effect on RDS or GCP services.
  storageAmount: 5Gi

# All the Secrets Used in Diffgram.
diffgramSecrets:
  STRIPE_API_KEY: none
  DIFFGRAM_AWS_ACCESS_KEY_ID: none
  DIFFGRAM_AWS_ACCESS_KEY_SECRET: none
  _ANALYTICS_WRITE_KEY: provided_by_diffgram_team
  MAILGUN_KEY: provided_by_diffgram_team
  HUB_SPOT_KEY: provided_by_diffgram_team
  SECRET_KEY: provided_by_diffgram_team
  INTER_SERVICE_SECRET: provided_by_diffgram_team
  # Use diffgram-postgres, postgres-rds-service depending on which DB service you set on dbSettings
  USER_PASSWORDS_SECRET: provided_by_diffgram_team
  # The service account JSON for GCP Static Storage Encoded in Base64.
  SERVICE_ACCOUNT_JSON_B64: put_your_gcp_secret_in_base_64_here
  DIFFGRAM_AZURE_CONNECTION_STRING: *HIDDEN*

diffgramSettings:
  USERDOMAIN: kubernetes
  DIFFGRAM_SYSTEM_MODE: production
  DIFFGRAM_STATIC_STORAGE_PROVIDER: azure
  DIFFGRAM_S3_BUCKET_NAME: none
  DIFFGRAM_AZURE_CONTAINER_NAME: diffgram-dev
  ML__DIFFGRAM_AZURE_CONTAINER_NAME: diffgram-dev
  ML__DIFFGRAM_S3_BUCKET_NAME: diffgram-testing
  CLOUD_STORAGE_BUCKET: diffgram-testing
  ML__CLOUD_STORAGE_BUCKET: diffgram-testing
  GOOGLE_APPLICATION_CREDENTIALS: /etc/gcp/sa_credentials.json # Check the volume in deployment.yaml and service_account_secret.yaml
  SERVICE_ACCOUNT_FULL_PATH: /etc/gcp/sa_credentials.json
  
  # Set this value if you want to use GCP as your storage. Put your json service account encoded in base 64
  SERVICE_ACCOUNT_JSON_B64: none

  SERVICE_ACCOUNT: sa_credentials.json

imagePullCredentials:
  # The service account with permissions to pull from the GCR Repository. [Should be Provided by Diffgram Team.]
  gcrCredentials: provided_by_diffgram_team

# The service for API calls.
# This are minimal defaults. Please feel free to change them as you start having more usage
defaultService:
  numReplicas: 1
  requests:
    cpu: "2.0"
    memory: "2G"
  limits:
    cpu: "2.0"
    memory: "2G"
# The service for the UI frontend.
# This are minimal defaults. Please feel free to change them as you start having more usage
frontendService:
  numReplicas: 1
  requests:
    cpu: "1.0"
    memory: "2G"
  limits:
    cpu: "1.0"
    memory: "2G"
# The service for video processing. This is where the heavy processing takes place.
# This are minimal defaults. Please feel free to change them as you start having more usage
walrusService:
  numReplicas: 1
  requests:
    cpu: "1.0"
    memory: "2G"
  limits:
    cpu: "1.0"
    memory: "2G"

The logs of the container (the error is the same on both):

[2021-06-25 10:49:00 +0000] [8] [INFO] Starting gunicorn 20.0.4
[2021-06-25 10:49:00 +0000] [8] [INFO] Listening at: http://0.0.0.0:8080 (8)
[2021-06-25 10:49:00 +0000] [8] [INFO] Using worker: sync
[2021-06-25 10:49:00 +0000] [11] [INFO] Booting worker with pid: 11
[2021-06-25 10:49:00 +0000] [12] [INFO] Booting worker with pid: 12
[2021-06-25 10:49:00 +0000] [13] [INFO] Booting worker with pid: 13
[2021-06-25 10:49:00 +0000] [14] [INFO] Booting worker with pid: 14
[2021-06-25 10:49:01 +0000] [15] [INFO] Booting worker with pid: 15
[2021-06-25 10:49:05 +0000] [11] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/gunicorn/arbiter.py", line 583, in spawn_worker
    worker.init_process()
  File "/usr/local/lib/python3.6/dist-packages/gunicorn/workers/base.py", line 119, in init_process
    self.load_wsgi()
  File "/usr/local/lib/python3.6/dist-packages/gunicorn/workers/base.py", line 144, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/usr/local/lib/python3.6/dist-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/usr/local/lib/python3.6/dist-packages/gunicorn/app/wsgiapp.py", line 49, in load
    return self.load_wsgiapp()
  File "/usr/local/lib/python3.6/dist-packages/gunicorn/app/wsgiapp.py", line 39, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/usr/local/lib/python3.6/dist-packages/gunicorn/util.py", line 358, in import_app
    mod = importlib.import_module(module)
  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/app/main.py", line 37, in <module>
    import shared.database_setup_supporting
  File "/app/shared/database_setup_supporting.py", line 2, in <module>
    from shared.database.discussion.discussion_comment import DiscussionComment
  File "/app/shared/database/discussion/discussion_comment.py", line 2, in <module>
    from shared.database.common import *
  File "/app/shared/database/common.py", line 41, in <module>
    from shared import data_tools_core
  File "/app/shared/data_tools_core.py", line 3, in <module>
    from shared.data_tools_core_gcp import DataToolsGCP
  File "/app/shared/data_tools_core_gcp.py", line 15, in <module>
    logger = get_shared_logger()
  File "/app/shared/shared_logger.py", line 8, in get_shared_logger
    shared_abstract_logger.configure_concrete_logger(system_mode=settings.DIFFGRAM_SYSTEM_MODE)
  File "/app/shared/utils/logging.py", line 53, in configure_concrete_logger
    self.logger = self.configure_gcp_logger()
  File "/app/shared/utils/logging.py", line 72, in configure_gcp_logger
    logging_client = gcp_logging.Client()
  File "/usr/local/lib/python3.6/dist-packages/google/cloud/logging/client.py", line 127, in __init__
    project=project, credentials=credentials, _http=_http
  File "/usr/local/lib/python3.6/dist-packages/google/cloud/client.py", line 318, in __init__
    _ClientProjectMixin.__init__(self, project=project, credentials=credentials)
  File "/usr/local/lib/python3.6/dist-packages/google/cloud/client.py", line 266, in __init__
    project = self._determine_default(project)
  File "/usr/local/lib/python3.6/dist-packages/google/cloud/client.py", line 285, in _determine_default
    return _determine_default_project(project)
  File "/usr/local/lib/python3.6/dist-packages/google/cloud/_helpers.py", line 186, in _determine_default_project
    _, project = google.auth.default()
  File "/usr/local/lib/python3.6/dist-packages/google/auth/_default.py", line 454, in default
    credentials, project_id = checker()
  File "/usr/local/lib/python3.6/dist-packages/google/auth/_default.py", line 222, in _get_explicit_environ_credentials
    os.environ[environment_vars.CREDENTIALS]
  File "/usr/local/lib/python3.6/dist-packages/google/auth/_default.py", line 108, in load_credentials_from_file
    "File {} was not found.".format(filename)
google.auth.exceptions.DefaultCredentialsError: File /etc/gcp/sa_credentials.json was not found.
No module named 'shared.settings.secrets'
[2021-06-25 10:49:05 +0000] [11] [INFO] Worker exiting (pid: 11)
PROCESS_MEDIA_TRY_BLOCK_ON True
PROCESS_MEDIA_REMOTE_QUEUE_ON True
PROCESS_MEDIA_ENQUEUE_LOCALLY_IMMEDIATELY False
DIFFGRAM_SYSTEM_MODE DIFFGRAM_SYSTEM_MODE  production
DATABASE_URL DATABASE_URL  True

I specified the variable SERVICE_ACCOUNT_FULL_PATH with a dummy path since it is recommended in this issue #68. Also, I followed this tutorial: https://medium.com/diffgram/tutorial-installing-diffgram-on-azure-aks-b9447685e271

Appreciate any help.

[PJEstrada]

Hello @binofc

Thanks for sharing this issue with us. We've identified the problem and are working on the fix here: #123

This should be released on the next release 0.6.3 in when the PR is ready: https://github.com/diffgram/diffgram/releases/tag/untagged-071ac06654659d08a22d

As a workaround, you can just change the DIFFGRAM_SYSTEM_MODE to sandbox in your values.yaml and the problem should not appear, as this is related to the logger that is using GCP credentials in the production mode.

Please let us know if this helps and if you have any other questions :)

[vitorjcorreia]

Hi again @PJEstrada!

Well, didn't know this problem was related to the logger that is using GCP credentials, thus I submitted this issue. Sorry about that.

I created a service account in GCP and injected the credentials in a Kubernetes secret (since I didn't know that changing DIFFGRAM_SYSTEM_MODE to sandbox could solve that) and successfully deployed it.

Tested it a few moments ago and after changing the mode to sandbox its up and running. Thanks! Looking forward to the next release anyways :)

Should I close this issue?

[PJEstrada]

Awesome :D

I'll make sure the release is live and close it when we have it ready :) Thank you for trying Diffgram out

[PJEstrada]

Fixed in: https://github.com/diffgram/diffgram/releases/tag/0.6.3