Unable to configure Google Secrets Manager in 2.3.4

[aspain]

Apache Airflow version

2.3.4

What happened

I am attempting to configure a Google Secrets Manager secrets backend using the gcp_keyfile_dict param in a .env file with the following ENV Vars:

AIRFLOW__SECRETS__BACKEND=airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend
AIRFLOW__SECRETS__BACKEND_KWARGS='{"connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables", "gcp_keyfile_dict": <json-keyfile>}'

In previous versions including 2.3.3 this worked without issue

After upgrading to Astro Runtime 5.0.8 I get the following error taken from the scheduler container logs. The scheduler, webserver, and triggerer are continually restarting

Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 5, in <module>
    from airflow.__main__ import main
  File "/usr/local/lib/python3.9/site-packages/airflow/__init__.py", line 35, in <module>
    from airflow import settings
  File "/usr/local/lib/python3.9/site-packages/airflow/settings.py", line 35, in <module>
    from airflow.configuration import AIRFLOW_HOME, WEBSERVER_CONFIG, conf  # NOQA F401
  File "/usr/local/lib/python3.9/site-packages/airflow/configuration.py", line 1618, in <module>
    secrets_backend_list = initialize_secrets_backends()
  File "/usr/local/lib/python3.9/site-packages/airflow/configuration.py", line 1540, in initialize_secrets_backends
    custom_secret_backend = get_custom_secret_backend()
  File "/usr/local/lib/python3.9/site-packages/airflow/configuration.py", line 1523, in get_custom_secret_backend
    return _custom_secrets_backend(secrets_backend_cls, **alternative_secrets_config_dict)
TypeError: unhashable type: 'dict'

What you think should happen instead

Containers should remain healthy and the secrets backend should successfully be added

How to reproduce

astro dev init a fresh project

Dockerfile:
FROM quay.io/astronomer/astro-runtime:5.0.8

.env file:

AIRFLOW__SECRETS__BACKEND=airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend
AIRFLOW__SECRETS__BACKEND_KWARGS='{"connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables", "gcp_keyfile_dict": <service-acct-json-keyfile>}'

astro dev start

Operating System

macOS 11.6.8

Versions of Apache Airflow Providers

apache-airflow-providers-google 8.1.0

Deployment

Astronomer

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

@pdebelak - I think this is caused by the LRU cache introduced in #25556 - is it possible you take a look and see if it can be fixed/workarounded ?

[potiuk]

I believe the problem is that dict-indeed is not hashable, and you can pass the dict as parameter of the secret backend configuration.

For now, I don't see an easy workaround other than using gcp_key_path and putting the key in the same path in your workers - would that be a feasible workaround for now @aspain ?

[aspain]

With an Astronomer project I don't have access to the workers (other than locally) and would have to include the keyfile in my repository the project deploys from, ideally the keyfile would not need to be in the repository

In my local environment I am using a .env file but not pushing it to the repo, and in the Astro UI I am able to add environment variables

[pdebelak]

Yes, this is related to the new lru_cache in 2.3.4, I didn't realize this would break in this way. There isn't an easy workaround. We might need to revert that change in this case and add a test to make sure we don't break it in the same way again.

[pdebelak]

I see a fix for this that I will PR, but I don't see a workaround for version 2.3.4 if you have a AIRFLOW__SECRETS__BACKEND_KWARGS containing a nested dictionary.

[potiuk]

yeah. There is no easy workaround I could see for that one. I will raise it to the release mgmt team (we have one more bug that might make us do 2.3.5 before we reelase 2.4.0. In the meantime @pdebelak - looking forward to a fix :D