backend secrets usage AWS SystemsManager

[y0zg]

Hello,
I'm struggling to integrate DB connection from SystemsManager.
I use official helm chart and I know there is some configuration precedence

# 1.set as an environment variable (AIRFLOW__CORE__SQL_ALCHEMY_CONN)
# 2.set as a command environment variable (AIRFLOW__CORE__SQL_ALCHEMY_CONN_CMD)
# 3.set as a secret environment variable (AIRFLOW__CORE__SQL_ALCHEMY_CONN_SECRET)
# 4.set in airflow.cfg
# 5.command in airflow.cfg
# 6.secret key in airflow.cfg
# 7.Airflow's built in defaults

My current setup works with RDS if I use configuration from metadataConnection as

postgresql:
    enabled: false
data:
    metadataConnection: 
        user: xxx
        pass: xxx
        host: xxx.us-east-1.rds.amazonaws.com
        port: 5432
        db: postgres

airflow.cfg contains secrets backend:

[core]
colored_console_log = False
dags_folder = /opt/airflow/dags/repo/dags
executor = KubernetesExecutor
load_examples = False
remote_logging = False
sql_alchemy_conn_secret = sql_alchemy_conn

[database]
sql_alchemy_conn_secret = sql_alchemy_conn

[secrets]
backend = airflow.providers.amazon.aws.secrets.systems_manager.SystemsManagerParameterStoreBackend
backend_kwargs = {"connections_prefix": "/dev/airflow","variables_prefix": null,"config_prefix": null,"region_name": "us-east-1"}

Mu custom values file looks:

executor: "KubernetesExecutor"

config: 
  core:
    sql_alchemy_conn_secret: "sql_alchemy_conn" #"sql_alchemy_conn"
  secrets:
    backend: airflow.providers.amazon.aws.secrets.systems_manager.SystemsManagerParameterStoreBackend
    backend_kwargs: '{"connections_prefix": "/dev/airflow","variables_prefix": null,"config_prefix": null,"region_name": "us-east-1"}'
  database:
    sql_alchemy_conn_secret: "sql_alchemy_conn"

enableBuiltInSecretEnvVars:
  # AIRFLOW__CORE__SQL_ALCHEMY_CONN: false
  AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: false

env:
  - name: "AIRFLOW__LOGGING__LOGGING_LEVEL"
    value: "DEBUG"
  - name: "AIRFLOW__CORE__REMOTE_LOGGING"
    value: "False"
  - name: "AWS_DEFAULT_REGION"
    value: us-east-1
 
pgbouncer:
  enabled: false

redis:
  enabled: false

postgresql:
    enabled: false
data:
    metadataConnection: 
        user: xxx
        pass: xxx
        host: xxx.us-east-1.rds.amazonaws.com
        port: 5432
        db: postgres

When I run this command airflow connections get connections I get correct database URI from AWS ParameterStore

I was also playing with precedence and these vars

AIRFLOW__CORE__SQL_ALCHEMY_CONN
AIRFLOW__DATABASE__SQL_ALCHEMY_CONN
AIRFLOW__DATABASE__SQL_ALCHEMY_CONN_SECRET

and yes, I read docs:

• https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html#generating-a-connection-uri
• https://airflow.apache.org/docs/apache-airflow-providers-amazon/7.3.0/secrets-backends/aws-ssm-parameter-store.html

I appreciate if you provide what's pending for making integration to work

[potiuk]

There is no explanation what's wrong and what problem are you dealing with. Does it work? Does it not work? What does not work if it does not? What are the errors you get and what did you try to do to solve them?