Connections and variables are not being created based on extraSecrets and extraConfigMaps

[belaytzev]

Official Helm Chart version

1.10.0 (latest released)

Apache Airflow version

v2.7.1

Kubernetes Version

1.27

Helm Chart configuration

No response

Docker Image customizations

install apache-airflow[postgres,amazon,cncf.kubernetes,daskexecutor]

What happened

I configured extraSecrets and extraConfigMaps for my PostgreSQL connection and some additional variables. I also added extraEnvFrom to attach those extras to Airflow containers.

extraSecrets:
  '{{ .Release.Name }}-airflow-connections':
    type: 'Opaque'
    data: |
      AIRFLOW_CONN_MY_DB: 'base64_encoded_my_db_string'

extraConfigMaps:
  '{{ .Release.Name }}-airflow-variables':
    data: |
      AIRFLOW_VAR_MY_VAR: 'my_var_string'

extraEnvFrom: |
  - secretRef:
      name: '{{ .Release.Name }}-airflow-connections'
  - configMapRef:
      name: '{{ .Release.Name }}-airflow-variables'

Variables are typically added to containers in Airflow, but they do not affect the parameters.

airflow@apache-airflow-webserver:/opt/airflow$ airflow connections list
No data found
airflow@apache-airflow-webserver:/opt/airflow$ airflow variables list
No data found
airflow@apache-airflow-webserver:/opt/airflow$ airflow version
2.7.1
airflow@apache-airflow-webserver:/opt/airflow$ printenv | grep AIRFLOW_CONN
AIRFLOW_CONN_MY_DB=postgres://user:pass@host:5432
airflow@apache-airflow-webserver:/opt/airflow$ printenv | grep AIRFLOW_VAR
AIRFLOW_VAR_MY_VAR=my_var

What you think should happen instead

No response

How to reproduce

Install helm chart with extraSecrets, extraConfigMaps and extraEnvFrom

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[Taragolis]

airflow@apache-airflow-webserver:/opt/airflow$ airflow connections list
No data found
airflow@apache-airflow-webserver:/opt/airflow$ airflow variables list
No data found

CLI commands only lookup connections/variables in Airflow Metabase

[belaytzev]

airflow@apache-airflow-webserver:/opt/airflow$ airflow connections list
No data found
airflow@apache-airflow-webserver:/opt/airflow$ airflow variables list
No data found

CLI commands only lookup connections/variables in Airflow Metabase

Okay, but why didn't Airflow submit parameters from environment variables into Airflow Metabase?

[Taragolis]

It not how it initially designed, airflow just lookup in configured storage during the execution, see Secrets Backend

[belaytzev]

It not how it initially designed, airflow just lookup in configured storage during the execution, see Secrets Backend

Okay, then how can I autofill connections/variables in Airflow? Because in the UI, I don't see any changes after I added environment variables to containers.

In addition to retrieving connections & variables from environment variables or the metastore database

As I understand it, this means that Airflow must display connections/variables in the UI if I have added AIRFLOW_CONN_* or AIRFLOW_VAR_*?

I used this documentation for reference: https://airflow.apache.org/docs/helm-chart/stable/adding-connections-and-variables.html