Airflow does not start on Openshift using Local Filesystem Secret Backend

[cmerod]

Hello,

I have a problem with implementing Airflow when using Local Filesystem as secret backend.

I have a deployment of Airflow 2.5.3 on Openshift using the Helm Chart supplied by the community. I'm using the CeleryExecutor and have a postgres as the metadb.

Airflow works fine when Openshift secrets that contain sql_alchemy_conn and fernet_key etc. are supplied as environment variables. The scheduler and webserver pods starts and everything works as intended.

When I try to run Airflow using LocalFilesystemBackend the application is unable to connect to the postgres db.
I used the following documentation when trying to implement the secret backend:
Setting Configuration Options
Local Filesystem Secrets Backend

The log supplies the following error:

....................
ERROR! Maximum number of retries (20) reached.
Last check result:
$ airflow db check
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 5, in <module>
from airflow.__main__ import main
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/__init__.py", line 49, in <module>
from airflow import configuration
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/configuration.py", line 1799, in <module>
conf.validate()
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/configuration.py", line 341, in validate
self._validate_config_dependencies()
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/configuration.py", line 443, in _validate_config_dependencies
raise AirflowConfigException(f"error: cannot use sqlite with the {self.get('core', 'executor')}")
airflow.exceptions.AirflowConfigException: error: cannot use sqlite with the CeleryExecutor

I understand from the error that Airflow fails to find the files (or find the variables/connections in them) that are intended as the secret backend.

In the helm chart, I updated the airflow.cfg by adding the secret section and specifying what type of backend:

kubernetes_executor:
    namespace: '{{ .Release.Namespace }}'
    airflow_configmap: '{{ include "airflow_config" . }}'
    airflow_local_settings_configmap: '{{ include "airflow_config" . }}'
    pod_template_file: '{{ include "airflow_pod_template_file" . }}/pod_template_file.yaml'
    worker_container_repository: '{{ .Values.images.airflow.repository | default .Values.defaultAirflowRepository }}'
    worker_container_tag: '{{ .Values.images.airflow.tag | default .Values.defaultAirflowTag }}'
    multi_namespace_mode: '{{ ternary "True" "False" .Values.multiNamespaceMode }}'
 secrets:
    backend: 'airflow.secrets.local_filesystem.LocalFilesystemBackend'
    backend_kwargs: '{"variables_file_path": "/opt/airflow/backend/airflow-backend-variables.json", "connections_file_path": "/opt/airflow/backend/airflow-backend-connections.json"}'
 database:
    sql_alchemy_conn_secret: 'sql_alchemy_conn'

I also defined the defined the _secret properties in airflow.cfg, for example the sql_alchemy_conn_secret seen above.

Then in values.yaml file I defined the extra volumes for scheduler, webserver, triggerer and worker:

extraVolumes: 
  - name: backend-secrets
    secret:
      secretName: airflow-backend-secrets

extraVolumeMounts:
  - name: backend-secrets
    readOnly: true
    mountPath: /opt/airflow/backend/airflow-backend-variables.json
    subPath: airflow-backend-variables.json
  - name: backend-secrets
    readOnly: true
    mountPath: /opt/airflow/backend/airflow-backend-connections.json
    subPath: airflow-backend-connections.json

During testing I became uncertain whether Airflow treats the properties as variables or connections, so I put all of the properties in both airflow-backend-connections.json and airflow-backend-variables.json:

{
    "fernet_key": "fernetkey",
    "secret_key": "webserverkey",
    "sql_alchemy_conn": "postgresql://user:password@airflow-postgresql.namespace:5432/airflow_db",
    "result_backend": "db+postgresql://user:password@airflow-postgresql:5432/airflow_db",
    "broker_url": "redis://:password@airflow-redis:6379/0"
}

When I access the scheduler pod, I can see that the files and updates to airflow.cfg are there as intended.

The files are in place. I set the permission to all to see if that was the issue.

default@airflow-scheduler-6bcc985989-lf8rm:/opt/airflow/backend$ pwd
/opt/airflow/backend
default@airflow-scheduler-6bcc985989-lf8rm:/opt/airflow/backend$ ls -al
total 8
drwxr-xr-x. 2 root    root        84 Apr 26 08:46 .
drwxrwxr-x. 1 airflow root       102 Apr 26 08:46 ..
-rwxrwxrwx. 1 root    1000630000 403 Apr 26 08:46 airflow-backend-connections.json
-rwxrwxrwx. 1 root    1000630000 404 Apr 26 08:46 airflow-backend-variables.json
default@airflow-scheduler-6bcc985989-lf8rm:/opt/airflow/backend$ 

Airflow.cfg have the updated properties.

default@airflow-scheduler-6bcc985989-lf8rm:/opt/airflow$ cat airflow.cfg 
[celery]
broker_url_secret = broker_url
result_backend_secret = result_backend
worker_concurrency = 16

[celery_kubernetes_executor]
kubernetes_queue = kubernetes

[core]
colored_console_log = False
dags_folder = /opt/airflow/dags/repo/dags
executor = CeleryExecutor
fernet_key_secret = fernet_key
load_examples = False
remote_logging = False

[database]
sql_alchemy_conn_secret = sql_alchemy_conn

[elasticsearch]
json_format = True
log_id_template = {dag_id}_{task_id}_{execution_date}_{try_number}

[elasticsearch_configs]
max_retries = 3
retry_timeout = True
timeout = 30

[kerberos]
ccache = /var/kerberos-ccache/cache
keytab = /etc/airflow.keytab
principal = airflow@FOO.COM
reinit_frequency = 3600

[kubernetes_executor]
airflow_configmap = airflow-airflow-config
airflow_local_settings_configmap = airflow-airflow-config
multi_namespace_mode = False
namespace = namespace
pod_template_file = /opt/airflow/pod_templates/pod_template_file.yaml
worker_container_repository = repositoryurl:18079/airflow
worker_container_tag = 1040

[logging]
colored_console_log = False
logging_level = DEBUG
remote_logging = False

[metrics]
statsd_host = airflow-statsd
statsd_on = False
statsd_port = 9125
statsd_prefix = airflow

[scheduler]
run_duration = 41460
standalone_dag_processor = False
statsd_host = airflow-statsd
statsd_on = False
statsd_port = 9125
statsd_prefix = airflow

[secrets]
backend = airflow.secrets.local_filesystem.LocalFilesystemBackend
backend_kwargs = {"variables_file_path": "/opt/airflow/backend/airflow-backend-variables.json", "connections_file_path": "/opt/airflow/backend/airflow-backend-connections.json"}

[webserver]
enable_proxy_fix = True
expose_stacktrace = True
rbac = True
secret_key_secret = secret_key

When I try to execute command airflow config list, I get the error message I mentioned earlier.

Other actions taken:
Attempted yaml and env file formats for the backend files instead of json.
Tried to point to the variables/connections using env var like AIRFLOW__DATABASE__SQL_ALCHEMY_CONN_SECRET = sql_alchemy_conn.
Supplied SQL_ALCHEMY_CONN as a variable but keeping the rest of the properties in the file backend. This allowed the scheduler and webserver to start up, but any activity involving for example the fernet key gets an error.

I am out of ideas of what detail I am missing to make this work and would appreciate any assistance in making Airflow use the Local Filesystem Secret Backend.

Regards,
Marcus

[potiuk]

LocalFilesystemBackend does not support storing config - only connections and variables.

You should configure your connections/fernet key differently (directly via K8S secrets for example) - see examples in our production-guide and helm values.
https://airflow.apache.org/docs/helm-chart/stable/production-guide.html#kubernetes-secret