Mounting DAGS from an externally populated PVC doesn't work in K8 Executor

[rchojn]

Apache Airflow version: 1.10.12

Kubernetes version (if you are using kubernetes) (use kubectl version): 1.18

What happened: Mounting DAGS from an externally populated PVC doesn't work:

  --set dags.persistence.enabled=true \
  --set dags.persistence.existingClaim=my-volume-claim
  --set dags.gitSync.enabled=false

Envionment variables from K8 Executor worker

│ Environment:
││ AIRFLOW_HOME: /opt/airflow ││ AIRFLOW__CORE__DAGS_FOLDER: /opt/airflow/dags/repo/
││ AIRFLOW__CORE__DAG_CONCURRENCY: 5 │
│ AIRFLOW__CORE__EXECUTOR: LocalExecutor │
│ AIRFLOW__CORE__FERNET_KEY: <set to the key 'fernet-key' in secret 'airflow-fernet-key'> Optional: false │
│ AIRFLOW__CORE__PARALLELISM: 5 │
│ AIRFLOW__CORE__SQL_ALCHEMY_CONN: <set to the key 'connection' in secret 'airflow-airflow-metadata'> Optional: false │
│ AIRFLOW__KUBERNETES__DAGS_VOLUME_SUBPATH: repo/

What you expected to happen: Dags mounted in workers from PVC

How to reproduce it: Use chart from master and set variables as above

[kaxil]

@GardnerDev Can you confirm if this worked before Airflow 1.10.12?

cc @dimberman

[potiuk]

Is that correct?:

AIRFLOW__CORE__EXECUTOR: LocalExecutor

Should not that be KubernetesExecutor?

[kaxil]

No that is correct, the Kubernetes worker runs the tasks locally with LocalExecutor.

Scheduler uses Kubernetes Executor to launch a pod, but ince launched that pod uses local exexutor

[potiuk]

[rchojn]

@kaxil yes, airflow 1.10.10 behaves in the same way

[rchojn]

@dimberman @kaxil where can I found worker definition of K8 executor in charts?

[dimberman]

@GardnerDev can you please post the entire yaml for the worker? If it was mounted via PVC I imagine that would be in the volume mounts?

[cheunhong]

@dimberman I have faced this issue today too, solved it by modifying this line

airflow/chart/files/pod-template-file.kubernetes-helm-yaml

Line 68 in bb00f16

{{- if .Values.dags.persistence.enabled }}

Should have been {{- if .Values.dags.gitSync.enabled }}