How do I specify a Persistent Claim reference for an AirFlow worker pod to persist a file that is not a Dag or Log.

[bruciebruce]

Official Helm Chart version

1.5.0 (latest released)

Apache Airflow version

2.2.4 (latest released)

Kubernetes Version

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.7+rke2r2", GitCommit:"b56e432f2191419647a6a13b9f5867801850f969", GitTreeState:"clean", BuildDate:"2022-03-04T23:17:38Z", GoVersion:"go1.16.14b7", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.7+rke2r2", GitCommit:"b56e432f2191419647a6a13b9f5867801850f969", GitTreeState:"clean", BuildDate:"2022-03-04T23:17:38Z", GoVersion:"go1.16.14b7", Compiler:"gc", Platform:"linux/amd64"}

Helm Chart configuration

No response

Docker Image customisations

No response

What happened

I have spun up a local RKE2 cluster using Helm charts for most pods and would like to have some Python Airflow Dags persist data to the local file system.

I have created a persistent volume like this:

and a persistent volume claim like this:

The PV and PVC are created.

$ kubectl get pv

$ kubectl get pvc

Question: I see that you can configure the web and worker section of the airflow yaml to persist into the dag and log directories. This is not what I want. I would like to persist files into a directory, like in the above mountPoint /mnt/data

Thank you

What you think should happen instead

No response

How to reproduce

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[bruciebruce]

I have some dags files that read data off of http sites, does data smashing and persists Pandas dataframes to s3. I need to change those s3 references to the local file system in k8s and wish to persist as a .csv file. I have set up a pv and a pvc as seen above.

[bruciebruce]

.