Make KubernetesExecutor PersistentVolumeClaim mounts more flexible.

[jimmajure]

Description

In the KubernetesExecutor configuration, allow the DAGs, logs, plugins folders to be supported by a single PersistentVolumeClaim. Currently the configuration forces a separate PVC for each directory.

Use case / motivation

Mounting volumes via a PVC is an expensive task that requires multiple calls to the K8S API server, so it is beneficial to be able to mount fewer PVC volumes for each pod. We have run into scalability issues in AWS EKS when using EFS drives to support DAGs and logs.

The configuration should allow users to mount as many PVC mounts as required. If people choose to put DAGs, logs and plugins in a single PVC mount, that should be supported. If they choose to put each in a separate PVC mount, that can be supported also.

This should be able to be supported by separating the purpose of the mount from the mount, itself. Simply allow one or more mounts and let the user map the directories to the appropriate folder, irrespective of which mount it is on.

Related Issues

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[mik-laj]

Can the expected result be achieved with pod_template_file option?
https://airflow.readthedocs.io/en/latest/configurations-ref.html#pod-template-file

[github-actions]

This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.

[github-actions]

This issue has been closed because it has not received response from the issue author.