Add Helm chart to support StatefulSet workers without PVC

[ktaborski]

Decouple StatefulSet deployment from PVC creation in worker configuration. Previously, workers.persistence.enabled controlled both StatefulSet usage and PVC creation, making it impossible to use StatefulSet without volumes.

Changes:

• Modify volume template logic to use logs.persistence.enabled for PVC
• Allow StatefulSet workers with emptyDir when workers.persistence.enabled is true but logs.persistence.enabled is false
• Maintain backward compatibility: default behavior unchanged

This enables stable network identity for workers without persistent storage.

[jscheffl]

Can you describe a reason why you want to have a statefulset?
I assume (missing the legacy reason) there was an idea behind that if no persistence that a deployment is used and not a statefulset.

[ktaborski]

Hi,
advantage of using statefulset is, that I can have real-time logs in UI from workers (when they are running). When finished, logs are fetched from s3.

[ktaborski]

It seems, that changes are not backward compatible, I need to rework them

[jscheffl]

Hi, advantage of using statefulset is, that I can have real-time logs in UI from workers (when they are running). When finished, logs are fetched from s3.

Yeah also had a bit of challenge, when deploying in K8s as "deployment" and not staefulset then you need to adjust one property tp fix nameserver lookup and deployment instances have no DNS.

Personally I am not favoring statefulesets too much as a rolling release really takes long as K8s upgrades one-by-one. With a deployment all is in parallel with a defined volume.

Would it maybe make sense to prevent such problems as alternative option to use config https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#hostname-callable setting it to airflow.utils.net.get_host_ip_address?

[ktaborski]

@jscheffl works like a charm, thank you!