feat: add liveness probe for celery workers

[nickwood]

What issues does your PR fix?

• fixes the celery worker indefinitely not processing any tasks after airflow-redis pod restarted  #600

What does your PR do?

Currently celery workers can enter a 'zombie' state where they become disconnected from celery and will no longer pick up new jobs.

This PR adds a iveness probe (enabled by default) to detect such pods so they can be killed by k8s and recreated.

Checklist

For all Pull Requests

• Commits are signed off
• Commits have semantic messages
• Documentation updated
• Passes ct linting

[Talador12]

@thesuperzapper this one comes from our org, bringing a change that helped us upstream

[Talador12]

LGTM

[thesuperzapper]

@nickwood thanks for this!

I have made a significant change in 20e7546 to rewrite the probe in Python, so it should now be ready to include in the next release of the chart.

This is because the approach that you proposed did not work for all versions of Airflow (for example it did not work in Airflow 2.7.0 because the --app name changed).

[nglehuy]

FYI, for anyone who has airflow worker liveness probe failure TypeError: argument of type 'NoneType' is not iterable.
This is because of celery inspect can not see the worker running.
We need to set the environment variable AIRFLOW__CELERY__WORKER_ENABLE_REMOTE_CONTROL=True instead of False to make the celery inspect ping works.

[Talador12]

Could we set this `AIRFLOW__CELERY__WORKER_ENABLE_REMOTE_CONTROL` field as the default since the liveness probe is also default behavior?

…

On Fri, Nov 3, 2023 at 2:58 AM Nguyễn Lê Huy ***@***.***> wrote: FYI, for anyone who has airflow worker liveness probe failure TypeError: argument of type 'NoneType' is not iterable. This is because of celery inspect can not see the worker running. We need to set the environment variable AIRFLOW__CELERY__WORKER_ENABLE_REMOTE_CONTROL=True instead of False to make the celery inspect ping works. — Reply to this email directly, view it on GitHub <#766 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AASBCZMVLVPYNAOCDGCD5L3YCSP3VAVCNFSM6AAAAAA3BOPBBOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOJSGAYDSNZSGQ> . You are receiving this because you commented.Message ID: ***@***.***>

[nglehuy]

Could we set this AIRFLOW__CELERY__WORKER_ENABLE_REMOTE_CONTROL field as

It's default 'true' in airflow config: https://airflow.apache.org/docs/apache-airflow-providers-celery/stable/configurations-ref.html#worker-enable-remote-control
The FYI is just for anyone who somehow already set it to false.