[FIX] Scheduler crashloopbackoff when using hostname_callable

[V0lantis]

Fix a scheduler crashLoopBackOff when hostname_callable was defined with a function which returned something different than the hostname command from linux.

Context

In our production Airflow, I updated the hostname_callable config with airflow.utils.net.get_host_ip_address so that we could read the pod logs produced by our celery workers. But this new config was making kubernetes to restart the scheduler pods because the command defined in helm was returning a "No alive jobs found."
I therefore updated the cli command jobs to allow the use of --use-hostname-callable in:

airflow jobs check --job-type SchedulerJob --use-hostname-callable

to use the hostname_callable that the user defined in its Airflow conf. Also updated the helm chart to with this new conf since I think it will be more reliable than to use linux hostname command.

[jedcunningham]

Not sure I love use-hostname-callable. Maybe local-only? @dstandish @ephraimbuddy

[V0lantis]

It's a nice suggestion @jedcunningham. I don't like so much use-hostname-callable either, but I am not sure local-only would be very telling.

[ephraimbuddy]

What about not adding a new arg but having the hostname arg default to get_hostname?
i.e

ARG_JOB_HOSTNAME_FILTER = Arg(
    ("--hostname",),
    default=get_hostname(),
    type=str,
    help="The hostname of job(s) that will be checked.",
)

The effect of this would be that the CLI check command would always check with hostname instead of none.

As per the name, what do you think about --use-configured-hostname or --use-default-hostname

[V0lantis]

Sounds very good, thanks for the proposal. Going to implement this

[V0lantis]

Thanks both for your review. I finally decided to implement the great suggestion of @ephraimbuddy, giving the get_hostname default argument to the check jobs command. I also created a if ... else condition to use this new functionality only from Airflow 2.4.0 release

[jedcunningham]

Just adding a default for hostname is a good idea, but it does break backwards compatibility. At the very least we'd need a way do not filter by hostname, and even then I'm not sure about this.

[V0lantis]

Just adding a default for hostname is a good idea, but it does break backwards compatibility. At the very least we'd need a way do not filter by hostname, and even then I'm not sure about this.

Tank you @jedcunningham. So am I going back with the previous argument I implemented at first (use-hostname-callable local-only or whatever we want to call it)?

[jedcunningham]

Yeah, I think we need to. I'm having trouble coming up with anything better than --local though.

[V0lantis]

@jedcunningham, I rebased on main, remove the default param on ARG_JOB_HOSTNAME_FILTER. Hope it will suit you.

[V0lantis]

Thank you @jedcunningham