-
Notifications
You must be signed in to change notification settings - Fork 14.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FIX] Scheduler crashloopbackoff when using hostname_callable
#24999
[FIX] Scheduler crashloopbackoff when using hostname_callable
#24999
Conversation
airflow/cli/cli_parser.py
Outdated
@@ -897,6 +897,13 @@ def string_lower_type(val): | |||
help="The hostname of job(s) that will be checked.", | |||
) | |||
|
|||
ARG_JOB_HOSTNAME_CALLABLE_FILTER = Arg( | |||
("--use-hostname-callable",), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure I love use-hostname-callable
. Maybe local-only
? @dstandish @ephraimbuddy
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's a nice suggestion @jedcunningham. I don't like so much use-hostname-callable
either, but I am not sure local-only
would be very telling.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What about not adding a new arg but having the hostname arg default to get_hostname
?
i.e
ARG_JOB_HOSTNAME_FILTER = Arg(
("--hostname",),
default=get_hostname(),
type=str,
help="The hostname of job(s) that will be checked.",
)
The effect of this would be that the CLI check command would always check with hostname instead of none.
As per the name, what do you think about --use-configured-hostname
or --use-default-hostname
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sounds very good, thanks for the proposal. Going to implement this
5c88862
to
dd34bd2
Compare
Thanks both for your review. I finally decided to implement the great suggestion of @ephraimbuddy, giving the |
Just adding a default for hostname is a good idea, but it does break backwards compatibility. At the very least we'd need a way do not filter by hostname, and even then I'm not sure about this. |
Tank you @jedcunningham. So am I going back with the previous argument I implemented at first ( |
Yeah, I think we need to. I'm having trouble coming up with anything better than |
e41c302
to
07f2cc3
Compare
@jedcunningham, I rebased on |
db3f314
to
6472e54
Compare
0e80b7f
to
1265559
Compare
1265559
to
fb2590f
Compare
Fix a scheduler crashLoopBackOff when `hostname_callable` was defined with a function which returned something different than the hostname command from linux.
fb2590f
to
7992f75
Compare
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
Thank you @jedcunningham |
Fix a scheduler
crashLoopBackOff
whenhostname_callable
was defined with a function which returned something different than thehostname
command from linux.Context
In our production Airflow, I updated the
hostname_callable
config withairflow.utils.net.get_host_ip_address
so that we could read the pod logs produced by our celery workers. But this new config was making kubernetes to restart the scheduler pods because the command defined in helm was returning a "No alive jobs found."I therefore updated the cli command jobs to allow the use of
--use-hostname-callable
in:airflow jobs check --job-type SchedulerJob --use-hostname-callable
to use the
hostname_callable
that the user defined in its Airflow conf. Also updated the helm chart to with this new conf since I think it will be more reliable than to use linuxhostname
command.