Fix triggerers alive check and add a new conf for triggerer heartbeat rate #32123
Conversation
… rate Signed-off-by: Hussein Awala <hussein@awala.fr>
hussein-awala
requested review from
dstandish,
kaxil,
ashb and
XD-DENG
as code owners
|
@ephraimbuddy @pierrejeambrun I add a new Airflow configuration in this PR, but I can fix the issue without it where it fallback to scheduler heartbeat rate.
WDYT? |
|
I think it is fine to both: add new config and fallback to 5 seconds. |
| job_heartbeat_sec: | ||
| description: | | ||
| How often to heartbeat the Triggerer job to ensure it hasn't been killed. | ||
| version_added: 2.6.3 |
There was a problem hiding this comment.
Since we are adding a new configuration, this should be moved to 2.7.0
There was a problem hiding this comment.
Not necessarily. I think we do not have a rule "do not add new configuration options" in patchlevel. We have the rule of not adding new features, but new configuration might be added in order to implement a bugfix (which I think is the case here).
I think it's OK (and we've done that in the past) that we introduced new configuration in patchlevel version if they served bugfix purpose. There are a few configs already that were added in non- .0 versions (smtp ones for example).
There was a problem hiding this comment.
(and yes, that was also my initial reaction, but I thought about it and I looked at the past configuration entries we created and it's not obvious that "new configuration == new feature".
There was a problem hiding this comment.
@ephraimbuddy Do you agree with @potiuk, or should I move it to 2.7.0?
|
There were some "stalled" GA jobs running for that one I closed/reopened to rebuild. I take the thumbs-up reaction of @ephraimbuddy as agremeent :). And I think this one is worthy cherry-picking to 2.6.3 :) |
Great! I asked him because of this thumbs-up. |
closes: #32121
This PR:
assign_unassignedmethod: thhe static 30-second threshold is replaced with a dynamically calculated threshold obtained by multiplying the triggerer heartbeat rate by a grace multiplier (2.1).^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.