-
Notifications
You must be signed in to change notification settings - Fork 14.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix broken SLA Mechanism #14056
Merged
Merged
Fix broken SLA Mechanism #14056
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
closes apache#14050 We were not de-serializing `BaseOperator.sla` properly, hence we were returning float instead of `timedelta` object. Example: 100.0 instead of timedelta(seconds=100) And because we had a check in _manage_sla in `SchedulerJob` and `DagFileProcessor`, we were skipping SLA. SchedulerJob: https://github.com/apache/airflow/blob/88bdcfa0df5bcb4c489486e05826544b428c8f43/airflow/jobs/scheduler_job.py#L1766-L1768 DagFileProcessor: https://github.com/apache/airflow/blob/88bdcfa0df5bcb4c489486e05826544b428c8f43/airflow/jobs/scheduler_job.py#L395-L397
boring-cyborg
bot
added
area:Scheduler
including HA (high availability) scheduler
area:serialization
labels
Feb 4, 2021
potiuk
approved these changes
Feb 4, 2021
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest master at your convenience, or amend the last commit of the PR, and push it with --force-with-lease. |
github-actions
bot
added
the
full tests needed
We need to run full set of tests for this PR to merge
label
Feb 4, 2021
kaxil
added a commit
that referenced
this pull request
Feb 4, 2021
closes #14050 We were not de-serializing `BaseOperator.sla` properly, hence we were returning float instead of `timedelta` object. Example: 100.0 instead of timedelta(seconds=100) And because we had a check in _manage_sla in `SchedulerJob` and `DagFileProcessor`, we were skipping SLA. SchedulerJob: https://github.com/apache/airflow/blob/88bdcfa0df5bcb4c489486e05826544b428c8f43/airflow/jobs/scheduler_job.py#L1766-L1768 DagFileProcessor: https://github.com/apache/airflow/blob/88bdcfa0df5bcb4c489486e05826544b428c8f43/airflow/jobs/scheduler_job.py#L395-L397 (cherry picked from commit 604a37e)
turbaszek
added a commit
to PolideaInternal/airflow
that referenced
this pull request
Feb 4, 2021
Users have to provide SLA as timedelta. If not provided it should be a default None value. This change will result in run time errors if other type than timedelta is passed as SLA. With this change spotting apache#14056 would be easier.
kaxil
added a commit
that referenced
this pull request
Feb 4, 2021
closes #14050 We were not de-serializing `BaseOperator.sla` properly, hence we were returning float instead of `timedelta` object. Example: 100.0 instead of timedelta(seconds=100) And because we had a check in _manage_sla in `SchedulerJob` and `DagFileProcessor`, we were skipping SLA. SchedulerJob: https://github.com/apache/airflow/blob/88bdcfa0df5bcb4c489486e05826544b428c8f43/airflow/jobs/scheduler_job.py#L1766-L1768 DagFileProcessor: https://github.com/apache/airflow/blob/88bdcfa0df5bcb4c489486e05826544b428c8f43/airflow/jobs/scheduler_job.py#L395-L397 (cherry picked from commit 604a37e)
kaxil
pushed a commit
that referenced
this pull request
Feb 7, 2021
Users have to provide SLA as timedelta. If not provided it should be a default None value. This change will result in run time errors if other type than timedelta is passed as SLA. With this change spotting #14056 would be easier.
kaxil
added a commit
to astronomer/airflow
that referenced
this pull request
Feb 16, 2021
closes apache#14050 We were not de-serializing `BaseOperator.sla` properly, hence we were returning float instead of `timedelta` object. Example: 100.0 instead of timedelta(seconds=100) And because we had a check in _manage_sla in `SchedulerJob` and `DagFileProcessor`, we were skipping SLA. SchedulerJob: https://github.com/apache/airflow/blob/88bdcfa0df5bcb4c489486e05826544b428c8f43/airflow/jobs/scheduler_job.py#L1766-L1768 DagFileProcessor: https://github.com/apache/airflow/blob/88bdcfa0df5bcb4c489486e05826544b428c8f43/airflow/jobs/scheduler_job.py#L395-L397 (cherry picked from commit 604a37e) (cherry picked from commit 8f643b8)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
area:Scheduler
including HA (high availability) scheduler
area:serialization
full tests needed
We need to run full set of tests for this PR to merge
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
closes #14050
We were not de-serializing
BaseOperator.sla
properly, hencewe were returning float instead of
timedelta
object.Example: 100.0 instead of timedelta(seconds=100)
And because we had a check in _manage_sla in
SchedulerJob
andDagFileProcessor
,we were skipping SLA.
SchedulerJob:
airflow/airflow/jobs/scheduler_job.py
Lines 1766 to 1768 in 88bdcfa
DagFileProcessor:
airflow/airflow/jobs/scheduler_job.py
Lines 395 to 397 in 88bdcfa
Tested it with breeze too: #14050 (comment)
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.