SLA Rewrite - Deprecate Existing SLA Logic

[alex-astronomer]

Topic of Discussion

I've heard a lot of buzz regarding rewriting SLA functionality in Airflow. I agree with this change, I think with the modern version of Airflow, that SLAs are in serious need of a reword. There are many open issues with SLAs around the modern architecture of Airflow like multiple emails being sent when using 2 schedulers and other bugs that are present in the new version of Airflow. See also this link for another problem involving DAGFileProcessorProcess timeout while processing a large amount of SLAs at the same time.

There are also other feature requests about functionality that is not possible with the current implementation of SLA. There is even a comment on that one where @eladkal talks about rewriting the SLA feature, which was part of the inspiration for this discussion and @potiuk seems to agree.

Some of the bugs that happen for SLAs are because of new features that have been implemented in Airflow since the SLA function was implemented. This includes the Taskflow API and multiple HA schedulers in Airflow 2, that can cause issues.

At a high level, I'd like to discuss ideas for features for an SLA rewrite that we'd like to see implemented. Then after the feature set is decided, we can discuss the flow of SLAs through the code and more concrete solution details. Would this become an AIP? I'm happy to learn about that process if so.

Features

Behavior and Definitions

• Have DAG-level SLAs and Task-level SLAs.
  • SLAs can be specified to either DAGs as a whole or Tasks individually
  • Allows for different tasks in the DAG to have different SLAs
  • Can be combined however the user chooses the give the behavior desired
• Ability for the user to specify a plugin that adds a new type of behavior for SLAs besides the default absolute and relative SLAs defined below
• Absolute vs. Relative SLAs
  • Adds additional definition and behavior to DAG and Task level SLAs
  • Absolute SLAs
    • SLA Interval is defined as the time difference between Earliest Start Date and Earliest Start Date + TimeDelta Specified
    • Specified as a timedelta
    • Plain English Story - "I want to be notified if my DAG/Task does not finish within a specified timedelta interval using the earliest time the DAG can start as the reference."
  • Relative SLAs
    • SLA Interval is defined as the time difference between Actual Start Time and Actual End Time
    • Also specified as a timedelta
    • Plain English "I want to be notified if the runtime of my DAG/Task exceeds a runtime longer than a timedelta that I specify. This uses the actual start time of the DAG/Task as the reference point."
  • Examples:

---

• User Stories
  • "The customer expects this data every hour, we start the DAG at 50 minutes until the hour, and the DAG must complete and deliver the data before 0 minutes at the next hour."
  • "In order to make improvements to the efficiency of the process, we want Task B to have a runtime of 4 minutes or less. This SLA notification will allow us to benchmark this specific task and see our improvements once the notifications stop. We can calculate the percentage of the times that the DAG misses its relative 4 minute SLA, and identify where improvements need to be made by to this specific task by collecting those statistics."
  • "There is a secondary, but equally important need to finish Task F within two minutes of the hour. This task has an absolute SLA of 8 minutes because the data delivered before the 'finishing tasks' (G & H) must be delivered so that customers can start preparing their systems before the start of the hour, when our DAG is expected to complete by."

---

• User Stories
  • "In order to optimize the compute used by our cluster, we don't want this DAG to have a runtime of more than one hour. We don't really care when the DAG completes. What is really important to our team is that the amount of time that the DAG takes from start to end is less than that hour."

---

Improvements over Existing SLAs

• sla_miss_callback is called as soon as the SLA is missed, not when the Task or the DAG is completed.
• In code, introduce some sort of separation of concerns in adding/managing SLAMisss as they happen and notifications and other external calls
  • Right now we do all of this in one function inside of the DagFileProcessor process: manage_slas
• No repeat notifications EVER!
  • Multiple schedulers can cause this
  • DAGFileProcessorProcess timeout can cause this
• Option to turn on/off email notifications for SLA miss independent of sla_miss_callback
  • Would allow emails to be turned off if desired, for example, when sla_miss_callback notifies the user in a different way and emails will just duplicate this process. Issue description
• Cleaner code and more defined process
  • Allows the code to be more maintainable
  • Easier to read and understand the flow of when an SLAMiss is raised, and the process that eventually handles the callback and adding a new SLAMiss to the database
• Possibly a new process that handles SLAs? Allows for more detailed SLA logging and debugging
  • This would also prevent the DagFileProcessor process from timing out and causing more issues with SLAs that are hard to debug.

---

Thoughts and discussion below please!

[alex-astronomer]

Also tagging @jedcunningham @kaxil @ashb @ephraimbuddy and @dstandish from Astronomer to get more thoughts and feedback. Feel free to tag anyone important that I missed about this.

[potiuk]

Absolutely agree with rewrite. I think this is a great start of the discussion and I think eventually it should end-up with AIP proposal on the devlist. I think it would also be great to let people know at the devlist that we are discussing it here (which I think is a much better than discussing technical details in the devlist BTW).

[trd-yd]

Hello. Did anyone created a PR to address the issue of the sla_miss_callback not being called as soon as the SLA is missed. If so, which Airflow version has the fix? Thank you

[potiuk]

So far there is this AIP that waits for implementation. And it definitely would need someone who could implement it so if you would like to make PRs to implement it - you are most welcome to start contributing to Airflow. You can announce yourself as redy to help on slack or devlist if you would like to spend time an effort on making it happen.

https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-57+Refactor+SLA+Feature