Logging Details Message to Address Scheduler Heartbeat Loss and Downtime Challenges

[BShraman]

Description

Data engineers often encounter challenges related to scheduler heartbeat loss and scheduler downtime while using Apache Airflow. These issues can cause disruptions and delays in critical data pipelines. In order to streamline troubleshooting and enhance efficiency, it would be valuable to add new features to Apache Airflow that enable logging of detailed messages regarding these problems.

By logging comprehensive and specific messages related to scheduler heartbeat loss and downtime incidents in airflow UI or in backend database, data engineers can quickly identify the root causes of issues, significantly reducing the time required for troubleshooting. This feature would provide clear visibility into the problems and empower data engineers to take prompt action in resolving them.

The benefits of implementing this feature in Apache Airflow would include but not limited to

1. Accelerated Troubleshooting
2. Enhanced Efficiency
3. Improved Collaboration

Slack Threads : https://app.slack.com/client/TCQ18L22Z/threads

Thanks,

Use case/motivation

No response

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[jscheffl]

Hi @BShraman thanks for filing this issue. I marked it as improvement as it something that enhances the situation.

I understand that users expect a very reliable platform. To better understand your demand, can you specify a bit more your expectation? Would you "just log" internal warnings? Shall logs be structured and machine readable? To be integrated in any kind of (existing?) monitoring solution or just exposing data? Do you expect also some kind of alarming? If yes, how?

Can you describe what kind of disruptions and impact you have noticed or are users afraid of? Do you have a dedicated support for the platform? Are you searching for (commercial) support delivering an SLA? Or do you expect data engineers to fix themself? Do you have an operations team? Are you using a Kubernetes and the standard helm chart which includes Liveness-probes for standard self-healing?

[BShraman]

Hi @jens-scheffler-bosch
As you're aware, identifying the cause of Scheduler heartbeat issues can be challenging, given that Airflow lacks a dedicated logging mechanism for these messages. Implementing a mechanism to log this exposed data into the backend database would significantly assist engineers during troubleshooting.

Troubleshooting these errors can often be time-consuming, and the lack of historical data to generate reports when jobs are impacted by Scheduler heartbeats adds to the complexity. Currently, I'm using the standard Helm chart with liveness-probes to restart the container, which is deployed in Kubernetes and uses the Kubernetes Executor for each task.

Given our DevOps practices, engineers handle the deployment and product support simultaneously. It would be beneficial to explore solutions that can enhance visibility and facilitate the root cause analysis of Scheduler heartbeat issues.

Open to your insights and suggestions ...

[potiuk]

As you're aware, identifying the cause of Scheduler heartbeat issues can be challenging, given that Airflow lacks a dedicated logging mechanism for these messages. Implementing a mechanism to log this exposed data into the backend database would significantly assist engineers during troubleshooting.

Surely that's a nice improvement. We likely approve PRs for such small quality-of-live improvements weekly or daily. It's great to get those improvements coming from > 2500 contributors we have

Open to your insights and suggestions ...

You seem to know what exactly would have helped you and what kind of messages and troubleshooting would be needed. Undoubtedly (you wrote it) you had seen many of those kind of problems in the past. So likely you (and your team) are the best people in the world to add such logging. Adding such logging messages is usually a small PR and if you look at the opened PR there are quite a few of those that are already handling some parts of the situation for exampe see this PR: #31996 connected to this issue: #31810 which is precisely about improving some logging and behaviours around heartbeating. This is really (as I see it) duplicate of #31810 - so I will close that one as duplicate - and if you feel like there is more to it better to discuss it there.

So when you ask for suggestions. My suggestion is .... Just do it @BShraman. Contribute PR with what you want to propose. This is the best way forward.

Open PRs or add review comments in the existing ones. This way you can become one of the contributors to Airflow and this is the best way to turn it into reality. It's as simple as that. There isn not even a need to have an issue and discuss it upfront. We do not require (or even expect) issues for such kind of small quality-of-life improvements, we do expect people to propose PRs and then all kind of discussions around it are easier if we can talk over an actual proposal in the code.