Add average duration markline in task and dagrun duration charts.

[tirkarthi]

Add average duration for run and queued time in task duration and dag run duration. This can help in visualizing potential delays in runs like below dag for one month where there is more sleep time in the last 5 days compared to previous days.

Related: #22132
Ref comment in PR review : #35863 (comment)

from datetime import datetime, timedelta

from airflow import DAG
from airflow.decorators import task
from airflow.operators.empty import EmptyOperator

from datetime import timedelta


with DAG(
    dag_id="task_duration_average",
    start_date=datetime(2024, 1, 1),
    end_date=datetime(2024, 1, 31),
    catchup=True,
    schedule_interval="@daily",
) as dag:

    @task
    def sleeper(ds=None):
        import random, time, pendulum

        end_term = pendulum.datetime(2024, 1, 25)
        execution_date = pendulum.parse(ds)

        if execution_date < end_term:
            duration = random.randint(1, 5)
        else:
            duration = random.randint(10, 20)

        time.sleep(duration)

    sleeper()

[eladkal]

That is very nice!
Is there a way the user can set if they want to see avg, p50, p75 ?
avg may be less useful in case there is 1 very long run and alot of short runs.

[tirkarthi]

The value is calculated by echarts using type option. It only supports average, min, max and median values. But since runduration and queued duration is calculated in javascript the p50, p70 values can also be calculated to be supplied to markline and they can have different colors. As per my understanding p50 and median are equivalent which could be more useful than average.

https://echarts.apache.org/en/option.html#series-bar.markLine.data.0.type
https://math.stackexchange.com/questions/2048470/is-50th-percentile-equal-to-median

Here median is red and average is blue for the same task as above.

[eladkal]

Here median is red and average is blue for the same task as above.

That is cool and useful!
We will need a legend for the distinction.
I am not a UX expert but lets see if we can choose color pallet that will be color blind friendly. We got several requests about this in the past.

As per my understanding p50 and median are equivalent

yes

[dirrao]

That is very nice!
Is there a way the user can set if they want to see avg, p50, p75 ?
avg may be less useful in case there is 1 very long run and alot of short runs.

Nice. Percentile calculations are going to be very useful.

[bbovenzi]

Nice work!

If we do median and average, let's change the tick style so we rely on more than just color. Also, we should make sure the colors are separate from any state colors like red and green.

I also agree that we need a legend.

Later on, we can move more of this logic to the backend and we can do more complex metrics. I would still like to do a historic gantt chart with a box-and-whisker plot of task durations.

[bbovenzi]

But if we do just average, this looks good. Does this also work for Run Duration when "Show Landing Times" is checked?

[tirkarthi]

But if we do just average, this looks good. Does this also work for Run Duration when "Show Landing Times" is checked?

After using this in several charts of different variations I agree with @eladkal that median looks more useful since one of the outlier task that takes more time to run skews the average to be higher when other tasks executed in few minutes and median looks more useful in the chart compared to average value. I am updating the PR to show median value only to skip average.

Please feel free to suggest color that is accessible since I just used the default one provided by echarts.

Thanks

[bbovenzi]

Nice work!
I think I'll move the dag runs summary from the Details panel to below the graph on the Run Duration page. Then we can show appropriate stats.