Automatic workflow duration prediction

[alexec]

Summary

Users would like to be able to understand the how long similar workflows took to see if they start to take longer.

Motivation

When I'm using workflow templates, is this template taking longer to run?

Proposal

Add summary data of recent runs of the template status?

Related to #1658, #3557

---

Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

[alexec]

@dtaniwaki @sarabala1979 would it be possible do you think?

[alexec]

could we solve by just adding duration to list view?

[dtaniwaki]

I think we can have stats of workflow templates in RDB or somewhere and calculate the average, median and etc. and show it in the list view.

But it may affect the performance of the controller. We may need to has an option to disable the feature.

[dtaniwaki]

Actually, we want to use these stats in the scheduler and optimize the scheduling algorithm. For example, there is few resources available and a pod is waiting with high priority, now, a low priority pod came. If we can predict how long the pod runs, we can let it run before the high priority waiting pod.

[alexec]

This specific use case I want to solve - seeing historical durations is shown in the UI already.

[alexec]

There might be other things we want to aggregate, so I'll leave this open.

[alexec]

#3558 produces a way to determine likely execution time and to dig into problems.

[alexec]

Similar to #1658.

[tszngai]

How do I enable this feature to see the estimation time on my UI?

[alexec]

It is automatically enabled. It estimates based on the most recently completed workflow that used the same workflow template or cron workflow.

[tszngai]

Thanks for the quick reply. I'm using v2.12.5 and I don't see the estimation time on it:

[tobisinghania]

Is it also possible to retrieve those metrics via the prometheus metrics endpoint?

[alexec]

Short answer - no.
Long answer - there are metrics related to workflow execution that might help you.

[tobisinghania]

Thx for your answer....although it's not the one I was hoping for ;)
Unfortunately I have not been able to create a similar diagram as you have in the UI with using the duration workflow metric together with a gauge in grafana (this might be only because a lack of skill on my end, though).

Maybe I will create the metric via a workaround which is using your api and pushes to prometheus...

Thank you