Tracking execution progress?

[orcaman]

Hi friends,

Is there a way of getting a list of tasks that are currently running? Our DAG is comprised of hundreds of tasks. We have logs for state change handling, so we can see when tasks have started and when they have finished. We could write a logs query to pick up the tasks that have started but haven't finished and thus presume that they are running - but is there a better way of doing this within Prefect that we're missing?

Thanks

[cicdw]

Hi @orcaman! This sort of pattern requires an external monitoring system which keeps track of Task states, which is precisely where Prefect Cloud comes in. In Cloud, you can query the database for which tasks are in a Running state, and for those that are in a Running state you can inspect the latest heartbeat time to verify that the task process is still alive.

[orcaman]

I guess one could implement by keeping track in-memory?

Anyways, cool. We are looking forward to the cloud to be available :-)

[cicdw]

Awesome!

And yea, if you want to track your task's states in a custom way I usually recommend people implement their own state handler (which it sounds like you've essentially done).