REST API: dag status stream.

[lromor]

Description

It would be really nice to provide a streaming endpoint so that clients can receive a realtime status update of a running dag instance.

Use case / motivation

from a rest client, instead of polling for updates or creating intermediate task triggers I want to know the status of a running dag.

-->

Are you willing to submit a PR?

It depends by how complex (in terms of time) would be a potential implementation and how much we will invest on airflow in the future.

Related Issues

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[vemikhaylov]

Should it be like a websockets solution or a client can be notified with webhooks? Or would be great to have the both?

I'll be happy to implement such feature if it's necessity's confirmed by maintainers.

[lromor]

Hi @vemikhaylov , thank you for message!

I don't think websockets are necessary. A standard http streamed response should be more than enough.

[ashb]

Yes, this would be nice, but doing it efficiently is a lot of work.

[mik-laj]

Airflow uses a database for communication, which means there is no efficient way to send messages from the scheduler to the webserver. If we wanted to do it right, we would have to have a queue where the scheduler / worker could publish information, and the webserver could respond to these messages and forward to the user. This is an interesting change, but would require a major architectural change. For now, we have to use HTTP/database pooling

[mik-laj]

What do you think about webhook support? #15322 Will this meet your business requirements?

[lromor]

Yes, would be fine for us.