New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Expose an API which can return dagRun status in most optimized time #27782
Comments
Thanks for opening your first issue here! Be sure to follow the issue template! |
Marked as good first issue. |
As discussed in #27765 just exposing updated_at should be enough |
@potiuk Can i take this task up? |
assigned. |
Hey @Adityamalik123. Did you get a chance to start working on this task? There is absolutely no rush, I was just asking in case you need help |
Thanks for checking in @vincbeck. I am planning to get started (and probably wrap this task up) this week, I will definitely reach you out in case i get stuck. |
Hi @Adityamalik123, can you share the documentation for the API you wrote as part of this task? |
Description
Airflow/MWAA does not seem to have any scalable API for returning the status of a dagRun, the APIs states-for-dag-run or list-runs are not scaling well. To fetch the dagRun status, every team seems to have some custom solution using sns_notification or updating the status to some external data store via Airflow callbacks.
The ask is to expose an API which can return dagRun status in most optimized time/by an internal query operation and not a scan.
Discussion #27765
Use case/motivation
My use case is to fetch the Dag status of all the Active runs and update the status tables in the system. There is a poller (with a timeout of 150s configured based on our SLA). The states-for-dag-run API seems to be doing scan operation internally. As the number of DAG runs in system increases, the time to get the status of dagRun increases further. Initially, fetching the status of 100 runs took 2.5 minutes. With increase of dagRuns in the system by 50, the fetch operation to get status for 100 dagRuns is taking more than 5 minutes.
Related issues
NA
Are you willing to submit a PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: