task_queued_timeout Override in DAGs

[hbc-acai]

Hello community,

I am not sure if this already is supported. I think it is in general useful to allow task_queued_timeout to be overridden on the DAG level. Consider these 2 DAGs:

• Considering a DAG that has a lot of parallel tasks. When there are not enough worker, the tasks can be queued for long time. This could be a valid setup, in particular it can be dedicated to a separated celery queue. In this case, I want to have a high task_queued_timeout.

• On the same time, one might have a "lower priority DAG" ( say clean up some old files") that runs only there is a free worker, otherwise it is OK to mark it as fail after the task_queued_timeout. In this case, I prefer task_queued_timeout to be low.

In order to allow them to co-exist, I think a task_queued_timeout override variable to DAGs is going to be useful.

[potiuk]

No, not possible now - but if you would like to implement it - feel free. PRs are always welcome. It would be rather complex to implement, because you would have to store expected timeout per task and query them per task rather than fail all the tasks that have been queued before (now() - task_queued_timeout) as it is implemented now. Being able to distinguish different tasks / dags for that makes it quite a bit more complex, but if you feel like providing PR with robust/performant implementation, I see no general problem with it.

[hbc-acai]

Sounds good. I do not have the expertise yet to implement. Can you point it to me in general how it works if my firm wants to sponsor the community to implement a proposal like this?

[potiuk]

Sure. Github Sponsors is one way. Paying someone directly for their time on working on it is another. There are platforms like https://www.ringerhq.com/

[hbc-acai]

Thank you