Add scheduler by Dataset and/or interval

[FelipeArisi]

Description

In recent features it is possible to add dataset scheduling with conditional expressions (https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/datasets.html#advanced-dataset-scheduling-with-conditional-expressions)

The idea would be to add a time-based schedule to these expressions as well. For example: schedule=['0 0 1 * *' | Dataset(path)]

Use case/motivation

There are some events involving many Datasets as triggers. If a Dataset is late or gives an error, the entire pipeline will depend on it. The idea would be to add a time condition to start the DAG anyway.

Another problem is, if all the datasets are ready and the pipeline is ready, the pipeline must run after a certain time.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[tirkarthi]

Does DatasetOrTimeSchedule help your use case?

https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/timetable.html#dataset-event-based-scheduling-with-time-based-scheduling

[FelipeArisi]

I didn't know this one! It will work, thank you very much!

[Saratqpy]

Really interesting proposal —I’ve been working on a somewhat related scheduling case and wanted to see if this has come up elsewhere.

In my situation, the requirement is kind of the opposite: I need the DAG to trigger only when both a specific dataset has been updated AND a scheduled cron tick has occurred. Basically an AND condition instead of OR.

The reason is to avoid premature runs — for example, if the dataset is delayed or missing, we want to wait until the next cron window and make sure the data is ready.

I’m curious whether this kind of logic (cron AND dataset) is something that’s been discussed or considered for future versions?

Thanks!