UI: Add data interval override option for manual DAG runs

[bastienmenis]

This PR modifies the modal for triggering a single DAG run manually. The primary change is to allow the user to specify the data interval manually.

List of changes:

• If the DAG has a timetable, added a section in the form for specifying the data interval. There are two options:
  • "Interred from Logical Data and Timetable" (default): This will set data_interval_start and data_interval_end as null in the API call (same as current behaviour
  • "Specify Manually": When selected, displayed 2 date/time pickers for the user to enter the date range. There is some validation: Both dates are mandatory and the end date should be greater or equal to start date.
• Moved "Logical Date" and "Data Interval" out of the "Advanced Options" section.
• Split TriggerDAGForm into two components, breaking TriggerDGAdvancedOptions to its own file.
• Added some new localisation strings to components.json. I am not familiar with the process for adding translations, or any related automations; I have only included the English one.

*Screenshots
DAG with time table - Default option:

DAG with time table - Manual interval:

DAG without time table:

Reasoning behind this change

• data_interval_start and data_interval_end are optional when making the API call. This is not transparent to the user, I think it would be better UX if the user can see and alter the data interval that will be used when triggering a DAG.
• There is a comment in here that seems to infer that the data interval should always be defined when DAGs are automatically scheduled. If it is the case, it would make sense to have the same requirement for manual triggers.
• About putting "Logical Date" and "Data Interval" outside of "Advanced Options": This change might be controversial, I believe that those are such key parameters to DAG runs that they deserve being always displayed to the user. That makes the modal bigger as a consequence but it is still smaller than with the "Backfill" option.

** Possible future improvements **
When selecting "Inferred Data Interval", it would be better UX to display to the user what this inferred interval is. It could be achieved by adding an API endpoint could be added to fetch the data interval for a specified logical date.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

• Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
• Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[jscheffl]

Is there a practical use for data interval start/end in your use case that you need it for? Other than adding consistentcy?

In my view the parameters are mainly covered for histporical scheduled use cases to define the data intervals. Manually triggering a Dag is in almost all cases (looking for an exception) out of such interval bounds.
I would expect if you want to manually run such things that it is a special advanced case or the option to make a backfill are the right way to go.

I am asking for a real use case because the trigger form is already over-loaded with options (in my view) and I fear that there are more users confused by the options than it brings a benefit. I'd accept it if there is a real use case driving this addition. There might be.

[bastienmenis]

Is there a practical use for data interval start/end in your use case that you need it for? Other than adding consistentcy?

In my view the parameters are mainly covered for histporical scheduled use cases to define the data intervals. Manually triggering a Dag is in almost all cases (looking for an exception) out of such interval bounds. I would expect if you want to manually run such things that it is a special advanced case or the option to make a backfill are the right way to go.

I am asking for a real use case because the trigger form is already over-loaded with options (in my view) and I fear that there are more users confused by the options than it brings a benefit. I'd accept it if there is a real use case driving this addition. There might be.

@jscheffl Hi Jens, thanks for your response. Indeed, I do have a use-case in mind!

We have a DAG responsible for doing some data aggregation over some time series data. It is scheduled on a custom timetable: It runs once a day, and uses a data interval covering the previous 72 hours.

What we would like to do is run a backfill, but a "normal" backfill would be wasteful as it would process data for each day three times (since the data intervals for consecutive runs overlap). So it would be useful to be able to trigger the DAG manually but with a custom data interval, that is different than the default one set in the timetable.

Of course we could create a separate DAG just for the purpose of backfilling, but that doesn't feel as neat as using the same DAG.

Here is how I understand the use cases for manually triggering DAG runs:

• If the user want to trigger runs as per the timetable, they use the "Backfill" option
• If the user want to trigger a run independently from the timetable, they use the "Single run" option. In that case, since the logic that determines the data interval for each run is tied to the timetable, it feels that prompting the user for the data interval makes sense. I'd actually suggest that those field could sit in the main form, rather than in the "Advanced option" section.

Of course that is my interpretation based on our narrow use-case 😆. We haven't been using Airflow for long so I understand that our usage might differ from the majority of users

[jscheffl]

I am asking for a real use case because the trigger form is already over-loaded with options (in my view) and I fear that there are more users confused by the options than it brings a benefit. I'd accept it if there is a real use case driving this addition. There might be.

@jscheffl Hi Jens, thanks for your response. Indeed, I do have a use-case in mind!

Thanks. This sounds a bit like a special case. So calling via API might be resonable for you, not 100% sure if it makes sense to extend the UI for such niece use case.
@dstandish & @uranusjr You had a bit of discussions as lead about scheduling and Backfil, do you have an option about this (missing) feature? I would be +0.

[ashb]

Not speaking to the implementation, but if the API allows specifying data_interval than I'm +1 to allowing the UI user to specify it too.

If we think most users won't want this we could think about hiding it behind an "Advanced" collapsed-by-default section.

So it would be useful to be able to trigger the DAG manually but with a custom data interval, that is different than the default one set in the timetable.

This is a use case I've had in the past, and was in some of the inspiration for creating explicit data_interval range distinct from the logical date of the dag run.

[pierrejeambrun]

Yes the API accepts both data_interval_start and data_interval_end, and I'm fine adding that to the form.

Can you address the remaining comments and fix the CI so we can move forward with the PR please?

[ddeath]

My usecase is probably also "special".

We are in dev phase of project and we would like to have ability to trigger custom data interval just for testing. We test dag locally, then we deploy it to AWS and we would like to test it in AWS as well.
Since project is in early phase not always we have consistent data for each time frame so making data interval wider help us to test dag.
API Access is restricted in deployed AWS case so it would be harder for data engineers to use it.

[pierrejeambrun]

@bastienmenis Any chance you can address the comments so we can get his merged?

Also address the backend part that need some tweaking

[bastienmenis]

@bastienmenis Any chance you can address the comments so we can get his merged?

Also address the backend part that need some tweaking

@pierrejeambrun I will try to have a look at this at some point this week.

[bastienmenis]

I have reworked the changes and committed a bigger refactor. I believe it would be a significant UX improvement. The PR description has been updated accordingly (with screenshots this time!). I didn't update the backend as it is not strictly related to the changes in this pull request.

[pierrejeambrun]

Nice thanks.

[boring-cyborg]

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.