Add triggered_by field to DAG Run model to distinguish the source of a trigger

[molcay]

Add the triggered_by field to the DAG Run model to distinguish the source of a trigger.
This is for the discussion in #37087.

Currently this PR contains only the backend changes. No changes done on the frontend. We can discuss and add some UI elements (different border types, colors, or icons etc.) to have a more distinct look on the UI as well.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[Taragolis]

airflow/api/client/* files exists for mainly to support legacy experimental API, I don't think we need to extend those

[molcay]

Since they currently exist, I thought that I need to extend those files also. I can revert those changes.

[molcay]

I reverted changes for the airflow/api/client folder. Since some of the clients are using other methods, I only touched on fixing the related places

[uranusjr]

This does not read like English

[molcay]

I changed the comment wording

[uranusjr]

This should probably be say TIME or TIMETABLE or something to distinguish from DATASET. Maybe something like TIME_TRIGGERD and EVENT_TRIGGERED would work too.

[molcay]

Actually, I was thinking that the SCHEDULER value is for timetable related triggering and DATASET value is for Dataset related triggering.
I saw this image on the documentation and it states "Triggered by Datasets" so I was thinking the DATASET is a good value for this.

[ephraimbuddy]

Do we still need run_type? I think it should be removed in a follow-up PR and this triggered_by is used in places where run_type is used

[uranusjr]

I don’t disagree with using triggered_by, but am not sure about removing run_type due to compatibility issues. And logically the two might not line up exactly…? (In the sense that many runs triggered by the same component may have different types, and vice versa.)

[molcay]

Side note: this PR and the triggered_by field is a result of an attempt to add a new run_type in #37087. When we think about the backward compatibility I think it makes sense to keep the run_type field.

[jscheffl]

I was taking a look to this PR and see that the field will add valuable information. But not completely as besides adding the marker as type we still would not know "which CLI", which "User" in the UI or which DAG run/Operator triggered a DAG run. So it adds parts of traceability with a lot of changes in the code (1000+LoC code changed).

Have you considered adding the details for your needs as traceability to the LOG / audit log table? This would just be another type in the audit logs and would prevent the need to extend the DB scheme for the DAG run table.

[molcay]

Hi @jscheffl,

Thank you for your message and your point of view. The thing is that at the beginning we were trying to address a very specific use case to distinguish operator-triggered runs and scheduled runs. While trying to find a solution, I implemented #37087 but we had a discussion on that PR and as a conclusion we decided to add the triggered_by field to distinguish the source for the dag run. It might be handy to know which component triggered the DAG Run.

Since we can trigger a DAG run with different sources, it was making sense to store this information in the table.
To be able to distinguish who started the run from the UI, or which operator/task triggered the run in the code a bit more excessive, at least at this point. Maybe adding a record to LOG/audit log table can be useful to know which specific task triggered the run or who started the run from the UI.