Shedule details in default_args does not get passed to DAG model

[ddeepwell]

Apache Airflow version

2.4.0

What happened

I have defined a number of things relating to the DAG in the default_args dictionary and expected these parameters to be applied to the DAG in addition to the individual operators within the dag.

My major problem was with "catchup", "schedule_interval", and "start_date". I had specified values for these in "default_args", but the DAG details did not match. This led to runs being scheduled when I did not expect them to.

What you think should happen instead

I might be misunderstanding the use of default_args, but I had expected it to be a means to specify defaults for the DAG. I expected the DAG to have the values of "catchup", "schedule_interval", and "start_date" from "default_args" rather than the defaults.

How to reproduce

Here is my DAG file

from datetime import datetime
from airflow.operators.dummy import DummyOperator
from airflow.models.dag import DAG

# Create defaults settings
default_args = {
    'owner': 'airflow',
    'schedule_interval': "@hourly",
    'start_date': datetime(2022, 9, 10, 10, 0, 0),
    'catchup': False,
}

# Executing Tasks and TaskGroups
with DAG(
        dag_id = "test_pipeline",
        # schedule_interval = default_args['schedule_interval'],
        # start_date = default_args['start_date'],
        # catchup = default_args['catchup'],
        default_args = default_args,
    ) as dag:

    task1 = DummyOperator(task_id = 'task1')
    task2 = DummyOperator(task_id = 'task2')

    task1 >> task2

However, the details page on the UI shows that the catchup, start_date, and schedule_interval was not changed from the defaults.

This is likely the expected behaviour, but it is easy to assume that items in the default_args will be applied to the DAG, especially since it is passed in as an argument when the DAG is generated.

Operating System

MacOS 12.6 and Rocky Linux 8.6

Versions of Apache Airflow Providers

No response

Deployment

Virtualenv installation

Deployment details

No response

Anything else

I would suggest that either values from "default_args" are used to population specifications of the DAG (where appropriate), or change the name of "default_args" to make the misunderstanding more difficult to make.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[ephraimbuddy]

default_args is for tasks not Dag

[ephraimbuddy]

Closing as this is not a bug

[ddeepwell]

My point is that labeling the argument as default_args is potentially confusing because it is passed as an argument to the Dag and therefore could be understood as something that will be applied to the Dag.

It should be named something like default_task_args or default_operator_args to make it apparent that it is applied to tasks within the dag rather than the dag itself.

[potiuk]

My point is that labeling the argument as default_args is potentially confusing because it is passed as an argument to the Dag and therefore could be understood as something that will be applied to the Dag.
It should be named something like default_task_args or default_operator_args to make it apparent that it is applied to tasks within the dag rather than the dag itself.

My opinion It's definitely not a bug. It's how it is defined. There is no ultimate best approach how it should be named. You can have an opinion on that but unless you are ready to pay the cost of changing it, this is an opinion, and it's not a bug, and while it could be named differently, it is not. But defintely should and bug are not appropriate for that.

Just to set expectations here you are free to open a discussion in the devlist. This is that caliber of a change that will impact literally milions of dags "in the wild" and changing mindset of tens of thousands of yousers who used 'default_args" for about 8-9 years, I hope you are prepared to defend that all the cost involved in changing it for all the users. And you are well prepared to lead the change?

This is how open source works - if you do not like something, you not only propose it, but you usually have to defend it, justify cost of change, leading it to completion and then answering questions of users when the change is implemented. Airlfow is created by ~ 2200 users like you and this is usually what happens when you want to improve something that you are responsible for convincing others that this is worth it and then implementing it. You would have to discuss it, get others to agree and change your opinion into a decision by the community.

I am quite sure that there are many more improvements that have far less cost and more impact, but if you would like to start discussion on it (and then providing it will succed) lead the implementation of it, feel free - devlist is where such proposal should happen.

[ddeepwell]

Thanks @potiuk and @ephraimbuddy for the explanations, comments, and directing me to the place to make suggestions.