Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TaskFlow DAGs don't register implicitly #29009

Closed
1 of 2 tasks
BasPH opened this issue Jan 18, 2023 · 2 comments
Closed
1 of 2 tasks

TaskFlow DAGs don't register implicitly #29009

BasPH opened this issue Jan 18, 2023 · 2 comments
Labels
area:core kind:bug This is a clearly a bug

Comments

@BasPH
Copy link
Contributor

BasPH commented Jan 18, 2023

Apache Airflow version

2.5.0

What happened

TaskFlow @dags don't register without explicitly calling the function that defines it.

What you think should happen instead

My understanding is that since Airflow 2.4 DAGs don't require explicit registration as a global object anymore (this is also mentioned in https://airflow.apache.org/docs/apache-airflow/stable/tutorial/taskflow.html#instantiate-a-dag). However, this seems to only be true for the classic style and not for taskflow style DAGs.

How to reproduce

import datetime

from airflow import DAG
from airflow.decorators import dag, task
from airflow.operators.empty import EmptyOperator

with DAG(dag_id="no_global_dag_registration", start_date=datetime.datetime(2023, 1, 1), schedule=None):
    EmptyOperator(task_id="test")


@dag(start_date=datetime.datetime(2023, 1, 1), schedule=None)
def no_global_dag_registration_taskflow():
    @task
    def test():
        pass

My understanding both styles should register the DAG, however only DAG "no_global_dag_registration" does. For the taskflow-style DAG I have to call no_global_dag_registration_taskflow() for it to register.

Operating System

varying

Versions of Apache Airflow Providers

No response

Deployment

Other

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@BasPH BasPH added kind:bug This is a clearly a bug area:core labels Jan 18, 2023
@Taragolis
Copy link
Contributor

Taragolis commented Jan 18, 2023

Maybe doc has not clear description, but it actually mentioned how it work.

Changed in version 2.4: It’s no longer required to “register” the DAG
into a global variable for Airflow to be able to detect the dag if that DAG 
is used inside a with block, or if it is the result of a @dag decorated function.

My understanding of differences

# Before Airflow 2.4
dag = decorated_function()
# or
_ = decorated_function()

with DAG(...) as daaaag:
    ...


# Since Airflow 2.4
decorated_function()

with DAG(...):
    ...

@BasPH
Copy link
Contributor Author

BasPH commented Jan 18, 2023

Doh! I'll make a PR for the docs to clarify this.

@BasPH BasPH closed this as completed Jan 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:core kind:bug This is a clearly a bug
Projects
None yet
Development

No branches or pull requests

2 participants