Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

In triggered SubDag (schedule_interval=None), when clearing a successful Subdag, child tasks aren't run #13295

Closed
yarongolan opened this issue Dec 24, 2020 · 2 comments · Fixed by #14776
Labels
affected_version:2.0 Issues Reported for 2.0 kind:bug This is a clearly a bug priority:medium Bug that should be fixed before next release but would not block a release
Milestone

Comments

@yarongolan
Copy link

yarongolan commented Dec 24, 2020

Apache Airflow version:
Airflow 2.0

Environment:
Ubuntu 20.04 (WSL on Windows 10)

What happened:
After successfully running a SUBDAG, clearing it (including downstream+recursive) doesn't trigger the inner tasks. Instead, the subdag is marked successful and the inner tasks all stay cleared and aren't re-run.

What you expected to happen:
Expected Clear with DownStream + Recursive to re-run all subdag tasks.

How to reproduce it:

  1. Using a slightly modified version of https://airflow.apache.org/docs/apache-airflow/stable/concepts.html#subdags:
from airflow import DAG
from airflow.example_dags.subdags.subdag import subdag
from airflow.operators.dummy import DummyOperator
from airflow.operators.subdag import SubDagOperator
from airflow.utils.dates import days_ago

def subdag(parent_dag_name, child_dag_name, args):
    dag_subdag = DAG(
        dag_id=f'{parent_dag_name}.{child_dag_name}',
        default_args=args,
        start_date=days_ago(2),
        schedule_interval=None,
    )

    for i in range(5):
        DummyOperator(
            task_id='{}-task-{}'.format(child_dag_name, i + 1),
            default_args=args,
            dag=dag_subdag,
        )

    return dag_subdag
	
DAG_NAME = 'example_subdag_operator'

args = {
    'owner': 'airflow',
}

dag = DAG(
    dag_id=DAG_NAME, default_args=args, start_date=days_ago(2), schedule_interval=None, tags=['example']
)

start = DummyOperator(
    task_id='start',
    dag=dag,
)

section_1 = SubDagOperator(
    task_id='section-1',
    subdag=subdag(DAG_NAME, 'section-1', args),
    dag=dag,
)

some_other_task = DummyOperator(
    task_id='some-other-task',
    dag=dag,
)

section_2 = SubDagOperator(
    task_id='section-2',
    subdag=subdag(DAG_NAME, 'section-2', args),
    dag=dag,
)

end = DummyOperator(
    task_id='end',
    dag=dag,
)

start >> section_1 >> some_other_task >> section_2 >> end
  1. Run the subdag fully.
  2. Clear (with recursive/downstream) any of the SubDags.
  3. The Subdag will be marked successful, but if you zoom into the subdag, you'll see all the child tasks were not run.
@yarongolan yarongolan added the kind:bug This is a clearly a bug label Dec 24, 2020
@boring-cyborg
Copy link

boring-cyborg bot commented Dec 24, 2020

Thanks for opening your first issue here! Be sure to follow the issue template!

@potiuk potiuk added this to the Airflow 2.0.1 milestone Dec 24, 2020
@yarongolan
Copy link
Author

yarongolan commented Jan 13, 2021

Just clarifying that this breaks the SUBDAG functionality completely

@vikramkoka vikramkoka added the affected_version:2.0 Issues Reported for 2.0 label Jan 14, 2021
@kaxil kaxil added the priority:medium Bug that should be fixed before next release but would not block a release label Jan 19, 2021
@kaxil kaxil modified the milestones: Airflow 2.0.1, Airflow 2.0.2 Feb 4, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affected_version:2.0 Issues Reported for 2.0 kind:bug This is a clearly a bug priority:medium Bug that should be fixed before next release but would not block a release
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants