Skip to content

Star Arg Expansion on XComArg Causes DagImport Timeout #21775

@MatrixManAtYrService

Description

@MatrixManAtYrService

Apache Airflow version

2.2.4 (latest released)

What happened

This DAG has a problem: star-expansion can't be used on XComArgs

from airflow import DAG
from airflow.decorators import task
from datetime import datetime

@task
def xcomify():
    return [1, 2]

@task
def add(a, b):
    return a + b

with DAG(dag_id="some_numbers", start_date=datetime(1970, 1, 1)) as the_dag:
    add(*xcomify())

If you check it for errors like python dags/the_dag.py it'll just hang without error output until interrupted.

If you include it in your dags folder, you'll get an error:

Broken DAG: [/usr/local/airflow/dags/star_pos_expansion.py] Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/airflow/models/xcom_arg.py", line 63, in __init__
    def __init__(self, operator: BaseOperator, key: str = XCOM_RETURN_KEY):
  File "/usr/local/lib/python3.9/site-packages/airflow/utils/timeout.py", line 37, in handle_timeout
    raise AirflowTaskTimeout(self.error_message)
airflow.exceptions.AirflowTaskTimeout: DagBag import timeout for /usr/local/airflow/dags/star_pos_expansion.py after 30.0s.
Please take a look at these docs to improve your DAG import time:
* http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/best-practices.html#top-level-python-code
* http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/best-practices.html#reducing-dag-complexity, PID: 169

What you expected to happen

A more specific error, and faster. Something that warns that XComArgs can't be expanded as if they were lists (which they might reference, but which is not known by Airflow at dag-parse time).

How to reproduce

Run a python file containing the dag above. Notice that it runs forever without error.
Add it to your dags list. Notice that you get a timeout error instead of something useful.

Operating System

docker/debian

Versions of Apache Airflow Providers

N/A

Deployment

Astronomer

Deployment details

astro dev start

Dockerfile:

FROM quay.io/astronomer/ap-airflow:2.2.4-onbuild

Anything else

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions