-
Notifications
You must be signed in to change notification settings - Fork 16.4k
Closed
Labels
Description
Apache Airflow version
2.2.4 (latest released)
What happened
This DAG has a problem: star-expansion can't be used on XComArgs
from airflow import DAG
from airflow.decorators import task
from datetime import datetime
@task
def xcomify():
return [1, 2]
@task
def add(a, b):
return a + b
with DAG(dag_id="some_numbers", start_date=datetime(1970, 1, 1)) as the_dag:
add(*xcomify())
If you check it for errors like python dags/the_dag.py it'll just hang without error output until interrupted.
If you include it in your dags folder, you'll get an error:
Broken DAG: [/usr/local/airflow/dags/star_pos_expansion.py] Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/airflow/models/xcom_arg.py", line 63, in __init__ def __init__(self, operator: BaseOperator, key: str = XCOM_RETURN_KEY): File "/usr/local/lib/python3.9/site-packages/airflow/utils/timeout.py", line 37, in handle_timeout raise AirflowTaskTimeout(self.error_message) airflow.exceptions.AirflowTaskTimeout: DagBag import timeout for /usr/local/airflow/dags/star_pos_expansion.py after 30.0s. Please take a look at these docs to improve your DAG import time: * http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/best-practices.html#top-level-python-code * http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/best-practices.html#reducing-dag-complexity, PID: 169
What you expected to happen
A more specific error, and faster. Something that warns that XComArgs can't be expanded as if they were lists (which they might reference, but which is not known by Airflow at dag-parse time).
How to reproduce
Run a python file containing the dag above. Notice that it runs forever without error.
Add it to your dags list. Notice that you get a timeout error instead of something useful.
Operating System
docker/debian
Versions of Apache Airflow Providers
N/A
Deployment
Astronomer
Deployment details
astro dev start
Dockerfile:
FROM quay.io/astronomer/ap-airflow:2.2.4-onbuild
Anything else
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Reactions are currently unavailable