-
Notifications
You must be signed in to change notification settings - Fork 14.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
100+ DAGs fail to import after update to 2.7.1 due to "params without default values" #34227
Comments
looks like it was explicitly changed here #33141 |
Yes. I think the behaviour when param value was set to @jens-scheffler-bosch @hussein-awala @uranusjr -> WDYT? |
IMHO we should consider the params defined without a default value as required params, and the fail/exception should be raised when trying to create a DagRun without providing these required params. |
WRONG: I assume currently it is caused by an inconsistency. Because the check assumes to have a
I am currently looking though the history of the code, understanding when the problem has been introduced. I volunteer for a fix but want to understand the history first :-D |
Apache Airflow version
2.7.1
What happened
120 DAGs broke when I updated to Airflow 2.7.1. I had to revert to 2.7.0 again, so the tests below are from a local deployment.
The error message is the same for all DAGs:
After testing, I found that if I change the blob_prefix param to "" instead of None, the DAG imports successfully. I have various other params which are None that I will probably need to assign default values for (in reality, the function which Airflow runs handles cases if the value is None, but I guess I need to add a default in the DAG). All 120 DAGs with errors had a field like this in the "params" dictionary I pass to Airflow params.
What you think should happen instead
These are legacy DAGs which have run for a long time without changes (old Airflow instance), so I did not expect an upgrade to break them.
How to reproduce
I believe the root cause is passing a param with the value None to the DAG params:
Operating System
Ubuntu 22.04
Versions of Apache Airflow Providers
Providers info
apache-airflow-providers-celery | 3.3.3
apache-airflow-providers-common-sql | 1.7.1
apache-airflow-providers-docker | 3.7.4
apache-airflow-providers-ftp | 3.5.1
apache-airflow-providers-google | 10.7.0
apache-airflow-providers-http | 4.5.1
apache-airflow-providers-imap | 3.3.1
apache-airflow-providers-microsoft-azure | 6.3.0
apache-airflow-providers-mysql | 5.3.0
apache-airflow-providers-odbc | 4.0.0
apache-airflow-providers-openlineage | 1.0.2
apache-airflow-providers-postgres | 5.6.0
apache-airflow-providers-redis | 3.3.1
apache-airflow-providers-sftp | 4.6.0
apache-airflow-providers-sqlite | 3.4.3
apache-airflow-providers-ssh | 3.7.2
Deployment
Other Docker-based deployment
Deployment details
Docker Swarm using docker stack deploy
Anything else
DAGs fail to import when deployed on 2.7.1
Are you willing to submit PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: