-
Notifications
You must be signed in to change notification settings - Fork 13.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix do_xcom_push=False bug in SnowflakeOperator #29599
fix do_xcom_push=False bug in SnowflakeOperator #29599
Conversation
Test DAG to fix (see linked issue for Test DAG Demonstrating error):
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since this method is defined and called in the super class SQLExecuteQueryOperator
, I prefer to add the check in this bloc to avoid adding it in all the sub classes, and please add a unit test to check if everything works as expected.
@hussein-awala - the bug is specifically in the SnowflakeProvider - the base class just does a no-op with --- a/airflow/providers/common/sql/operators/sql.py
+++ b/airflow/providers/common/sql/operators/sql.py
@@ -265,6 +265,9 @@ class SQLExecuteQueryOperator(BaseSQLOperator):
return_last=self.return_last,
**extra_kwargs,
)
+ # Handle do_xcom_push=False
+ if output is None or output == [None]:
+ return []
if return_single_query_results(self.sql, self.return_last, self.split_statements):
# For simplicity, we pass always list as input to _process_output, regardless if
# single query results are going to be returned, and we return the first element
--- a/airflow/providers/snowflake/operators/snowflake.py
+++ b/airflow/providers/snowflake/operators/snowflake.py
@@ -108,9 +108,6 @@ class SnowflakeOperator(SQLExecuteQueryOperator):
results: Optional[list[Any]],
descriptions: list[Sequence[Sequence] | None]
) -> list[Any]:
- # Handle do_xcom_push=False
- if results is None or results == [None]:
- return []
validated_descriptions: list[Sequence[Sequence]] = []
for idx, description in enumerate(descriptions):
if not description: |
Unfortunetly Statics Check and also required tests for avoid regression. |
I think adding a condition on the --- a/airflow/providers/common/sql/operators/sql.py
+++ b/airflow/providers/common/sql/operators/sql.py
@@ -265,6 +265,8 @@ class SQLExecuteQueryOperator(BaseSQLOperator):
return_last=self.return_last,
**extra_kwargs,
)
+ if not self.do_xcom_push:
+ return None
if return_single_query_results(self.sql, self.return_last, self.split_statements):
# For simplicity, we pass always list as input to _process_output, regardless if
# single query results are going to be returned, and we return the first element What do you think? |
Oh! Brilliant! Just short circuit and skip all the rest. That makes sense. Not sure why I was worried that method might be needed for side-effects other than xcom |
b9e40b5
to
a85537c
Compare
Did a quick rebase just to clean up commits. |
a85537c
to
29138f2
Compare
closes: #29593
Adds a guard check to short-circuit out of
_process_output
in the event of no output (such asdo_xcom_push=False
)