Add Snowflake Data Quality Operators #17738
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fix `airflow celery stop` to accept the pid file. (apache#17278) Fix isort (apache#17330) Fix failing celery test (apache#17337) This change fixes failing test due to mocking
When dest_key is given as a full s3:// filepath and dest_bucket is also provided, operator throws an error in its init() method. A test is added for this case.
… given When dest_key is given as a full s3:// file path, the parent operator does not expect a dest_bucket. A TypeError is raised when dest_bucket is not None in this case.
denimalpaca
requested review from
mik-laj,
potiuk and
turbaszek
as code owners
boring-cyborg
bot
added
area:providers
provider:amazon
denimalpaca
force-pushed
the
feature/snowflake_dq_operators
branch
from
d91b750 to
a11f669
Compare
Member
|
Something very wrong happened during this rebase :) |
Contributor
Author
Yes it did.... I am trying to figure it out. Hopefully all the work wasn't lost. |
Member
GIT never forgets (unless you run Garbage-Collection). Try |
Added three new tests, one for each new Snowflake Operator, in the same style as the BigQuery check operators. These tests ensure the hook is properly called. Not sure how helpful that really is, but it's coverage, and I have used these new operators (well, two of them) in an actual DAG with a live connection, and they're working. The snowflake.py file was touched to fix the method name I changed when trying other solutions, it was restored to make the SnowflakeOperator pass its test again. Add exception when incompatible parameters are given to Operator Throw a TypeError when the dest_bucket is provided and the dest_key already specifies a full s3:// file path. This check does not occur at higher levels, so it must be enforced here. When dest_bucket is not needed, it should be None and not a string, hence the TypeError.
denimalpaca
force-pushed
the
feature/snowflake_dq_operators
branch
from
4e1589a to
c29e6be
Compare
The SnowflakeCheckOperator, SnowflakeValueCheckOperator, and SnowflakeIntervalCheckOperators are added as subclasses of their respective SQL Operators. These additions follow the conventions set in the BigQueryOperators subclassing from the same SQL_CheckOperators. This decision was made despite the existing SnowflakeOperator, as subclassing both SnowflakeOperator and a SQL_CheckOperator caused a couple issues: 1. Both operators have an execute method already, so some precaution would be necessary to choose the correct one. 2. An issue with the `sql` parameter occured, where if it was given as a kwarg (sql=sql), then the BaseOperator init() would through an exception for missing args, as if it weren't being added to **kwargs correctly.
denimalpaca
force-pushed
the
feature/snowflake_dq_operators
branch
from
c29e6be to
6e2b03b
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add three new Snowflake operators based on SQL Checks
The SnowflakeCheckOperator, SnowflakeValueCheckOperator, and
SnowflakeIntervalCheckOperators are added as subclasses of their respective
SQL Operators. These additions follow the conventions set in the BigQueryOperators
subclassing from the same SQL_CheckOperators.
closes: #17694