-
Notifications
You must be signed in to change notification settings - Fork 2.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(snowflake): add config option to specify deny patterns for upstreams #7962
feat(snowflake): add config option to specify deny patterns for upstreams #7962
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A couple minor comments but LGTM. I'd really like to get some focused connector tests around this at some point (if priorities allow), because regexes can be hard to debug
): | ||
raise ValueError( | ||
"Can not perform Deletion Detection, Lineage Extraction, Profiling without extracting snowflake technical schema. Set `include_technical_schema` to True or disable Deletion Detection, Lineage Extraction, Profiling." | ||
"Can not perform Deletion Detection, Profiling without extracting snowflake technical schema. Set `include_technical_schema` to True or disable Deletion Detection, Lineage Extraction, Profiling." |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you want to remove the reference to lineage extraction at the end of the comment here?
) | ||
|
||
|
||
def create_deny_regex_sql_filter( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Although will be annoying, definitely something we should test in connector tests in some way
@@ -426,20 +447,30 @@ def view_lineage_history( | |||
) = 1 | |||
""" | |||
|
|||
# Note on use of `upstreams_deny_pattern`` to ignore temporary tables: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Extra ` here
AND upstream_column_table_name NOT LIKE '%.FIVETRAN\\_%\\_STAGING.%' | ||
AND upstream_column_table_name NOT LIKE '%\\_\\_DBT\\_TMP' | ||
AND upstream_table_name NOT LIKE '%.FIVETRAN\\_%\\_STAGING.%' | ||
AND upstream_table_name NOT LIKE '%\\_\\_DBT\\_TMP' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We escape underscores here but not in DEFAULT_UPSTREAMS_DENY_LIST
, just want to make sure this is ok.
Followup on #7894
Checklist