Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update SnowflakeOperator and UcToSnowflakeOperator to accept sql_file and custom dbx sql #116

Closed
wants to merge 30 commits into from

Conversation

bsangars
Copy link
Contributor

@bsangars bsangars commented May 15, 2024

Update Snowflake and UctoSnowflakeOperator Operators to accept sql file and custom db_sql queries as inputs

Description

It is an enhancement to pass sql_file to Snowflake Operator and custom sql to extract data from Unity catalog.

Below are the changes implemented as part of this PR

  1. sql_file parameter is added to SnowflakeOperator
  2. Raise an exception if both query_string and sql_file is passed
  3. Add an additional parameter dbx_sql to UcToSnowflakeOperator to use custom sql when extracting data from Unity Catalog
  4. Update Mandatory keys to expect one Of dbx_sql or (dbx_catalog, dbx_database, dbx_table)
  5. Add an additional (optional) parameter write_mode to specify while writing to Snowflake from Unity Catalog

Related Issue

#115

Motivation and Context

This is an enhancement to account for below use cases

  1. passing sql file instead of query string ( which would be an additional hop everytime we call operator)
  2. passing in custom sql to extract data from Unity catalog. Current code only accepts guard rail sql filters against select statements. using this enhancement user has better control on the source

How Has This Been Tested?

Tested the updated operators for multiple scenarios

  1. using the sql file
  2. using query string
  3. using both (negative testing)
    Tested the UcToSnowflakeOperator for below scenarios
  4. Incremental
  5. Full load with dbx_sql and tables
  6. write mode

Screenshots (if appropriate):

Make check result
Screenshot 2024-05-15 at 11 51 33 AM
Updated brickflow operators workflow
image
image
image
image

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have read the CONTRIBUTING document.
  • I have added tests to cover my changes.
  • All new and existing tests passed.

Copy link
Contributor

@ckonuganti ckonuganti left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changes related to the new feature of running sql files instead of building or passing queries looks good to me .

@bsangars bsangars closed this May 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants