-
Notifications
You must be signed in to change notification settings - Fork 40
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add merge_operator for Redshift database #753
Conversation
Codecov Report
@@ Coverage Diff @@
## main #753 +/- ##
==========================================
+ Coverage 93.52% 93.57% +0.05%
==========================================
Files 43 43
Lines 1776 1822 +46
Branches 216 226 +10
==========================================
+ Hits 1661 1705 +44
Misses 93 93
- Partials 22 24 +2
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. |
05b2e1c
to
7564b4e
Compare
Redshift does not enforce constraints although it accepts those in the table DDL. Ref: https://docs.aws.amazon.com/redshift/latest/dg/t_Defining_constraints.html As a result, in case of conflicts, e.g. uniqueness constraint, it does allow insertion of duplicate values and expects you to clean up duplicated. Additionally, for merge redshift documentation suggests making use of a staging table. Ref: https://docs.aws.amazon.com/redshift/latest/dg/t_updating-inserting-using-staging-tables-.html The merge implementation involves the following steps: i. Create a staging table having schema identical to the target table ii. Load data from source table into stage table by taking care of mapping source table column to stage table column using the given source_to_target_columns_map. iii. Handle conflicts. There are two strategies to handle conflict and they are explained in detail as below: a. ignore: To ignore rows in case of conflicts, we remove the matching rows from stage table that are present in the target table. b. update: To update rows in case of conflicts, we remove the matching rows from the target table that are present in the stage table. iv. Load the stage table data into target table
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
Co-authored-by: Utkarsh Sharma <utkarsharma2@gmail.com>
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
7564b4e
to
c702980
Compare
""" | ||
source_table_name = self.get_table_qualified_name(source_table) | ||
target_table_name = self.get_table_qualified_name(target_table) | ||
stage_table_name = self.get_table_qualified_name(Table()) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@pankajkoti I think it would be a good idea to create this table in the temp schema. right now we are creating this in the default database scheme. WDYT?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Redshift does not accept schema option for temp tables. As per the docs here: https://docs.aws.amazon.com/redshift/latest/dg/r_CREATE_TABLE_NEW.html
TEMPORARY | TEMP
Keyword that creates a temporary table that is visible only within the current session. The table is automatically dropped at the end of the session in which it is created. The temporary table can have the same name as a permanent table. The temporary table is created in a separate, session-specific schema. (You can't specify a name for this schema.) This temporary schema becomes the first schema in the search path, so the temporary table will take precedence over the permanent table unless you qualify the table name with the schema name to access the permanent table. For more information about schemas and precedence, see search_path.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! Nice work! :)
Description
Redshift does not enforce constraints although it accepts those
in the table DDL.
Ref: https://docs.aws.amazon.com/redshift/latest/dg/t_Defining_constraints.html
As a result, in case of conflicts, e.g. uniqueness constraints, it
does allow insertion of duplicate values and expects you to clean
up duplicates.
Additionally, for merge redshift documentation suggests making use
of a staging table.
Ref: https://docs.aws.amazon.com/redshift/latest/dg/t_updating-inserting-using-staging-tables-.html
The merge implementation involves the following steps:
The above merge steps happen within a transaction block; meaning if one of the statement
fails, everything will be rolled back.
What is the current behavior?
No support for merge_operator in case of Redshift database.
What is the new behavior?
Implements support for merge_operator in case of Redshift database.
Does this introduce a breaking change?
Checklist
closes: #754