-
Notifications
You must be signed in to change notification settings - Fork 3.7k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
🎉 MySQL destination: normalization (#4163)
* Add mysql dbt package * Add mysql normalization support in java * Add mysql normalization support in python * Fix unit tests * Update readme * Setup mysql container in integration test * Add macros * Depend on dbt-mysql from git repo * Remove mysql limitation test * Test normalization * Revert protocol format change * Fix mysel json macros * Fix two more macros * Fix table name length * Fix array macro * Fix equality test macro * Update replace-identifiers * Add more identifiers to replace * Fix unnest macro * Fix equality macro * Check in mysql test output * Update column limit test for mysql * Escape parentheses * Remove unnecessary mysql test * Remove mysql output for easier code review * Remove unnecessary mysql test * Remove parentheses * Update dependencies * Skip mysql instead of manually write out types * Bump version * Check in unit test for mysql name transformer * Fix type conversion * Use json_value to extract scalar json fields * Move dbt-mysql to Dockerfile (#4459) * Format code * Check in mysql dbt output * Remove unnecessary quote * Update mysql equality test to match 0.19.0 * Check in schema_test update * Update readme * Bump base normalization version * Update document Co-authored-by: Christophe Duong <christophe.duong@gmail.com>
- Loading branch information
1 parent
9531c4d
commit 2caf390
Showing
124 changed files
with
2,880 additions
and
76 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
3 changes: 3 additions & 0 deletions
3
...tegrations/bases/base-normalization/dbt-project-template/macros/cross_db_utils/except.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
{% macro mysql__except() %} | ||
{% do exceptions.warn("MySQL does not support EXCEPT operator") %} | ||
{% endmacro %} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
60 changes: 60 additions & 0 deletions
60
...tegrations/bases/base-normalization/dbt-project-template/macros/schema_tests/equality.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,60 @@ | ||
{# | ||
-- Adapted from https://github.com/dbt-labs/dbt-utils/blob/0-19-0-updates/macros/schema_tests/equality.sql | ||
-- dbt-utils version: 0.6.4 | ||
-- This macro needs to be updated accordingly when dbt-utils is upgraded. | ||
-- This is needed because MySQL does not support the EXCEPT operator! | ||
#} | ||
|
||
{% macro mysql__test_equality(model, compare_model, compare_columns=None) %} | ||
|
||
{%- if not execute -%} | ||
{{ return('') }} | ||
{% endif %} | ||
|
||
{%- do dbt_utils._is_relation(model, 'test_equality') -%} | ||
|
||
{%- if not compare_columns -%} | ||
{%- do dbt_utils._is_ephemeral(model, 'test_equality') -%} | ||
{%- set compare_columns = adapter.get_columns_in_relation(model) | map(attribute='quoted') -%} | ||
{%- endif -%} | ||
|
||
{% set compare_cols_csv = compare_columns | join(', ') %} | ||
|
||
with a as ( | ||
select * from {{ model }} | ||
), | ||
|
||
b as ( | ||
select * from {{ compare_model }} | ||
), | ||
|
||
a_minus_b as ( | ||
select {{ compare_cols_csv }} from a | ||
where ({{ compare_cols_csv }}) not in | ||
(select {{ compare_cols_csv }} from b) | ||
), | ||
|
||
b_minus_a as ( | ||
select {{ compare_cols_csv }} from b | ||
where ({{ compare_cols_csv }}) not in | ||
(select {{ compare_cols_csv }} from a) | ||
), | ||
|
||
unioned as ( | ||
select * from a_minus_b | ||
union all | ||
select * from b_minus_a | ||
), | ||
|
||
final as ( | ||
select (select count(*) from unioned) + | ||
(select abs( | ||
(select count(*) from a_minus_b) - | ||
(select count(*) from b_minus_a) | ||
)) | ||
as count | ||
) | ||
|
||
select count from final | ||
|
||
{% endmacro %} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.