Destination Snowflake: Enable in-connector normalization #25782

edgao · 2023-05-03T16:57:18Z

What

Enable in-connector normalization for snowflake. See https://github.com/airbytehq/airbyte-platform-internal/pull/6349 for sync screenshot.

How

Mostly just replicate the bigquery changes.

🚨 User Impact 🚨

None.

github-actions · 2023-05-03T16:57:37Z

github-actions · 2023-05-03T17:01:21Z

Affected Connector Report

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to do the following as needed:

Run integration tests
Bump connector or module version
Add changelog
Publish the new version

✅ Sources (0)

Connector	Version	Changelog	Publish

See "Actionable Items" below for how to resolve warnings and errors.

❌ Destinations (1)

Connector	Version	Changelog	Publish
`destination-snowflake`	`1.0.5`	✅	❌ (diff seed version)

See "Actionable Items" below for how to resolve warnings and errors.

👀 Other Modules (1)

base-normalization

Actionable Items

(click to expand)

Category	Status	Actionable Item
Version	❌ mismatch	The version of the connector is different from its normal variant. Please bump the version of the connector.
	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
Changelog	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
	❌ changelog missing	There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog.
Publish	⚠ not in seed	The connector is not in the cloud or oss registry, so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that you have added a metadata.yaml file and the expected registries are enabled.

tools/ci_connector_ops/ci_connector_ops/pipelines/actions/environments.py

edgao · 2023-05-08T18:57:52Z

/test connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/4918585004
✅ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/4918585004
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 15      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    18      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     171     10    94%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         195     48    75%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         65     39    40%
normalization/transform_catalog/stream_processor.py                 595    400    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1461    632    57%

Build Passed

Test summary info:

All Passed

ryankfu

LGTM

edgao · 2023-05-08T23:40:53Z

/publish connector=connectors/destination-snowflake pre-release=true

🕑 Publishing the following connectors:
connectors/destination-snowflake
https://github.com/airbytehq/airbyte/actions/runs/4920667036

Connector	Version	Did it publish?	Were definitions generated?
connectors/destination-snowflake	1.0.3-dev.cf1e1399	❌	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

airbyte-integrations/connectors/destination-snowflake/Dockerfile

evantahler · 2023-05-09T23:46:39Z

airbyte-integrations/connectors/destination-snowflake/build.gradle

@@ -27,7 +27,7 @@ application {
 dependencies {
    implementation 'com.google.cloud:google-cloud-storage:1.113.16'
    implementation 'com.google.auth:google-auth-library-oauth2-http:0.25.5'
-  // Updating to any newer version (e.g. 3.13.22) is causing a regression with normalization. 
+  // Updating to any newer version (e.g. 3.13.22) is causing a regression with normalization.


edgao · 2023-05-10T00:01:13Z

/publish connector=connectors/destination-snowflake pre-release=true

🕑 Publishing the following connectors:
connectors/destination-snowflake
https://github.com/airbytehq/airbyte/actions/runs/4931600004

Connector	Version	Did it publish?	Were definitions generated?
connectors/destination-snowflake	1.0.3-dev.9e72dced	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

tools/ci_connector_ops/ci_connector_ops/pipelines/actions/environments.py

cynthiaxyin · 2023-05-10T23:54:14Z

/publish connector=connectors/destination-snowflake pre-release=true

🕑 Publishing the following connectors:
connectors/destination-snowflake
https://github.com/airbytehq/airbyte/actions/runs/4942377147

Connector	Version	Did it publish?	Were definitions generated?
connectors/destination-snowflake	1.0.3-dev.70a78910	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

airbyte-integrations/connectors/destination-snowflake/Dockerfile

cynthiaxyin · 2023-05-25T22:26:57Z

/test connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/5085153700
✅ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/5085153700
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 15      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    18      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     171     10    94%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         195     48    75%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         65     39    40%
normalization/transform_catalog/stream_processor.py                 603    407    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1469    639    57%

Build Passed

Test summary info:

All Passed

edgao

I can't approve this, but thanks for taking this to the finish line!

github-actions · 2023-05-31T22:53:13Z

Coverage report for source-postgres

There is no coverage information present for the Files changed

Total Project Coverage	65.93%	🍏

octavia-squidington-iii added area/connectors Connector related issues connectors/destination/snowflake labels May 3, 2023

edgao changed the title ~~add support for snowflake~~ Destination Snowflake: Enable in-connector normalization May 5, 2023

edgao force-pushed the edgao/normalization_in_snowflake branch from 1bd9d9c to c4de8a4 Compare May 5, 2023 14:13

This comment was marked as outdated.

Sign in to view

octavia-squidington-iii added the connectors/source/snowflake label May 8, 2023

This comment was marked as outdated.

Sign in to view

edgao commented May 8, 2023

View reviewed changes

tools/ci_connector_ops/ci_connector_ops/pipelines/actions/environments.py Show resolved Hide resolved

edgao mentioned this pull request May 8, 2023

Implement in-destination normalization for clickhouse #25495

Closed

edgao force-pushed the edgao/normalization_in_snowflake branch from 7d0f8fc to cf1e139 Compare May 8, 2023 20:22

octavia-squidington-iii added the area/documentation Improvements or additions to documentation label May 8, 2023

edgao marked this pull request as ready for review May 8, 2023 20:31

edgao requested a review from a team as a code owner May 8, 2023 20:31

edgao requested review from jbfbell and cynthiaxyin May 8, 2023 20:31

ryankfu approved these changes May 8, 2023

View reviewed changes

evantahler reviewed May 9, 2023

View reviewed changes

edgao commented May 10, 2023

View reviewed changes

tools/ci_connector_ops/ci_connector_ops/pipelines/actions/environments.py Outdated Show resolved Hide resolved

cynthiaxyin force-pushed the edgao/normalization_in_snowflake branch from 9e72dce to 70a7891 Compare May 10, 2023 21:34

This comment was marked as outdated.

Sign in to view

edgao commented May 11, 2023

View reviewed changes

airbyte-integrations/connectors/destination-snowflake/Dockerfile Outdated Show resolved Hide resolved

edgao mentioned this pull request May 16, 2023

Destination Redshift: Install normalization #26165

Merged

jbfbell approved these changes May 18, 2023

View reviewed changes

edgao mentioned this pull request May 23, 2023

[Epic] Move normalization into destinations #24791

Closed

cynthiaxyin force-pushed the edgao/normalization_in_snowflake branch 2 times, most recently from 326793c to 370daae Compare May 25, 2023 22:24

edgao commented May 31, 2023

View reviewed changes

edgao and others added 10 commits May 31, 2023 15:37

add support for snowflake

31f1f20

do useful copy things

fcebb50

corrections

9e4c5b5

add build dependency

15fb1ed

deeeerp

f99b2e0

version bumps + changelog

dae7e03

try installing gcc-c++

471df17

move stuff around

98aa6ee

copy snowflake

bcd8400

bump version

c78361d

cynthiaxyin force-pushed the edgao/normalization_in_snowflake branch from 370daae to c78361d Compare May 31, 2023 22:39

cynthiaxyin merged commit 7a75c0c into master Jun 1, 2023
24 of 26 checks passed

cynthiaxyin deleted the edgao/normalization_in_snowflake branch June 1, 2023 21:14

edgao mentioned this pull request Jul 24, 2023

Rollout snowflake in-connector normalization #28623

Closed

cynthiaxyin mentioned this pull request Dec 19, 2023

Add in container normalization for postgres #33458

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Destination Snowflake: Enable in-connector normalization #25782

Destination Snowflake: Enable in-connector normalization #25782

edgao commented May 3, 2023 •

edited by cynthiaxyin

github-actions bot commented May 3, 2023 •

edited by cynthiaxyin

github-actions bot commented May 3, 2023 •

edited

✅ Sources (0)

❌ Destinations (1)

👀 Other Modules (1)

Actionable Items

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

edgao commented May 8, 2023 •

edited by github-actions bot

ryankfu left a comment

edgao commented May 8, 2023 •

edited by github-actions bot

evantahler May 9, 2023

edgao commented May 10, 2023 •

edited by github-actions bot

This comment was marked as outdated.

cynthiaxyin commented May 10, 2023 •

edited by github-actions bot

cynthiaxyin commented May 25, 2023 •

edited by github-actions bot

edgao left a comment

github-actions bot commented May 31, 2023

Destination Snowflake: Enable in-connector normalization #25782

Destination Snowflake: Enable in-connector normalization #25782

Conversation

edgao commented May 3, 2023 • edited by cynthiaxyin

What

How

🚨 User Impact 🚨

github-actions bot commented May 3, 2023 • edited by cynthiaxyin

Before Merging a Connector Pull Request

github-actions bot commented May 3, 2023 • edited

Affected Connector Report

✅ Sources (0)

❌ Destinations (1)

👀 Other Modules (1)

Actionable Items

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

edgao commented May 8, 2023 • edited by github-actions bot

Build Passed

ryankfu left a comment

Choose a reason for hiding this comment

edgao commented May 8, 2023 • edited by github-actions bot

evantahler May 9, 2023

Choose a reason for hiding this comment

edgao commented May 10, 2023 • edited by github-actions bot

This comment was marked as outdated.

cynthiaxyin commented May 10, 2023 • edited by github-actions bot

cynthiaxyin commented May 25, 2023 • edited by github-actions bot

Build Passed

edgao left a comment

Choose a reason for hiding this comment

github-actions bot commented May 31, 2023

Coverage report for source-postgres

edgao commented May 3, 2023 •

edited by cynthiaxyin

github-actions bot commented May 3, 2023 •

edited by cynthiaxyin

github-actions bot commented May 3, 2023 •

edited

edgao commented May 8, 2023 •

edited by github-actions bot

edgao commented May 8, 2023 •

edited by github-actions bot

edgao commented May 10, 2023 •

edited by github-actions bot

cynthiaxyin commented May 10, 2023 •

edited by github-actions bot

cynthiaxyin commented May 25, 2023 •

edited by github-actions bot