🚨 Snowflake produces permanent tables 🚨 #9063

edgao · 2021-12-22T19:18:47Z

What

Based on #9072

Update Snowflake's normalization implementation to create permanent (i.e. non-transient) tables.

Tests are failing with permissions errors - they were passing a few days ago locally, but I've never actually gotten them to pass on CI.

How

Actual changes:

Produce a new normalization-snowflake image, which sets the +transient: false flag in dbt_project.yml
- Map destination-snowflake to normalization-snowflake in the worker
Update the base-normalization tests to use the new dbt_project.yml
- note that the normalization_test_output for snowflake now produces permanent tables - e.g. https://github.com/airbytehq/airbyte/pull/9063/files#diff-c05d0a4c06813e52428812dac424bfc62132bcbb9f76de7e833668d4afc97c7aL3-R3
Update the snowflake insert acceptance test to check that it produces permanent tables. (arguably the s3-copy, gcs-copy, internal-staging tests could also do this check?)

Misc stuff:

Update the gitignore to also not-ignore the all-caps filenames (https://github.com/airbytehq/airbyte/pull/9063/files#diff-b9209cdd37191adc4197725f05457dd061548edc407ea21063fd0a6674836d3aR23)
Ignore an M1-specific error https://github.com/airbytehq/airbyte/pull/9063/files#diff-9a3bcae8cb5c56aa30c00548e06eade6ad771f3d4f098f6867ae9a183049dfd8R450
some random normalization_test_output diffs:
- autoformatting? (I don't think I ran a linter here, so not clear where these came from): switching double-quotes to single-quotes, whitespace, etc
- Some consequences of a recent clickhouse PR (these changes are why this PR is based on top of Marcos' PR) https://github.com/airbytehq/airbyte/pull/9063/files#diff-2cfce733889918c6e4e63842483b71819c04826855db2bac5ac7bd10dd29a026

🚨 User Impact 🚨

sort of - new tables will be created as permanent, so users who want to continue creating transient tables will need to create a new transient database for airbyte

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here
After the new connector version is published, connector version bumped in the seed directory as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

edgao · 2021-12-22T21:19:47Z

/test connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1613099842
❌ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1613099842
🐛

edgao · 2021-12-22T21:23:58Z

/test connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1613112056
❌ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1613112056
🐛

edgao · 2021-12-22T22:23:14Z

/test connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1613276555
❌ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1613276555
🐛

edgao · 2021-12-22T22:23:21Z

/test connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1613276690
❌ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1613276690
🐛

…st-pr-9029

sherifnada

LGTM, thanks for providing the reading order, it was super helpful

sherifnada · 2022-01-05T08:20:15Z

airbyte-integrations/bases/base-normalization/snowflake.Dockerfile

@@ -0,0 +1,33 @@
+FROM fishtownanalytics/dbt:0.21.1
+COPY --from=airbyte/base-airbyte-protocol-python:0.1.1 /airbyte /airbyte


do we need to get this docker image anymore? can't we just pip install airbyte-cdk which contains the CDK models? not blocking but a good sweep

is the COPY command equivalent to that pip install? (looks like the the docker image just contains base_python_structs)

sherifnada · 2022-01-05T08:20:48Z

...rc/main/java/io/airbyte/integrations/standardtest/destination/DestinationAcceptanceTest.java

@@ -79,6 +79,8 @@

 public abstract class DestinationAcceptanceTest {

+  private static final String NORMALIZATION_VERSION = "dev";


sherifnada · 2022-01-05T08:22:23Z

airbyte-integrations/bases/base-normalization/snowflake.Dockerfile

@@ -0,0 +1,33 @@
+FROM fishtownanalytics/dbt:0.21.1


are we creating a special dockerfile for snowflake because it's DBT yamls deviate from the standard yamls?

yeah. Chris suggested modifying entrypoint.sh to use the modified yaml instead of publishing a new docker image, but my thinking was to keep entrypoint.sh integration-agnostic.

docs/integrations/destinations/snowflake.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

edgao · 2022-01-06T16:26:32Z

/publish connector=connectors/destination-snowflake

edgao · 2022-01-06T16:28:51Z

/publish connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1663877333
✅ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1663877333

edgao · 2022-01-06T18:22:02Z

/publish connector=bases/base-normalization

Error: No ref found for: edgao/snowflake_permanent_tables

ChristopheDuong · 2022-01-11T13:28:18Z

Closes #6743

add normalization-clickhouse docker build step

c6ae621

github-actions bot added area/platform issues related to the platform area/worker Related to worker normalization labels Dec 22, 2021

edgao changed the title ~~Snowflake produces permanent tables~~ 🚨🚨 Snowflake produces permanent tables Dec 22, 2021

edgao temporarily deployed to more-secrets December 22, 2021 19:20 Inactive

edgao changed the title ~~🚨🚨 Snowflake produces permanent tables~~ 🚨 Snowflake produces permanent tables 🚨 Dec 22, 2021

edgao temporarily deployed to more-secrets December 22, 2021 19:56 Inactive

edgao force-pushed the edgao/snowflake_permanent_tables branch from 4980acb to 54da45c Compare December 22, 2021 21:19

edgao temporarily deployed to more-secrets December 22, 2021 21:20 Inactive

jrhizor temporarily deployed to more-secrets December 22, 2021 21:21 Inactive

jrhizor temporarily deployed to more-secrets December 22, 2021 21:25 Inactive

edgao temporarily deployed to more-secrets December 22, 2021 22:12 Inactive

edgao temporarily deployed to more-secrets December 22, 2021 22:23 Inactive

jrhizor temporarily deployed to more-secrets December 22, 2021 22:25 Inactive

edgao temporarily deployed to more-secrets December 22, 2021 22:56 Inactive

marcosmarxm added 2 commits December 22, 2021 23:20

Merge branch 'patch-4' of github.com:jzcruiser/airbyte into marcos/te…

b88225e

…st-pr-9029

bump normalization version

cc499c2

github-actions bot added the area/connectors Connector related issues label Dec 23, 2021

edgao temporarily deployed to more-secrets December 23, 2021 16:46 Inactive

small changes gradle

a2517a3

edgao force-pushed the edgao/snowflake_permanent_tables branch from b3a8ea3 to 24643b7 Compare December 23, 2021 17:49

edgao temporarily deployed to more-secrets December 23, 2021 17:51 Inactive

Merge branch 'master' into marcos/test-pr-9029

4a51799

edgao temporarily deployed to more-secrets December 24, 2021 00:32 Inactive

edgao force-pushed the edgao/snowflake_permanent_tables branch from 1292ebb to d8abb0c Compare January 5, 2022 00:04

edgao temporarily deployed to more-secrets January 5, 2022 00:06 Inactive

Base automatically changed from marcos/test-pr-9029 to master January 5, 2022 02:28

sherifnada approved these changes Jan 5, 2022

View reviewed changes

clarify documentation

baf0db2

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

edgao temporarily deployed to more-secrets January 5, 2022 18:18 Inactive

edgao added 2 commits January 5, 2022 10:35

Merge branch 'master' into edgao/snowflake_permanent_tables

fe10d82

regenerate normalization_test_output

01bcb67

edgao temporarily deployed to more-secrets January 5, 2022 21:01 Inactive

delete unused variable

ab662e3

edgao temporarily deployed to more-secrets January 6, 2022 00:24 Inactive

ChristopheDuong approved these changes Jan 6, 2022

View reviewed changes

minor bump actually

03e38dc

edgao temporarily deployed to more-secrets January 6, 2022 16:30 Inactive

jrhizor temporarily deployed to more-secrets January 6, 2022 16:31 Inactive

bump definition

dffe792

edgao temporarily deployed to more-secrets January 6, 2022 18:05 Inactive

Merge branch 'master' into edgao/snowflake_permanent_tables

663b498

edgao merged commit b6926d4 into master Jan 6, 2022

edgao deleted the edgao/snowflake_permanent_tables branch January 6, 2022 18:10

edgao temporarily deployed to more-secrets January 6, 2022 18:11 Inactive

This was referenced Jan 7, 2022

Bump Airbyte version from 0.35.3-alpha to 0.35.4-alpha #9353

Merged

fix migration test snowflake version comparison error #9370

Merged

karinakuz added connectors/destinations-warehouse connectors/destination/snowflake labels Jan 17, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🚨 Snowflake produces permanent tables 🚨 #9063

🚨 Snowflake produces permanent tables 🚨 #9063

edgao commented Dec 22, 2021 •

edited

Loading

edgao commented Dec 22, 2021 •

edited by github-actions bot

Loading

edgao commented Dec 22, 2021 •

edited by github-actions bot

Loading

edgao commented Dec 22, 2021 •

edited by github-actions bot

Loading

edgao commented Dec 22, 2021 •

edited by github-actions bot

Loading

sherifnada left a comment

sherifnada Jan 5, 2022

edgao Jan 5, 2022

sherifnada Jan 5, 2022

sherifnada Jan 5, 2022

edgao Jan 5, 2022

edgao commented Jan 6, 2022

edgao commented Jan 6, 2022 •

edited by github-actions bot

Loading

edgao commented Jan 6, 2022 •

edited by github-actions bot

Loading

ChristopheDuong commented Jan 11, 2022

		@@ -0,0 +1,33 @@
		FROM fishtownanalytics/dbt:0.21.1
		COPY --from=airbyte/base-airbyte-protocol-python:0.1.1 /airbyte /airbyte

		@@ -79,6 +79,8 @@

		public abstract class DestinationAcceptanceTest {

		private static final String NORMALIZATION_VERSION = "dev";

🚨 Snowflake produces permanent tables 🚨 #9063

🚨 Snowflake produces permanent tables 🚨 #9063

Conversation

edgao commented Dec 22, 2021 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

edgao commented Dec 22, 2021 • edited by github-actions bot Loading

edgao commented Dec 22, 2021 • edited by github-actions bot Loading

edgao commented Dec 22, 2021 • edited by github-actions bot Loading

edgao commented Dec 22, 2021 • edited by github-actions bot Loading

sherifnada left a comment

Choose a reason for hiding this comment

sherifnada Jan 5, 2022

Choose a reason for hiding this comment

edgao Jan 5, 2022

Choose a reason for hiding this comment

sherifnada Jan 5, 2022

Choose a reason for hiding this comment

sherifnada Jan 5, 2022

Choose a reason for hiding this comment

edgao Jan 5, 2022

Choose a reason for hiding this comment

edgao commented Jan 6, 2022

edgao commented Jan 6, 2022 • edited by github-actions bot Loading

edgao commented Jan 6, 2022 • edited by github-actions bot Loading

ChristopheDuong commented Jan 11, 2022

edgao commented Dec 22, 2021 •

edited

Loading

edgao commented Dec 22, 2021 •

edited by github-actions bot

Loading

edgao commented Dec 22, 2021 •

edited by github-actions bot

Loading

edgao commented Dec 22, 2021 •

edited by github-actions bot

Loading

edgao commented Dec 22, 2021 •

edited by github-actions bot

Loading

edgao commented Jan 6, 2022 •

edited by github-actions bot

Loading

edgao commented Jan 6, 2022 •

edited by github-actions bot

Loading