🚨 Snowflake produces permanent tables 🚨 by edgao · Pull Request #9063 · airbytehq/airbyte

Edward Gao (edgao) · 2021-12-22T19:18:47Z

What

Based on #9072

Update Snowflake's normalization implementation to create permanent (i.e. non-transient) tables.

Tests are failing with permissions errors - they were passing a few days ago locally, but I've never actually gotten them to pass on CI.

How

Actual changes:

Produce a new normalization-snowflake image, which sets the +transient: false flag in dbt_project.yml
- Map destination-snowflake to normalization-snowflake in the worker
Update the base-normalization tests to use the new dbt_project.yml
- note that the normalization_test_output for snowflake now produces permanent tables - e.g. https://github.com/airbytehq/airbyte/pull/9063/files#diff-c05d0a4c06813e52428812dac424bfc62132bcbb9f76de7e833668d4afc97c7aL3-R3
Update the snowflake insert acceptance test to check that it produces permanent tables. (arguably the s3-copy, gcs-copy, internal-staging tests could also do this check?)

Misc stuff:

Update the gitignore to also not-ignore the all-caps filenames (https://github.com/airbytehq/airbyte/pull/9063/files#diff-b9209cdd37191adc4197725f05457dd061548edc407ea21063fd0a6674836d3aR23)
Ignore an M1-specific error https://github.com/airbytehq/airbyte/pull/9063/files#diff-9a3bcae8cb5c56aa30c00548e06eade6ad771f3d4f098f6867ae9a183049dfd8R450
some random normalization_test_output diffs:
- autoformatting? (I don't think I ran a linter here, so not clear where these came from): switching double-quotes to single-quotes, whitespace, etc
- Some consequences of a recent clickhouse PR (these changes are why this PR is based on top of Marcos' PR) https://github.com/airbytehq/airbyte/pull/9063/files#diff-2cfce733889918c6e4e63842483b71819c04826855db2bac5ac7bd10dd29a026

🚨 User Impact 🚨

sort of - new tables will be created as permanent, so users who want to continue creating transient tables will need to create a new transient database for airbyte

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here
After the new connector version is published, connector version bumped in the seed directory as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Edward Gao (edgao) · 2021-12-22T21:19:47Z

/test connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1613099842
❌ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1613099842
🐛

Edward Gao (edgao) · 2021-12-22T21:23:58Z

/test connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1613112056
❌ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1613112056
🐛

Edward Gao (edgao) · 2021-12-22T22:23:14Z

/test connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1613276555
❌ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1613276555
🐛

Edward Gao (edgao) · 2021-12-22T22:23:21Z

/test connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1613276690
❌ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1613276690
🐛

…st-pr-9029

Sherif A. Nada (sherifnada)

LGTM, thanks for providing the reading order, it was super helpful

Sherif A. Nada (sherifnada) · 2022-01-05T08:20:15Z

@@ -0,0 +1,33 @@
+FROM fishtownanalytics/dbt:0.21.1
+COPY --from=airbyte/base-airbyte-protocol-python:0.1.1 /airbyte /airbyte


do we need to get this docker image anymore? can't we just pip install airbyte-cdk which contains the CDK models? not blocking but a good sweep

is the COPY command equivalent to that pip install? (looks like the the docker image just contains base_python_structs)

Sherif A. Nada (sherifnada) · 2022-01-05T08:20:48Z


 public abstract class DestinationAcceptanceTest {

+  private static final String NORMALIZATION_VERSION = "dev";


Sherif A. Nada (sherifnada) · 2022-01-05T08:22:23Z

@@ -0,0 +1,33 @@
+FROM fishtownanalytics/dbt:0.21.1


are we creating a special dockerfile for snowflake because it's DBT yamls deviate from the standard yamls?

yeah. Chris suggested modifying entrypoint.sh to use the modified yaml instead of publishing a new docker image, but my thinking was to keep entrypoint.sh integration-agnostic.

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

Edward Gao (edgao) · 2022-01-06T16:26:32Z

/publish connector=connectors/destination-snowflake

Edward Gao (edgao) · 2022-01-06T16:28:51Z

/publish connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1663877333
✅ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1663877333

Edward Gao (edgao) · 2022-01-06T18:22:02Z

/publish connector=bases/base-normalization

Error: No ref found for: edgao/snowflake_permanent_tables

Christophe Duong (ChristopheDuong) · 2022-01-11T13:28:18Z

Closes #6743

add normalization-clickhouse docker build step

c6ae621

github-actions Bot added area/platform issues related to the platform area/worker Related to worker normalization labels Dec 22, 2021

Edward Gao (edgao) changed the title ~~Snowflake produces permanent tables~~ 🚨🚨 Snowflake produces permanent tables Dec 22, 2021

Edward Gao (edgao) temporarily deployed to more-secrets December 22, 2021 19:20 Inactive

Edward Gao (edgao) changed the title ~~🚨🚨 Snowflake produces permanent tables~~ 🚨 Snowflake produces permanent tables 🚨 Dec 22, 2021

Edward Gao (edgao) temporarily deployed to more-secrets December 22, 2021 19:56 Inactive

Edward Gao (edgao) force-pushed the edgao/snowflake_permanent_tables branch from 4980acb to 54da45c Compare December 22, 2021 21:19

Edward Gao (edgao) temporarily deployed to more-secrets December 22, 2021 21:20 Inactive

Jared Rhizor (jrhizor) temporarily deployed to more-secrets December 22, 2021 21:21 Inactive

Jared Rhizor (jrhizor) temporarily deployed to more-secrets December 22, 2021 21:25 Inactive

Edward Gao (edgao) temporarily deployed to more-secrets December 22, 2021 22:12 Inactive

Edward Gao (edgao) temporarily deployed to more-secrets December 22, 2021 22:23 Inactive

Jared Rhizor (jrhizor) temporarily deployed to more-secrets December 22, 2021 22:25 Inactive

Edward Gao (edgao) temporarily deployed to more-secrets December 22, 2021 22:56 Inactive

M. Marx (marcosmarxm) added 2 commits December 22, 2021 23:20

Merge branch 'patch-4' of github.com:jzcruiser/airbyte into marcos/te…

b88225e

…st-pr-9029

bump normalization version

cc499c2

github-actions Bot added the area/connectors Connector related issues label Dec 23, 2021

Edward Gao (edgao) temporarily deployed to more-secrets December 23, 2021 16:46 Inactive

small changes gradle

a2517a3

Edward Gao (edgao) force-pushed the edgao/snowflake_permanent_tables branch from b3a8ea3 to 24643b7 Compare December 23, 2021 17:49

Edward Gao (edgao) temporarily deployed to more-secrets December 23, 2021 17:51 Inactive

Merge branch 'master' into marcos/test-pr-9029

4a51799

Edward Gao (edgao) temporarily deployed to more-secrets December 24, 2021 00:32 Inactive

Edward Gao (edgao) added 3 commits January 4, 2022 16:04

regenerate test output

0c55399

typo

b8db991

exclude snowflake dbt template from spotless

d8abb0c

Edward Gao (edgao) force-pushed the edgao/snowflake_permanent_tables branch from 1292ebb to d8abb0c Compare January 5, 2022 00:04

Edward Gao (edgao) temporarily deployed to more-secrets January 5, 2022 00:06 Inactive

Base automatically changed from marcos/test-pr-9029 to master January 5, 2022 02:28

Sherif A. Nada (sherifnada) approved these changes Jan 5, 2022

View reviewed changes

clarify documentation

baf0db2

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

Edward Gao (edgao) temporarily deployed to more-secrets January 5, 2022 18:18 Inactive

Edward Gao (edgao) added 2 commits January 5, 2022 10:35

Merge branch 'master' into edgao/snowflake_permanent_tables

fe10d82

regenerate normalization_test_output

01bcb67

Edward Gao (edgao) temporarily deployed to more-secrets January 5, 2022 21:01 Inactive

delete unused variable

ab662e3

Edward Gao (edgao) temporarily deployed to more-secrets January 6, 2022 00:24 Inactive

Christophe Duong (ChristopheDuong) approved these changes Jan 6, 2022

View reviewed changes

minor bump actually

03e38dc

Edward Gao (edgao) temporarily deployed to more-secrets January 6, 2022 16:30 Inactive

Jared Rhizor (jrhizor) temporarily deployed to more-secrets January 6, 2022 16:31 Inactive

bump definition

dffe792

Edward Gao (edgao) temporarily deployed to more-secrets January 6, 2022 18:05 Inactive

Merge branch 'master' into edgao/snowflake_permanent_tables

663b498

Edward Gao (edgao) merged commit b6926d4 into master Jan 6, 2022

Edward Gao (edgao) deleted the edgao/snowflake_permanent_tables branch January 6, 2022 18:10

Edward Gao (edgao) temporarily deployed to more-secrets January 6, 2022 18:11 Inactive

This was referenced Jan 7, 2022

Bump Airbyte version from 0.35.3-alpha to 0.35.4-alpha #9353

Merged

fix migration test snowflake version comparison error #9370

Merged

		@@ -0,0 +1,33 @@
		FROM fishtownanalytics/dbt:0.21.1
		COPY --from=airbyte/base-airbyte-protocol-python:0.1.1 /airbyte /airbyte


		public abstract class DestinationAcceptanceTest {

		private static final String NORMALIZATION_VERSION = "dev";

Uh oh!

Conversation

Edward Gao (edgao) commented Dec 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Uh oh!

Edward Gao (edgao) commented Dec 22, 2021 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Edward Gao (edgao) commented Dec 22, 2021 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Edward Gao (edgao) commented Dec 22, 2021 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Edward Gao (edgao) commented Dec 22, 2021 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Sherif A. Nada (sherifnada) left a comment

Choose a reason for hiding this comment

Uh oh!

Sherif A. Nada (sherifnada) Jan 5, 2022

Choose a reason for hiding this comment

Uh oh!

Edward Gao (edgao) Jan 5, 2022

Choose a reason for hiding this comment

Uh oh!

Sherif A. Nada (sherifnada) Jan 5, 2022

Choose a reason for hiding this comment

Uh oh!

Sherif A. Nada (sherifnada) Jan 5, 2022

Choose a reason for hiding this comment

Uh oh!

Edward Gao (edgao) Jan 5, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Edward Gao (edgao) commented Jan 6, 2022

Uh oh!

Edward Gao (edgao) commented Jan 6, 2022 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Edward Gao (edgao) commented Jan 6, 2022 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Christophe Duong (ChristopheDuong) commented Jan 11, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Edward Gao (edgao) commented Dec 22, 2021 •

edited

Loading

Edward Gao (edgao) commented Dec 22, 2021 •

edited by github-actions Bot

Loading

Edward Gao (edgao) commented Dec 22, 2021 •

edited by github-actions Bot

Loading

Edward Gao (edgao) commented Dec 22, 2021 •

edited by github-actions Bot

Loading

Edward Gao (edgao) commented Dec 22, 2021 •

edited by github-actions Bot

Loading

Edward Gao (edgao) commented Jan 6, 2022 •

edited by github-actions Bot

Loading

Edward Gao (edgao) commented Jan 6, 2022 •

edited by github-actions Bot

Loading