Merge main into 1.0.latest (#59)

* Slack alerts for failed nightly tests (#50)

* Fix: Support individual query tag configuration for Seeds and Snapshots (#48)

* Add Query Tag for Seed

* Add Query Tag for Snapshots

* Add Changelog for #48

* Add Snapshot with Query Tag test

* Add Seed with Query Tag test

* Update connections.py (#49)

* Update connections.py

Resolves issues when the Snowflake OCSP server is not reachable and the dbt user would like to run commands without checking the OCSP server. OCSP failures/inaccessibilitiy are possible due to network routing issues or when corporate network security has exposed Snowflake (especially in privatelink) but not the OCSP server.

* Update CHANGELOG.md

Adding changelog information

* Update CHANGELOG.md

Co-authored-by: Jeremy Cohen <jtcohen6@gmail.com>

* Update changelog and unit test

Updated changelog and unit tests as requested.

* Fix typo

Co-authored-by: Jeremy Cohen <jtcohen6@gmail.com>
Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>

* Add column comments to Snowflake views (#53)

* Add column comments to Snowflake views
* Update create_view_as macro to include column comments
* Add test for column-level view comments
* Add testing for column comments with non-lowercase column names

* Organizing adapter macros for column persistence

* Refactor query tag tests (#57)

* Refactor query tag tests

* Try this

* Try storing failures, too

* Keep trying

* Try it this way

* Bumping version to 1.0.0rc2 (#56)

* Bumping version to 1.0.0rc2

* Update changelog

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>

Co-authored-by: Anton Huck <anthu@users.noreply.github.com>
Co-authored-by: R. Joshua Huntley <JoshuaHuntley@users.noreply.github.com>
Co-authored-by: Jeremy Cohen <jtcohen6@gmail.com>
Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>
Co-authored-by: Spencer Taylor <85514383+spencer-taylor-workrise@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>

.bumpversion.cfg

@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 1.0.0rc1
+current_version = 1.0.0rc2
 parse = (?P<major>\d+)
 	\.(?P<minor>\d+)
 	\.(?P<patch>\d+)

.github/workflows/integration.yml

@@ -218,3 +218,19 @@ jobs:
             "You do not have permissions to run integration tests, @dbt-labs/core "\
             "needs to label this PR with `ok to test` in order to run integration tests!"
           check_for_duplicate_msg: true
+
+  slack-results:
+    runs-on: ubuntu-latest
+    needs: test 
+    if: always()
+
+    steps:
+      - name: Posting scheduled run failures
+        uses: ravsamhq/notify-slack-action@v1
+        if: ${{ github.event_name == 'schedule' }}
+        with:
+          notification_title: 'Snowflake nightly integration test failed'
+          status: ${{ job.status }}
+          notify_when: 'failure'
+    env:
+      SLACK_WEBHOOK_URL: ${{ secrets.SLACK_DEV_CORE_ALERTS }}

CHANGELOG.md

@@ -1,5 +1,19 @@
 ## dbt-snowflake 1.0.0 (Release TBD)
 
+## dbt-snowflake 1.0.0rc2 (November 24, 2021)
+
+### Fixes
+- Apply query tags for Seed and Snapshot materialisations ([#20](https://github.com/dbt-labs/dbt-snowflake/issues/20), [#48](https://github.com/dbt-labs/dbt-snowflake/issues/48))
+- Adds column-level comments to Snowflake views ([#17](https://github.com/dbt-labs/dbt-snowflake/issues/17))
+
+### Under the hood
+- Resolves an issue caused when the Snowflake OCSP server is not accessible, by exposing the `insecure_mode` boolean avalable in the Snowflake python connector ([#31](https://github.com/dbt-labs/dbt-snowflake/issues/31), [#49](https://github.com/dbt-labs/dbt-snowflake/pull/49))
+
+### Contributors
+- [@anthu](https://github.com/anthu) ([#48](https://github.com/dbt-labs/dbt-snowflake/pull/48))
+- [@JoshuaHuntley](https://github.com/JoshuaHuntley) ([#49](https://github.com/dbt-labs/dbt-snowflake/pull/49))
+- [@spencer-taylor-workrise](https://github.com/spencer-taylor-workrise) ([#17](https://github.com/dbt-labs/dbt-snowflake/issues/17))
+
 ## dbt-snowflake 1.0.0rc1 (November 10, 2021)
 
 ### Features

dbt/adapters/snowflake/__version__.py

@@ -1 +1 @@
-version = '1.0.0rc1'
+version = '1.0.0rc2'

dbt/adapters/snowflake/connections.py

@@ -52,6 +52,7 @@ class SnowflakeCredentials(Credentials):
     connect_timeout: int = 10
     retry_on_database_errors: bool = False
     retry_all: bool = False
+    insecure_mode: Optional[bool] = False
 
     def __post_init__(self):
         if (
@@ -252,6 +253,7 @@ def open(cls, connection):
                     autocommit=True,
                     client_session_keep_alive=creds.client_session_keep_alive,
                     application='dbt',
+                    insecure_mode=creds.insecure_mode,
                     **creds.auth_args()
                 )
 

dbt/include/snowflake/macros/adapters.sql

@@ -38,6 +38,23 @@
     {%- endif -%}
 {% endmacro %}
 
+{% macro get_column_comment_sql(column_name, column_dict) %}
+  {{ adapter.quote(column_name) if column_dict[column_name]['quote'] else column_name }} COMMENT $${{ column_dict[column_name]['description'] | replace('$', '[$]') }}$$
+{% endmacro %}
+
+{% macro get_persist_docs_column_list(model_columns, query_columns) %}
+(
+  {% for column_name in query_columns %}
+    {% if (column_name|upper in model_columns) or (column_name in model_columns) %}
+      {{ get_column_comment_sql(column_name, model_columns) }}
+    {% else %}
+      {{column_name}}
+    {% endif %}
+    {{ ", " if not loop.last else "" }}
+  {% endfor %}
+)
+{% endmacro %}
+
 {% macro snowflake__create_view_as(relation, sql) -%}
   {%- set secure = config.get('secure', default=false) -%}
   {%- set copy_grants = config.get('copy_grants', default=false) -%}
@@ -46,7 +63,14 @@
   {{ sql_header if sql_header is not none }}
   create or replace {% if secure -%}
     secure
-  {%- endif %} view {{ relation }} {% if copy_grants -%} copy grants {%- endif %} as (
+  {%- endif %} view {{ relation }} 
+  {% if config.persist_column_docs() -%}
+    {% set model_columns = model.columns %}
+    {% set query_columns = get_columns_in_query(sql) %}
+    {{ get_persist_docs_column_list(model_columns, query_columns) }}
+
+  {%- endif %}
+  {% if copy_grants -%} copy grants {%- endif %} as (
     {{ sql }}
   );
 {% endmacro %}
@@ -158,7 +182,7 @@
     {% set existing_columns = adapter.get_columns_in_relation(relation) | map(attribute="name") | list %}
     alter {{ relation.type }} {{ relation }} alter
     {% for column_name in column_dict if (column_name in existing_columns) or (column_name|upper in existing_columns) %}
-        {{ adapter.quote(column_name) if column_dict[column_name]['quote'] else column_name }} COMMENT $${{ column_dict[column_name]['description'] | replace('$', '[$]') }}$$ {{ ',' if not loop.last else ';' }}
+        {{ get_column_comment_sql(column_name, column_dict) }} {{ ',' if not loop.last else ';' }}
     {% endfor %}
 {% endmacro %}
 

dbt/include/snowflake/macros/materializations/seed.sql

@@ -35,3 +35,13 @@
     {# Return SQL so we can render it out into the compiled files #}
     {{ return(statements[0]) }}
 {% endmacro %}
+
+{% materialization seed, adapter='snowflake' %}
+    {% set original_query_tag = set_query_tag() %}
+
+    {% set relations = materialization_seed_default() %}
+
+    {% do unset_query_tag(original_query_tag) %}
+
+    {{ return(relations) }}
+{% endmaterialization %}

dbt/include/snowflake/macros/materializations/snapshot.sql

@@ -0,0 +1,9 @@
+{% materialization snapshot, adapter='snowflake' %}
+    {% set original_query_tag = set_query_tag() %}
+
+    {% set relations = materialization_snapshot_default() %}
+
+    {% do unset_query_tag(original_query_tag) %}
+
+    {{ return(relations) }}
+{% endmaterialization %}

tests/integration/column_comments_tests/models/quote_model.sql

@@ -1 +1 @@
-select 1 as {{ adapter.quote("2id") }}
+select 1 as {{ adapter.quote("2id") }}, 2 as {{ adapter.quote("3iD") }}

tests/integration/column_comments_tests/models/schema.yml

@@ -6,4 +6,6 @@ models:
       - name: 2id
         description: "XXX My description"
         quote: true
-
+      - name: 3iD
+        description: "XXX Testing for non-lowercase column name"
+        quote: true

tests/integration/column_comments_tests/test_column_comments.py

@@ -2,8 +2,7 @@
 
 from tests.integration.base import DBTIntegrationTest, use_profile
 
-
-class TestColumnComment(DBTIntegrationTest):
+class BaseTestColumnComment(DBTIntegrationTest):
     @property
     def schema(self):
         return "column_comment"
@@ -12,6 +11,19 @@ def schema(self):
     def models(self):
         return "models"
 
+    def run_has_comments(self):
+        self.run_dbt()
+        self.run_dbt(['docs', 'generate'])
+        with open('target/catalog.json') as fp:
+            catalog_data = json.load(fp)
+        assert 'nodes' in catalog_data
+        assert len(catalog_data['nodes']) == 1
+        column_node = catalog_data['nodes']['model.test.quote_model']
+        for column in column_node['columns'].keys():
+            column_comment = column_node['columns'][column]['comment']
+            assert column_comment.startswith('XXX')
+
+class TestColumnCommentInTable(BaseTestColumnComment):
     @property
     def project_config(self):
         return {
@@ -26,18 +38,25 @@ def project_config(self):
                 }
             }
         }
+    @use_profile('snowflake')
+    def test_snowflake_comments(self):
+        self.run_has_comments()
 
-    def run_has_comments(self):
-        self.run_dbt()
-        self.run_dbt(['docs', 'generate'])
-        with open('target/catalog.json') as fp:
-            catalog_data = json.load(fp)
-        assert 'nodes' in catalog_data
-        assert len(catalog_data['nodes']) == 1
-        column_node = catalog_data['nodes']['model.test.quote_model']
-        column_comment = column_node['columns']['2id']['comment']
-        assert column_comment.startswith('XXX')
-
+class TestColumnCommentInView(BaseTestColumnComment):
+    @property
+    def project_config(self):
+        return {
+            'config-version': 2,
+            'models': {
+                'test': {
+                    'materialized': 'view',
+                    '+persist_docs': {
+                        "relation": True,
+                        "columns": True,
+                    },
+                }
+            }
+        }
     @use_profile('snowflake')
     def test_snowflake_comments(self):
         self.run_has_comments()

tests/integration/query_tag_tests/macros/check_tag.sql

@@ -0,0 +1,10 @@
+{% macro check_query_tag() %}
+
+  {% if execute %}
+    {% set query_tag = get_current_query_tag() %}
+    {% if query_tag != var("query_tag") %}
+      {{ exceptions.raise_compiler_error("Query tag not used!") }}
+    {% endif %}
+  {% endif %}
+
+{% endmacro %}

tests/integration/query_tag_tests/models/incremental_model_query_tag.sql

@@ -0,0 +1,3 @@
+{{ config(materialized = 'incremental', unique_key = 'id') }}
+
+select 1 as id

tests/integration/query_tag_tests/models/table_model_query_tag.sql

@@ -0,0 +1,3 @@
+{{ config(materialized = 'table') }}
+
+select 1 as id

tests/integration/query_tag_tests/models/view_model_query_tag.sql

@@ -0,0 +1,3 @@
+{{ config(materialized = 'view') }}
+
+select 1 as id

tests/integration/query_tag_tests/seeds/seed_query_tag.csv

@@ -0,0 +1,2 @@
+id
+1

tests/integration/query_tag_tests/snapshots/snapshot_query_tag.sql

@@ -0,0 +1,12 @@
+{% snapshot snapshot_query_tag %}
+    {{
+        config(
+            target_database=database,
+            target_schema=schema,
+            unique_key='id',
+            strategy='check',
+            check_cols=['color'],
+        )
+    }}
+    select 1 as id, 'blue' as color
+{% endsnapshot %}

tests/integration/query_tag_tests/test_query_tags.py

@@ -0,0 +1,46 @@
+import os
+import csv
+from tests.integration.base import DBTIntegrationTest, use_profile
+
+class TestSeedWithQueryTag(DBTIntegrationTest):
+    @property
+    def schema(self):
+        return "query_tag"
+
+    @property
+    def models(self):
+        return "models"
+
+    @property
+    def project_config(self):
+        return {
+            'config-version': 2,
+            'models': {
+                'test': {
+                    'query_tag': self.prefix,
+                    'post-hook': '{{ check_query_tag() }}'
+                },
+            },
+            'seeds': {
+                'test': {
+                    'query_tag': self.prefix,
+                    'post-hook': '{{ check_query_tag() }}'
+                },
+            },
+            'snapshots': {
+                'test': {
+                    'query_tag': self.prefix,
+                    'post-hook': '{{ check_query_tag() }}'
+                },
+            },
+        }
+
+    def build_all_with_query_tags(self):
+        self.run_dbt(['build', '--vars', '{{"query_tag": "{}"}}'.format(self.prefix)])
+
+    @use_profile('snowflake')
+    def test__snowflake__build_tagged_twice(self):
+        self.build_all_with_query_tags()
+        self.build_all_with_query_tags()
+
+

tests/integration/simple_snapshot_test/check-snapshots-query-tag-expected/check_query_tag.sql

@@ -0,0 +1,16 @@
+
+
+with query as (
+
+    -- check that the current value for id=1 is red
+    select case when (
+        select count(*)
+        from table(information_schema.query_history_by_user())
+        where QUERY_TAG = '{{ var('query_tag') }}'
+    ) > 1 then 0 else 1 end as failures
+
+)
+
+select *
+from query
+where failures = 1

tests/unit/test_snowflake_adapter.py

@@ -261,7 +261,7 @@ def test_client_session_keep_alive_false_by_default(self):
                 account='test_account', autocommit=True,
                 client_session_keep_alive=False, database='test_database',
                 role=None, schema='public', user='test_user',
-                warehouse='test_warehouse', private_key=None, application='dbt')
+                warehouse='test_warehouse', private_key=None, application='dbt', insecure_mode=False)
         ])
 
     def test_client_session_keep_alive_true(self):
@@ -277,7 +277,7 @@ def test_client_session_keep_alive_true(self):
                 account='test_account', autocommit=True,
                 client_session_keep_alive=True, database='test_database',
                 role=None, schema='public', user='test_user',
-                warehouse='test_warehouse', private_key=None, application='dbt')
+                warehouse='test_warehouse', private_key=None, application='dbt', insecure_mode=False)
         ])
 
     def test_user_pass_authentication(self):
@@ -295,7 +295,7 @@ def test_user_pass_authentication(self):
                 client_session_keep_alive=False, database='test_database',
                 password='test_password', role=None, schema='public',
                 user='test_user', warehouse='test_warehouse', private_key=None,
-                application='dbt')
+                application='dbt', insecure_mode=False)
         ])
 
     def test_authenticator_user_pass_authentication(self):
@@ -315,7 +315,7 @@ def test_authenticator_user_pass_authentication(self):
                 password='test_password', role=None, schema='public',
                 user='test_user', warehouse='test_warehouse',
                 authenticator='test_sso_url', private_key=None,
-                application='dbt', client_store_temporary_credential=True)
+                application='dbt', client_store_temporary_credential=True, insecure_mode=False)
         ])
 
     def test_authenticator_externalbrowser_authentication(self):
@@ -333,7 +333,7 @@ def test_authenticator_externalbrowser_authentication(self):
                 client_session_keep_alive=False, database='test_database',
                 role=None, schema='public', user='test_user',
                 warehouse='test_warehouse', authenticator='externalbrowser',
-                private_key=None, application='dbt', client_store_temporary_credential=True)
+                private_key=None, application='dbt', client_store_temporary_credential=True, insecure_mode=False)
         ])
 
     def test_authenticator_oauth_authentication(self):
@@ -352,7 +352,7 @@ def test_authenticator_oauth_authentication(self):
                 client_session_keep_alive=False, database='test_database',
                 role=None, schema='public', user='test_user',
                 warehouse='test_warehouse', authenticator='oauth', token='my-oauth-token',
-                private_key=None, application='dbt', client_store_temporary_credential=True)
+                private_key=None, application='dbt', client_store_temporary_credential=True, insecure_mode=False)
         ])
 
     @mock.patch('dbt.adapters.snowflake.SnowflakeCredentials._get_private_key', return_value='test_key')
@@ -373,7 +373,7 @@ def test_authenticator_private_key_authentication(self, mock_get_private_key):
                 client_session_keep_alive=False, database='test_database',
                 role=None, schema='public', user='test_user',
                 warehouse='test_warehouse', private_key='test_key',
-                application='dbt')
+                application='dbt', insecure_mode=False)
         ])
 
     @mock.patch('dbt.adapters.snowflake.SnowflakeCredentials._get_private_key', return_value='test_key')
@@ -394,7 +394,7 @@ def test_authenticator_private_key_authentication_no_passphrase(self, mock_get_p
                 client_session_keep_alive=False, database='test_database',
                 role=None, schema='public', user='test_user',
                 warehouse='test_warehouse', private_key='test_key',
-                application='dbt')
+                application='dbt', insecure_mode=False)
         ])