Refactor of the Code Base (#18)

* Refactor of the Code Base * Non-implemented functions return an exception
vertica · Mar 2, 2022 · bec9908 · bec9908
1 parent d673287
commit bec9908
Show file tree

Hide file tree

Showing 27 changed files with 600 additions and 510 deletions.
diff --git a/README.md b/README.md
@@ -8,7 +8,8 @@ Uses [vertica-python](https://github.com/vertica/vertica-python) to connect to V
 
 ### dbt Core Features
 
-Below is a table for what features the current Vertica adapter supports for dbt. This is constantly improving and changing as both dbt adds new functionality, as well as the dbt-vertica driver improves.
+Below is a table for what features the current Vertica adapter supports for dbt. This is constantly improving and changing as both dbt adds new functionality, as well as the dbt-vertica driver improves. This list is based upon dbt 1.0.3.
+
 
 | dbt Core Features                                 | Supported   |
 | ------------------------------------------------- | ----------- |
@@ -45,6 +46,10 @@ Below is a table for what features the current Vertica adapter supports for Vert
 
 ## Changes
 
+### 1.0.3
+- Refactored the adapter to model after dbt's global_project macros
+- Unimplemented functions should throw an exception that it's not implemented. If you stumble across this, please open an Issue or PR so we can investigate.
+
 ### 1.0.2
 - Added support for snapshot timestamp with passing tests
 - Added support for snapshot check cols with passing tests

diff --git a/dbt/include/vertica/macros/README.md b/dbt/include/vertica/macros/README.md
@@ -0,0 +1,65 @@
+# dbt-vertica Macro Implementations
+
+Below is a table for what macros the current Vertica adapter supports for dbt. This is constantly improving and changing as both dbt adds new functionality, as well as the dbt-vertica driver improves. This list is based upon dbt 1.0.3, with the list found under https://github.com/dbt-labs/dbt-core/tree/main/core/dbt/include/global_project/macros.
+
+## dbt Macros - adapters
+
+| dbt Macros - adapters     | Function                            | Implemented |
+| ------------------------- | ----------------------------------- | ----------- |
+| adapters/columns.sql      | get_columns_in_relation()           | Yes         |
+| adapters/columns.sql      | sql_convert_columns_in_relation()   | No          |
+| adapters/columns.sql      | get_columns_in_query()              | Yes         |
+| adapters/columns.sql      | alter_column_type()                 | No          |
+| adapters/columns.sql      | alter_relation_add_remove_columns() | No          |
+| adapters/freshness.sql    | collect_freshness()                 | No          |
+| adapters/freshness.sql    | current_timestamp()                 | Yes         |
+| adapters/indexes.sql      | create_indexes()                    | Yes         |
+| adapters/indexes.sql      | get_create_index_sql()              | No          |
+| adapters/metadata.sql     | get_catalog()                       | Yes         |
+| adapters/metadata.sql     | information_schema_name()           | Yes         |
+| adapters/metadata.sql     | list_schemas()                      | Yes         |
+| adapters/metadata.sql     | list_relations_without_caching()    | Yes         |
+| adapters/persist_docs.sql | alter_column_comment()              | No          |
+| adapters/persist_docs.sql | alter_relation_comment()            | No          |
+| adapters/persist_docs.sql | persist_docs()                      | Yes         |
+| adapters/relation.sql     | drop_relation()                     | Yes         |
+| adapters/relation.sql     | drop_relation_if_exists()           | Yes         |
+| adapters/relation.sql     | get_or_create_relation()            | No          |
+| adapters/relation.sql     | load_relation()                     | No          |
+| adapters/relation.sql     | make_temp_relation()                | Yes         |
+| adapters/relation.sql     | rename_relation()                   | Yes         |
+| adapters/relation.sql     | truncate_relation()                 | Yes         |
+| adapters/schema.sql       | create_schema()                     | Yes         |
+| adapters/schema.sql       | drop_schema()                       | Yes         |
+
+## dbt Macros - materializations
+| dbt Macros - materializations                            | Function                         | Implemented |
+| -------------------------------------------------------- | -------------------------------- | ----------- |
+| materializations/models/incremental/merge.sql            | get_merge_sql()                  | Yes         |
+| materializations/models/incremental/merge.sql            | get_delete_insert_merge_sql()    | Yes         |
+| materializations/models/incremental/merge.sql            | get_insert_overwrite_merge_sql() | No          |
+| materializations/models/table/create_table_as.sql        | create_table_as()                | Yes         |
+| materializations/models/view/create_or_replace_view.sql  | create_or_replace_view()         | Yes         |
+| materializations/models/view/create_view_as.sql          | create_view_as()                 | Yes         |
+| materializations/models/view/create_view_as.sql          | get_create_view_as_sql()         | Yes         |
+| materializations/seeds/helpers.sql                       | create_csv_table()               | Yes         |
+| materializations/seeds/helpers.sql                       | reset_csv_table()                | Yes         |
+| materializations/seeds/helpers.sql                       | get_binding_char()               | Yes         |
+| materializations/seeds/helpers.sql                       | get_seed_column_quoted_csv()     | Yes         |
+| materializations/seeds/helpers.sql                       | load_csv_rows()                  | Yes         |
+| materializations/snapshots/helper.sql                    | build_snapshot_table()           | Yes         |
+| materializations/snapshots/helper.sql                    | create_columns()                 | Yes         |
+| materializations/snapshots/helper.sql                    | post_snapshot()                  | Yes         |
+| materializations/snapshots/helper.sql                    | snapshot_staging_table()         | Yes         |
+| materializations/snapshots/snapshot_merge.sql            | snapshot_merge_sql()             | Yes         |
+| materializations/snapshots/strategies.sql                | snapshot_get_time()              | Yes         |
+| materializations/snapshots/strategies.sql                | snapshot_hash_arguments()        | Yes         |
+| materializations/snapshots/strategies.sql                | snapshot_string_as_time()        | Yes         |
+| materializations/configs.sql                             | set_sql_header()                 | Yes         |
+| materializations/configs.sql                             | should_full_refresh()            | Yes         |
+| materializations/configs.sql                             | should_store_failures()          | Yes         |
+| materializations/hooks.sql                               | run_hooks()                      | Yes         |
+| materializations/hooks.sql                               | make_hook_config()               | Yes         |
+| materializations/hooks.sql                               | before_begin()                   | Yes         |
+| materializations/hooks.sql                               | in_transaction()                 | Yes         |
+| materializations/hooks.sql                               | after_commit()                   | Yes         |
diff --git a/dbt/include/vertica/macros/adapters/columns.sql b/dbt/include/vertica/macros/adapters/columns.sql
@@ -0,0 +1,59 @@
+{% macro vertica__get_columns_in_relation(relation) -%}
+  {% call statement('get_columns_in_relation', fetch_result=True) %}
+    select
+    column_name
+    , data_type
+    , character_maximum_length
+    , numeric_precision
+    , numeric_scale
+    from (
+        select
+        column_name
+        , data_type
+        , character_maximum_length
+        , numeric_precision
+        , numeric_scale
+        , ordinal_position
+        from v_catalog.columns
+        where table_schema = '{{ relation.schema }}'
+        and table_name = '{{ relation.identifier }}'
+        union all
+        select
+        column_name
+        , data_type
+        , character_maximum_length
+        , numeric_precision
+        , numeric_scale
+        , ordinal_position
+        from v_catalog.view_columns
+        where table_schema = '{{ relation.schema }}'
+        and table_name = '{{ relation.identifier }}'
+    ) t
+    order by ordinal_position
+  {% endcall %}
+  {% set table = load_result('get_columns_in_relation').table %}
+  {{ return(sql_convert_columns_in_relation(table)) }}
+{% endmacro %}
+
+
+{% macro vertica__sql_convert_columns_in_relation(table) -%}
+  {{ exceptions.raise_not_implemented(
+    'sql_convert_columns_in_relation macro not implemented for adapter '+adapter.type()) }}
+{%- endmacro %}
+
+
+{% macro vertica__alter_column_type(relation, column_name, new_column_type) -%}
+  {{ exceptions.raise_not_implemented(
+    'alter_column_type macro not implemented for adapter '+adapter.type()) }}
+{%- endmacro %}
+
+
+{% macro vertica__alter_relation_add_remove_columns(relation, column_name, new_column_type) -%}
+  {{ exceptions.raise_not_implemented(
+    'alter_relation_add_remove_columns macro not implemented for adapter '+adapter.type()) }}
+{%- endmacro %}
+
+
+{# 
+  No need to implement get_columns_in_query(). Syntax supported by default. 
+#}
diff --git a/dbt/include/vertica/macros/adapters/freshness.sql b/dbt/include/vertica/macros/adapters/freshness.sql
@@ -0,0 +1,9 @@
+{% macro vertica__current_timestamp() -%}
+  current_timestamp
+{%- endmacro %}
+
+
+{% macro vertica__collect_freshness() -%}
+  {{ exceptions.raise_not_implemented(
+    'collect_freshness macro not implemented for adapter '+adapter.type()) }}
+{%- endmacro %}
diff --git a/dbt/include/vertica/macros/adapters/indexes.sql b/dbt/include/vertica/macros/adapters/indexes.sql
@@ -0,0 +1,9 @@
+{% macro vertica__get_create_index_sql(relation, index_dict) -%}
+  {{ exceptions.raise_not_implemented(
+    'get_create_index_sql macro not implemented for adapter '+adapter.type()) }}
+{%- endmacro %}
+
+
+{# 
+  No need to implement create_indexes(). Syntax supported by default. 
+#}
diff --git a/dbt/include/vertica/macros/adapters.sql → ...lude/vertica/macros/adapters/metadata.sql b/dbt/include/vertica/macros/adapters.sql → ...lude/vertica/macros/adapters/metadata.sql
@@ -1,140 +1,3 @@
-
-{% macro vertica__information_schema_name(database) -%}
-  {%- if database -%}
-    {{ adapter.quote_as_configured(database, 'database') }}.v_catalog
-  {%- else -%}
-    v_catalog
-  {%- endif -%}
-{%- endmacro %}
-
-{% macro vertica__list_schemas(database) %}
-  {% call statement('list_schemas', fetch_result=True, auto_begin=False) %}
-    select schema_name
-    from v_catalog.schemata
-  {% endcall %}
-  {{ return(load_result('list_schemas').table) }}
-{% endmacro %}
-
-{% macro vertica__check_schema_exists(database, schema) -%}
-  {% call statement('check_schema_exists', fetch_result=True, auto_begin=False) -%}
-        select count(*)
-        from v_catalog.schemata
-        where schema_name='{{ schema }}'
-  {%- endcall %}
-  {{ return(load_result('check_schema_exists').table) }}
-{% endmacro %}
-
-{% macro vertica__drop_schema(relation) -%}
-  {% call statement('drop_schema') -%}
-    drop schema if exists {{ relation.without_identifier().include(database=False) }} cascade
-  {% endcall %}
-{% endmacro %}
-
-{% macro vertica__create_schema(relation) -%}
-  {%- call statement('create_schema') -%}
-    create schema if not exists {{ relation.without_identifier().include(database=False) }}
-  {% endcall %}
-{% endmacro %}
-
-{% macro vertica__list_relations_without_caching(schema_relation) %}
-  {% call statement('list_relations_without_caching', fetch_result=True) -%}
-    select
-      '{{ schema_relation.database }}' as database,
-      table_name as name,
-      table_schema as schema,
-      'table' as type
-    from v_catalog.tables
-    where table_schema ilike '{{ schema_relation.schema }}'
-    union all
-    select
-      '{{ schema_relation.database }}' as database,
-      table_name as name,
-      table_schema as schema,
-      'view' as type
-    from v_catalog.views
-    where table_schema ilike '{{ schema_relation.schema }}'
-  {% endcall %}
-  {{ return(load_result('list_relations_without_caching').table) }}
-  {% endmacro %}
-
-{% macro vertica__rename_relation(from_relation, to_relation) %}
-  {% set target_name = adapter.quote_as_configured(to_relation.identifier, 'identifier') %}
-  {% call statement('rename_relation') -%}
-    alter {{ from_relation.type }} {{ from_relation }} rename to {{ target_name }}
-  {%- endcall %}
-{% endmacro %}
-
-
-{% macro vertica__get_columns_in_relation(relation) -%}
-  {% call statement('get_columns_in_relation', fetch_result=True) %}
-    select
-    column_name
-    , data_type
-    , character_maximum_length
-    , numeric_precision
-    , numeric_scale
-    from (
-        select
-        column_name
-        , data_type
-        , character_maximum_length
-        , numeric_precision
-        , numeric_scale
-        , ordinal_position
-        from v_catalog.columns
-        where table_schema = '{{ relation.schema }}'
-        and table_name = '{{ relation.identifier }}'
-        union all
-        select
-        column_name
-        , data_type
-        , character_maximum_length
-        , numeric_precision
-        , numeric_scale
-        , ordinal_position
-        from v_catalog.view_columns
-        where table_schema = '{{ relation.schema }}'
-        and table_name = '{{ relation.identifier }}'
-    ) t
-    order by ordinal_position
-  {% endcall %}
-  {% set table = load_result('get_columns_in_relation').table %}
-  {{ return(sql_convert_columns_in_relation(table)) }}
-{% endmacro %}
-
-{% macro vertica__create_view_as(relation, sql) %}
-  {% set sql_header = config.get('sql_header', none) %}
-
-  {{ sql_header if sql_header is not none }}
-  create or replace view {{ relation }} as (
-    {{ sql }}
-  );
-
-{% endmacro %}
-
-{% macro vertica__create_table_as(temporary, relation, sql) -%}
-  {%- set sql_header = config.get('sql_header', none) -%}
-
-  {{ sql_header if sql_header is not none }}
-
-  create {% if temporary: -%}local temporary{%- endif %} table
-    {{ relation.include(database=(not temporary), schema=(not temporary)) }}
-    {% if temporary: -%}on commit preserve rows{%- endif %}
-  as (
-    {{ sql }}
-  );
-{% endmacro %}
-
-{% macro vertica__make_temp_relation(base_relation, suffix) %}
-    {% set tmp_identifier = base_relation.identifier ~ suffix %}
-    {% do return(base_relation.incorporate(
-                                  path={
-                                    "identifier": tmp_identifier,
-                                    "schema": none,
-                                    "database": none
-                                  })) -%}
-{% endmacro %}
-
 {% macro vertica__get_catalog(information_schema, schemas) -%}
   {% call statement('get_catalog', fetch_result=True) %}
 
@@ -185,6 +48,52 @@
   {{ return(load_result('get_catalog').table) }}
 {% endmacro %}
 
-{% macro vertica__current_timestamp() -%}
-  current_timestamp
+
+{% macro vertica__information_schema_name(database) -%}
+  {%- if database -%}
+    {{ adapter.quote_as_configured(database, 'database') }}.v_catalog
+  {%- else -%}
+    v_catalog
+  {%- endif -%}
 {%- endmacro %}
+
+
+{% macro vertica__list_schemas(database) %}
+  {% call statement('list_schemas', fetch_result=True, auto_begin=False) %}
+    select schema_name
+    from v_catalog.schemata
+  {% endcall %}
+  {{ return(load_result('list_schemas').table) }}
+{% endmacro %}
+
+
+{% macro vertica__check_schema_exists(database, schema) -%}
+  {% call statement('check_schema_exists', fetch_result=True, auto_begin=False) -%}
+        select count(*)
+        from v_catalog.schemata
+        where schema_name='{{ schema }}'
+  {%- endcall %}
+  {{ return(load_result('check_schema_exists').table) }}
+{% endmacro %}
+
+
+{% macro vertica__list_relations_without_caching(schema_relation) %}
+  {% call statement('list_relations_without_caching', fetch_result=True) -%}
+    select
+      '{{ schema_relation.database }}' as database,
+      table_name as name,
+      table_schema as schema,
+      'table' as type
+    from v_catalog.tables
+    where table_schema ilike '{{ schema_relation.schema }}'
+    union all
+    select
+      '{{ schema_relation.database }}' as database,
+      table_name as name,
+      table_schema as schema,
+      'view' as type
+    from v_catalog.views
+    where table_schema ilike '{{ schema_relation.schema }}'
+  {% endcall %}
+  {{ return(load_result('list_relations_without_caching').table) }}
+{% endmacro %}
diff --git a/dbt/include/vertica/macros/adapters/persist_docs.sql b/dbt/include/vertica/macros/adapters/persist_docs.sql
@@ -0,0 +1,15 @@
+{% macro vertica__alter_column_comment(relation, column_dict) -%}
+  {{ exceptions.raise_not_implemented(
+    'alter_column_comment macro not implemented for adapter '+adapter.type()) }}
+{%- endmacro %}
+
+
+{% macro vertica__alter_relation_comment(relation, relation_comment) -%}
+  {{ exceptions.raise_not_implemented(
+    'alter_relation_comment macro not implemented for adapter '+adapter.type()) }}
+{%- endmacro %}
+
+
+{# 
+  No need to implement persist_docs(). Syntax supported by default. 
+#}