Merge pull request #30 from fivetran/MagicBot/dbt-utils-cross-db-migr…

…ation

Updates for dbt-utils to dbt-core cross-db macro migration

.buildkite/scripts/run_models.sh

@@ -20,4 +20,4 @@ dbt seed --target "$db" --full-refresh
 dbt run --target "$db" --full-refresh
 dbt test --target "$db"
 dbt run --vars '{using_address: false,using_bill: false, using_credit_memo: false, using_department: false, using_deposit: false, using_estimate: false, using_invoice: false, using_invoice_bundle: false, using_journal_entry: false, using_payment: false, using_refund_receipt: false, using_transfer: false, using_vendor_credit: false, using_sales_receipt: false}' --target "$db" --full-refresh
-dbt test --target "$db"
+dbt test --target "$db"

.github/workflows/check_docs.yml

@@ -0,0 +1,35 @@
+name: 'check docs'
+on:
+  push:
+    branches:
+      - main
+  pull_request:
+    branches:
+      - main
+
+jobs:
+  changed-files:
+    runs-on: ubuntu-latest
+    name: test changed-files
+    steps:
+      - uses: actions/checkout@v3
+        with:
+          fetch-depth: 0
+
+      - name: Get changed files
+        id: changed-files
+        uses: tj-actions/changed-files@v29.0.4
+        with:
+          files: docs/**
+
+      - name: Check to see if docs folder hasn't changed
+        if: steps.changed-files.outputs.any_changed == 'false'
+        run: |
+          echo "Docs have not been regenerated."
+          exit 1
+      
+      - name: Check if docs folder has changed
+        if: steps.changed-files.outputs.any_changed == 'true'
+        run: |
+          echo "Docs have been regenerated!"
+          exit 0 

.gitignore

@@ -2,5 +2,4 @@
 target/
 dbt_modules/
 logs/
-
 dbt_packages/

CHANGELOG.md

@@ -1,3 +1,43 @@
+# dbt_quickbooks_source v0.6.0
+## 🚨 Breaking Changes 🚨:
+[PR #30](https://github.com/fivetran/dbt_quickbooks_source/pull/30) includes the following breaking changes:
+- Dispatch update for dbt-utils to dbt-core cross-db macros migration. Specifically `{{ dbt_utils.<macro> }}` have been updated to `{{ dbt.<macro> }}` for the below macros:
+    - `any_value`
+    - `bool_or`
+    - `cast_bool_to_text`
+    - `concat`
+    - `date_trunc`
+    - `dateadd`
+    - `datediff`
+    - `escape_single_quotes`
+    - `except`
+    - `hash`
+    - `intersect`
+    - `last_day`
+    - `length`
+    - `listagg`
+    - `position`
+    - `replace`
+    - `right`
+    - `safe_cast`
+    - `split_part`
+    - `string_literal`
+    - `type_bigint`
+    - `type_float`
+    - `type_int`
+    - `type_numeric`
+    - `type_string`
+    - `type_timestamp`
+    - `array_append`
+    - `array_concat`
+    - `array_construct`
+- For `current_timestamp` and `current_timestamp_in_utc` macros, the dispatch AND the macro names have been updated to the below, respectively:
+    - `dbt.current_timestamp_backcompat`
+    - `dbt.current_timestamp_in_utc_backcompat`
+- `dbt_utils.surrogate_key` has also been updated to `dbt_utils.generate_surrogate_key`. Since the method for creating surrogate keys differ, we suggest all users do a `full-refresh` for the most accurate data. For more information, please refer to dbt-utils [release notes](https://github.com/dbt-labs/dbt-utils/releases) for this update.
+- `packages.yml` has been updated to reflect new default `fivetran/fivetran_utils` version, previously `[">=0.3.0", "<0.4.0"]` now `[">=0.4.0", "<0.5.0"]`.
+## Under the Hood
+- A GitHub workflow has been added to ensure the dbt docs are regenerated before each merge to the `main` release branch. 
 # dbt_quickbooks_source v0.5.1
 ## 🎉 Feature Updates 🎉
 - Addition of the new `stg_quickbooks__credit_card_payment_txn` model that may be used to track credit card transactions within QBO and used in downstream financial statement reports. ([#32](https://github.com/fivetran/dbt_quickbooks_source/pull/32))

README.md

@@ -1,4 +1,14 @@
-[![Apache License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
+<p align="center">
+    <a alt="License"
+        href="https://github.com/fivetran/dbt_netsuite_source/blob/main/LICENSE">
+        <img src="https://img.shields.io/badge/License-Apache%202.0-blue.svg" /></a>
+    <a alt="dbt-core">
+        <img src="https://img.shields.io/badge/dbt_Core™_version->=1.3.0_<2.0.0-orange.svg" /></a>
+    <a alt="Maintained?">
+        <img src="https://img.shields.io/badge/Maintained%3F-yes-green.svg" /></a>
+    <a alt="PRs">
+        <img src="https://img.shields.io/badge/Contributions-welcome-blueviolet" /></a>
+</p>
 # QuickBooks Source
 
 This package models QuickBooks data from [Fivetran's connector](https://fivetran.com/docs/applications/quickbooks). It uses data in the format described by [this ERD](https://fivetran.com/docs/applications/quickbooks#schemainformation).
@@ -25,7 +35,7 @@ Include in your `packages.yml`
 ```yaml
 packages:
   - package: fivetran/quickbooks_source
-    version: [">=0.5.0", "<0.6.0"]
+    version: [">=0.6.0", "<0.7.0"]
 ```
 
 ## Configuration
@@ -56,7 +66,7 @@ models:
 ```
 ### Disabling models
 
-This package takes into consideration that not every QuickBooks account utilizes the same transactional tables, and allows you to disable the corresponding functionality. By default, most variables' values are assumed to be `true` (with exception of purchase orders). Add variables for only the tables you want to disable or enable respectively:
+This package takes into consideration that not every QuickBooks account utilizes the same transactional tables, and allows you to disable the corresponding functionality. By default, most variables' values are assumed to be `true` (with exception of purchase orders and credit card payments). Add variables for only the tables you want to disable or enable respectively:
 
 ```yml
 # dbt_project.yml
@@ -79,6 +89,7 @@ vars:
     using_vendor_credit:  false         #disable if you don't have vendor credits in QuickBooks
     using_sales_receipt:  false         #disable if you don't have sales receipts in QuickBooks
     using_purchase_order: true          #enable if you want to include purchase orders in your staging models
+    using_credit_card_payment_txn: true #enable if you want to include credit card payment transactions in your staging models
 ```
 
 ## Contributions

dbt_project.yml

@@ -1,17 +1,13 @@
 config-version: 2
-
 name: 'quickbooks_source'
-version: '0.5.1'
-
-require-dbt-version: [">=1.0.0", "<2.0.0"]
-
+version: '0.6.0'
+require-dbt-version: [">=1.3.0", "<2.0.0"]
 models:
   quickbooks_source:
     +schema: quickbooks_staging
     materialized: table
     tmp:
       materialized: view
-
 vars:
   quickbooks_source:
     account: "{{ source('quickbooks','account') }}"
@@ -23,6 +19,7 @@ vars:
     bill: "{{ source('quickbooks','bill') }}"
     bundle_item: "{{ source('quickbooks','bundle_item') }}"
     bundle: "{{ source('quickbooks','bundle') }}"
+    credit_card_payment_txn: "{{ source('quickbooks','credit_card_payment_txn') }}"
     credit_memo_line: "{{ source('quickbooks','credit_memo_line') }}"
     credit_memo: "{{ source('quickbooks','credit_memo') }}"
     credit_card_payment_txn: "{{ source('quickbooks','credit_card_payment_txn') }}"
@@ -54,8 +51,6 @@ vars:
     vendor_credit_line: "{{ source('quickbooks','vendor_credit_line') }}"
     vendor_credit: "{{ source('quickbooks','vendor_credit') }}"
     vendor: "{{ source('quickbooks','vendor') }}"
-
-    #Variables if your company utilizes the below parent and child transactional tables.
     using_address: true
     using_bill: true
     using_credit_memo: true
@@ -71,7 +66,6 @@ vars:
     using_vendor_credit: true
     using_sales_receipt: true
     using_purchase_order: false
-
 clean-targets:
 - target
 - dbt_modules

integration_tests/.gitignore

@@ -1,3 +1,4 @@
 target/
 dbt_modules/
+dbt_packages/
 logs/

integration_tests/ci/sample.profiles.yml

@@ -51,4 +51,4 @@ integration_tests:
       schema: quickbooks_source_integrations_tests
       threads: 2
       token: "{{ env_var('CI_DATABRICKS_DBT_TOKEN') }}"
-      type: databricks
+      type: databricks

integration_tests/dbt_project.yml

@@ -1,17 +1,18 @@
 name: 'quickbooks_source_integration_tests'
-version: '0.5.1'
+version: '0.6.0'
+
 profile: 'integration_tests'
 config-version: 2
-
-models: 
+models:
   quickbooks_source:
     materialized: table
     tmp:
       materialized: view
-
 vars:
+  quickbooks_schema: quickbooks_source_integrations_tests
   quickbooks_source:
     using_purchase_order: True
+    using_credit_card_payment_txn: True
     account:                    "{{ ref('account_data') }}"
     address:                    "{{ ref('address_data') }}"
     bill_line:                  "{{ ref('bill_line_data') }}"
@@ -21,6 +22,7 @@ vars:
     bill:                       "{{ ref('bill_data') }}"
     bundle_item:                "{{ ref('bundle_item_data') }}"
     bundle:                     "{{ ref('bundle_data') }}"
+    credit_card_payment_txn:    "{{ ref('credit_card_payment_txn_data') }}"
     credit_memo:                "{{ ref('credit_memo_data') }}"
     credit_memo_line:           "{{ ref('credit_memo_line_data') }}"
     credit_card_payment_txn:    "{{ ref('credit_card_payment_txn_data') }}"
@@ -54,8 +56,10 @@ vars:
     vendor:                     "{{ ref('vendor_data') }}"
 
 seeds:
+  quickbooks_source_integration_tests:
     +quote_columns: "{{ true if target.type == 'redshift' else false }}"
 
 clean-targets:
-    - target
-    - dbt_modules
+- target
+- dbt_modules
+- dbt_packages

integration_tests/requirements.txt

@@ -1,7 +1,7 @@
-dbt-snowflake>=1.0.0,<1.3.0
-dbt-bigquery>=1.0.0,<1.3.0
-dbt-redshift>=1.0.0,<1.3.0
-dbt-postgres>=1.0.0,<1.3.0
-dbt-spark>=1.0.0,<1.3.0
-dbt-spark[PyHive]>=1.0.0,<1.3.0
-dbt-databricks>=1.0.0,<1.3.0
+dbt-snowflake>=1.3.0,<2.0.0
+dbt-bigquery>=1.3.0,<2.0.0
+dbt-redshift>=1.3.0,<2.0.0
+dbt-postgres>=1.3.0,<2.0.0
+dbt-spark>=1.3.0,<2.0.0
+dbt-spark[PyHive]>=1.3.0,<2.0.0
+dbt-databricks>=1.3.0,<2.0.0

macros/get_account_columns.sql

@@ -1,25 +1,25 @@
 {% macro get_account_columns() %}
 
 {% set columns = [
-    {"name": "_fivetran_synced", "datatype": dbt_utils.type_timestamp()},
-    {"name": "account_number", "datatype": dbt_utils.type_string()},
-    {"name": "account_sub_type", "datatype": dbt_utils.type_string()},
-    {"name": "account_type", "datatype": dbt_utils.type_string()},
+    {"name": "_fivetran_synced", "datatype": dbt.type_timestamp()},
+    {"name": "account_number", "datatype": dbt.type_string()},
+    {"name": "account_sub_type", "datatype": dbt.type_string()},
+    {"name": "account_type", "datatype": dbt.type_string()},
     {"name": "active", "datatype": "boolean"},
-    {"name": "balance", "datatype": dbt_utils.type_float()},
-    {"name": "balance_with_sub_accounts", "datatype": dbt_utils.type_float()},
-    {"name": "classification", "datatype": dbt_utils.type_string()},
-    {"name": "created_at", "datatype": dbt_utils.type_timestamp()},
-    {"name": "currency_id", "datatype": dbt_utils.type_string()},
-    {"name": "description", "datatype": dbt_utils.type_string()},
-    {"name": "fully_qualified_name", "datatype": dbt_utils.type_string()},
-    {"name": "id", "datatype": dbt_utils.type_string()},
-    {"name": "name", "datatype": dbt_utils.type_string()},
-    {"name": "parent_account_id", "datatype": dbt_utils.type_string()},
+    {"name": "balance", "datatype": dbt.type_float()},
+    {"name": "balance_with_sub_accounts", "datatype": dbt.type_float()},
+    {"name": "classification", "datatype": dbt.type_string()},
+    {"name": "created_at", "datatype": dbt.type_timestamp()},
+    {"name": "currency_id", "datatype": dbt.type_string()},
+    {"name": "description", "datatype": dbt.type_string()},
+    {"name": "fully_qualified_name", "datatype": dbt.type_string()},
+    {"name": "id", "datatype": dbt.type_string()},
+    {"name": "name", "datatype": dbt.type_string()},
+    {"name": "parent_account_id", "datatype": dbt.type_string()},
     {"name": "sub_account", "datatype": "boolean"},
-    {"name": "sync_token", "datatype": dbt_utils.type_string()},
-    {"name": "tax_code_id", "datatype": dbt_utils.type_string()},
-    {"name": "updated_at", "datatype": dbt_utils.type_timestamp()}
+    {"name": "sync_token", "datatype": dbt.type_string()},
+    {"name": "tax_code_id", "datatype": dbt.type_string()},
+    {"name": "updated_at", "datatype": dbt.type_timestamp()}
 ] %}
 
 {{ return(columns) }}

macros/get_address_columns.sql

@@ -1,19 +1,19 @@
 {% macro get_address_columns() %}
 
 {% set columns = [
-    {"name": "_fivetran_synced", "datatype": dbt_utils.type_timestamp()},
-    {"name": "city", "datatype": dbt_utils.type_string()},
-    {"name": "country", "datatype": dbt_utils.type_string()},
-    {"name": "country_sub_division_code", "datatype": dbt_utils.type_string()},
-    {"name": "id", "datatype": dbt_utils.type_string()},
-    {"name": "latitude", "datatype": dbt_utils.type_string()},
-    {"name": "line_1", "datatype": dbt_utils.type_string()},
-    {"name": "line_2", "datatype": dbt_utils.type_string()},
-    {"name": "line_3", "datatype": dbt_utils.type_string()},
-    {"name": "line_4", "datatype": dbt_utils.type_string()},
-    {"name": "line_5", "datatype": dbt_utils.type_string()},
-    {"name": "longitude", "datatype": dbt_utils.type_string()},
-    {"name": "postal_code", "datatype": dbt_utils.type_string()}
+    {"name": "_fivetran_synced", "datatype": dbt.type_timestamp()},
+    {"name": "city", "datatype": dbt.type_string()},
+    {"name": "country", "datatype": dbt.type_string()},
+    {"name": "country_sub_division_code", "datatype": dbt.type_string()},
+    {"name": "id", "datatype": dbt.type_string()},
+    {"name": "latitude", "datatype": dbt.type_string()},
+    {"name": "line_1", "datatype": dbt.type_string()},
+    {"name": "line_2", "datatype": dbt.type_string()},
+    {"name": "line_3", "datatype": dbt.type_string()},
+    {"name": "line_4", "datatype": dbt.type_string()},
+    {"name": "line_5", "datatype": dbt.type_string()},
+    {"name": "longitude", "datatype": dbt.type_string()},
+    {"name": "postal_code", "datatype": dbt.type_string()}
 ] %}
 
 {{ return(columns) }}

macros/get_bill_columns.sql

@@ -2,25 +2,25 @@
 
 {% set columns = [
     {"name": "_fivetran_deleted", "datatype": "boolean"},
-    {"name": "_fivetran_synced", "datatype": dbt_utils.type_timestamp()},
-    {"name": "balance", "datatype": dbt_utils.type_float()},
-    {"name": "created_at", "datatype": dbt_utils.type_timestamp()},
-    {"name": "currency_id", "datatype": dbt_utils.type_string()},
-    {"name": "department_id", "datatype": dbt_utils.type_string()},
-    {"name": "doc_number", "datatype": dbt_utils.type_string()},
+    {"name": "_fivetran_synced", "datatype": dbt.type_timestamp()},
+    {"name": "balance", "datatype": dbt.type_float()},
+    {"name": "created_at", "datatype": dbt.type_timestamp()},
+    {"name": "currency_id", "datatype": dbt.type_string()},
+    {"name": "department_id", "datatype": dbt.type_string()},
+    {"name": "doc_number", "datatype": dbt.type_string()},
     {"name": "due_date", "datatype": "date"},
-    {"name": "exchange_rate", "datatype": dbt_utils.type_float()},
-    {"name": "global_tax_calculation", "datatype": dbt_utils.type_string()},
-    {"name": "home_balance", "datatype": dbt_utils.type_float()},
-    {"name": "id", "datatype": dbt_utils.type_string()},
-    {"name": "payable_account_id", "datatype": dbt_utils.type_string()},
-    {"name": "private_note", "datatype": dbt_utils.type_string()},
-    {"name": "sales_term_id", "datatype": dbt_utils.type_string()},
-    {"name": "sync_token", "datatype": dbt_utils.type_string()},
-    {"name": "total_amount", "datatype": dbt_utils.type_float()},
+    {"name": "exchange_rate", "datatype": dbt.type_float()},
+    {"name": "global_tax_calculation", "datatype": dbt.type_string()},
+    {"name": "home_balance", "datatype": dbt.type_float()},
+    {"name": "id", "datatype": dbt.type_string()},
+    {"name": "payable_account_id", "datatype": dbt.type_string()},
+    {"name": "private_note", "datatype": dbt.type_string()},
+    {"name": "sales_term_id", "datatype": dbt.type_string()},
+    {"name": "sync_token", "datatype": dbt.type_string()},
+    {"name": "total_amount", "datatype": dbt.type_float()},
     {"name": "transaction_date", "datatype": "date"},
-    {"name": "updated_at", "datatype": dbt_utils.type_timestamp()},
-    {"name": "vendor_id", "datatype": dbt_utils.type_string()}
+    {"name": "updated_at", "datatype": dbt.type_timestamp()},
+    {"name": "vendor_id", "datatype": dbt.type_string()}
 ] %}
 
 {{ return(columns) }}