Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature/performance-improvements #47

Merged

Conversation

fivetran-catfritz
Copy link
Contributor

@fivetran-catfritz fivetran-catfritz commented Apr 10, 2024

PR Overview

This PR will address the following Issue/Feature:

  • Internal ticket

This PR will result in the following new package version:

  • v0.10.0 breaking for materialization changes

Please provide the finalized CHANGELOG entry which details the relevant changes included in this PR:

🚨 Breaking Changes 🚨

  • Added column _fivetran_synced_date to model stg_netsuite2__transactions for use in downstream models.
  • To reduce storage, updated default materialization of staging models to views.

⚠️ Running a --full-refresh will be required if you have previously run these staging models as tables and get the following error:

Trying to create view <model path> but it currently exists as a table. Either drop <model path> manually, or run dbt with `--full-refresh` and dbt will drop it for you.

Under the Hood:

  • Added integration testing pipeline for Databricks SQL Warehouse.
  • Included auto-releaser GitHub Actions workflow to automate future releases.

PR Checklist

Basic Validation

Please acknowledge that you have successfully performed the following commands locally:

  • dbt run –full-refresh && dbt test
  • dbt run (if incremental models are present)

Before marking this PR as "ready for review" the following have been applied:

  • The appropriate issue has been linked, tagged, and properly assigned
  • All necessary documentation and version upgrades have been applied
  • docs were regenerated (unless this PR does not include any code or yml updates)
    • will regenerate docs in the release branch to avoid merge conflicts
  • BuildKite integration tests are passing
  • Detailed validation steps have been provided below

Detailed Validation

Please share any and all of your validation steps:

  • Updating materializations - no validation necessary.

Copy link
Contributor

@fivetran-joemarkiewicz fivetran-joemarkiewicz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is mainly good to go, but just have a few small comments before approving.

.buildkite/scripts/run_models.sh Outdated Show resolved Hide resolved
@@ -56,7 +60,7 @@ seeds:
due_date: timestamp
netsuite2_account_data:
+column_types:
eliminate: "{{ 'string' if target.type in ('bigquery', 'spark', 'databricks') else 'varchar' }}"
eliminate: "{{ 'string' if target.type in ('bigquery', 'spark', 'databricks', 'databricks-sql') else 'varchar' }}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The target.type for databricks-sql is still databricks. This update isn't necessary.

Copy link
Contributor

@fivetran-joemarkiewicz fivetran-joemarkiewicz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Comment on lines +2 to +5
# - package: fivetran/fivetran_utils
# version: [">=0.4.0", "<0.5.0"]
- git: https://github.com/fivetran/dbt_fivetran_utils.git
revision: feature/add-lookback-macro
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't forget to switch once live

@fivetran-catfritz fivetran-catfritz merged commit cfd6f63 into release/v0.10.0-apr-2024 Apr 22, 2024
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants