ADAP-869: Support atomic replace in replace macro #8539

mikealfare · 2023-08-31T23:42:12Z

resolves #
docs dbt-labs/docs.getdbt.com/#

Problem

The current replace macro does not support replacing an existing relation with a target relation of the same type using an atomic "create and replace". Instead it tries to do the stage/backup/swap method which uses at least two statements (one to drop the existing and one to create the target).

Solution

Update the replace macro to first check if the types are the same, and if so, if the type is a replaceable type. Some relation types are not replaceable (e.g. materialized views in Postgres). If this check fails, then head down the stage/backup/swap route.

Checklist

I have read the contributing guide and understand what's expected of me
I have run this code in development and it appears to resolve the stated issue
This PR includes tests, or tests are not required/relevant for this PR
This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX

This reverts commit 46e8aef.

This reverts commit d898469.

This reverts commit d755c51.

This reverts commit 03a15e7.

# Conflicts: # core/dbt/include/global_project/macros/relations/drop.sql # core/dbt/include/global_project/macros/relations/rename.sql # plugins/postgres/dbt/include/postgres/macros/relations/materialized_view/alter.sql

…relations` so that generics are close to the relation specific macros that they reference; also aligns with adapter macro files structure, to look more familiar

…r adapters that have a unique list of relation types, e.g. dbt-snowflake

…nd can be addressed properly at a later date

…default from drop_relation

…t from dbt-core

mikealfare · 2023-08-31T23:58:55Z

Since postgres does not use replace in this way yet (it's not implemented for tables and views and materialized views don't support create or replace), it was tested against dbt-snowflake. The tests that used to use drop/create methods are now using create or replace.

# Conflicts: # core/dbt/adapters/base/relation.py # core/dbt/include/global_project/macros/relations/materialized_view/_replace.sql # core/dbt/include/global_project/macros/relations/materialized_view/alter.sql # core/dbt/include/global_project/macros/relations/replace.sql

github-actions · 2023-09-01T00:47:22Z

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the contributing guide.

mikealfare · 2023-09-01T00:51:51Z

...os/materializations/models/materialized_view/get_materialized_view_configuration_changes.sql

Moved to relations/materialized_view/alter.sql

mikealfare · 2023-09-01T00:52:57Z

core/dbt/include/global_project/macros/relations/materialized_view/_replace.sql

Replaced with relations/materialized_view/replace.sql, should have shown as a rename. That file was then updated to what shows below.

mikealfare · 2023-09-01T00:54:19Z

core/dbt/include/global_project/macros/materializations/models/view/helpers.sql

Contents were moved to relations/view/replace.sql.

mikealfare · 2023-09-01T00:55:20Z

plugins/postgres/dbt/include/postgres/macros/relations/materialized_view/_replace.sql

Replaced with relations/materialized_view/replace.sql, should have shown as a rename. The contents of that file were then updated to the below.

mikealfare · 2023-09-01T00:56:40Z

plugins/postgres/dbt/include/postgres/macros/relations/view/replace.sql

This was taken from the default create view statement in dbt-core. Postgres did not overwrite it. It was then updated to run a create or replace instead of just replace and whitespace was adjusted for readability; otherwise it was left untouched.

mikealfare · 2023-09-01T00:57:11Z

plugins/postgres/dbt/include/postgres/macros/materializations/materialized_view.sql

Contents of this file were moved to relations/materialized_view/alter.sql.

…o set defaults

aranke

LGTM, but obviously needs Adapters 👍🏽 as well.
One nit on whether we can make the logic a bit more dynamic instead of having these large switch statements everywhere.

mikealfare added 30 commits August 17, 2023 18:02

first draft of adding in table - materialized view swap

f0b07d0

Merge branch 'main' into feature/materialized-views/adap-814

24d9c42

tests pass

7372440

table/view/materialized view can all replace each other

d20f7dc

changie

28e1612

update renameable relations to a config

4483f6c

revert to just reformatting changes

03a15e7

revert to just reformatting changes

d755c51

revert to just reformatting changes

d898469

revert to just reformatting changes

46e8aef

Revert "revert to just reformatting changes"

3572593

This reverts commit 46e8aef.

Revert "revert to just reformatting changes"

4404b85

This reverts commit d898469.

Revert "revert to just reformatting changes"

e7909aa

This reverts commit d755c51.

Revert "revert to just reformatting changes"

71cc6ed

This reverts commit 03a15e7.

Merge branch 'main' into feature/materialized-views/adap-814

5dc4aea

# Conflicts: # core/dbt/include/global_project/macros/relations/drop.sql # core/dbt/include/global_project/macros/relations/rename.sql # plugins/postgres/dbt/include/postgres/macros/relations/materialized_view/alter.sql

migrate relations macros from macros/adapters/relations to `macros/…

38bbbdb

…relations` so that generics are close to the relation specific macros that they reference; also aligns with adapter macro files structure, to look more familiar

update comment to be more clear

a6c5c0e

loosen type on relations_that_can_be_renamed to StrEnum to account fo…

6dbaabb

…r adapters that have a unique list of relation types, e.g. dbt-snowflake

move drop macro to drop macro file

aaec703

remove view and table from generic create template, it's not needed a…

6159fd4

…nd can be addressed properly at a later date

align the behavior of get_drop_sql and drop_relation, adopt existing …

99a1bb4

…default from drop_relation

whitespace/readability

17aae86

add explicit ddl for drop statements instead of inheriting the defaul…

a31189e

…t from dbt-core

update renameable relations attribute on BaseRelation

24ebc37

update replace macro dependent macros to align with naming standards

a7db607

update type for mashumaro, update related test

8b643bd

move create and replace macros into their own files

f513194

add way to register replaceable relation types

2f6cc57

add interface for applying create or replace

9a9c163

move config changes into alter.sql in alignment with other adapters

37cbab8

mikealfare requested review from a team as code owners August 31, 2023 23:57

mikealfare requested review from Fleid and aranke August 31, 2023 23:57

mikealfare added 2 commits August 31, 2023 20:48

changie

81098e8

remove empty file

922235b

mikealfare commented Sep 1, 2023

View reviewed changes

mikealfare requested review from nathaniel-may and removed request for Fleid September 1, 2023 00:57

mikealfare added 4 commits August 31, 2023 20:59

add unit test for can be replaced

de446b6

fixed defaults for renameable_relations and replaceable_relations

9d2b521

fixed tests for new defaults

957a21d

update renameable_relations and replaceable_relations to frozensets t…

882aac9

…o set defaults

aranke approved these changes Sep 5, 2023

View reviewed changes

McKnight-42 approved these changes Sep 6, 2023

View reviewed changes

McKnight-42 requested review from McKnight-42 and aranke September 6, 2023 20:38

mikealfare merged commit 9716690 into main Sep 11, 2023
50 checks passed

mikealfare deleted the feature/clone/replace branch September 11, 2023 20:23

dataders mentioned this pull request Oct 5, 2023

new paradigm for creating replacing and dropping relations databricks/dbt-databricks#473

Closed

tlento mentioned this pull request Nov 6, 2023

Update typing-extensions version to >=4.4 #9012

Merged

5 tasks

tlento mentioned this pull request Feb 26, 2024

Update dbt-semantic-interfaces dependency to compatible range #9671

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ADAP-869: Support atomic replace in replace macro #8539

ADAP-869: Support atomic replace in replace macro #8539

mikealfare commented Aug 31, 2023

mikealfare commented Aug 31, 2023

github-actions bot commented Sep 1, 2023

mikealfare Sep 1, 2023

mikealfare Sep 1, 2023

mikealfare Sep 1, 2023

mikealfare Sep 1, 2023

mikealfare Sep 1, 2023

mikealfare Sep 1, 2023

aranke left a comment

ADAP-869: Support atomic replace in replace macro #8539

ADAP-869: Support atomic replace in replace macro #8539

Conversation

mikealfare commented Aug 31, 2023

Problem

Solution

Checklist

mikealfare commented Aug 31, 2023

github-actions bot commented Sep 1, 2023

mikealfare Sep 1, 2023

Choose a reason for hiding this comment

mikealfare Sep 1, 2023

Choose a reason for hiding this comment

mikealfare Sep 1, 2023

Choose a reason for hiding this comment

mikealfare Sep 1, 2023

Choose a reason for hiding this comment

mikealfare Sep 1, 2023

Choose a reason for hiding this comment

mikealfare Sep 1, 2023

Choose a reason for hiding this comment

aranke left a comment

Choose a reason for hiding this comment