-
Notifications
You must be signed in to change notification settings - Fork 147
Description
Describe the bug
Building a project that has materialized views which reference dictionary objects can result is lost data.
Materialized views are unusual, in that a materialized view artifact form a previous build, is running throughout any subsequent dbt build process (except from the period in which it is being rebuilt). This means that if a materialized view depends on another artifact that is built by another model, and that artifact is dropped (even temporarily) during its model build, then the currently running materialized view can fail to process a batch of rows and hence data can be lost.
Steps to reproduce
- Make a dbt project that contains a materialized view that depends on a dictionary, such that the materialized view processes a large continuous data flow.
- Build the project to set the materialized view running
- Build the project again (possibly multiple times)
- Data will be lost and exceptions will appear in the clickhouse logs
Expected behaviour
The materialized view should not drop data as a result of dictionary not being found
dbt and/or ClickHouse server logs
<Error> executeQuery: Code: 36. DB::Exception: Dictionary ... not found
At a minimum dbt-clickhouse should use the DDL CREATE OR REPLACE DICTIONARY and not DROP DICTIONARY