Skip to content

Replacing dictionary is not atomic #392

@the4thamigo-uk

Description

@the4thamigo-uk

Describe the bug

Building a project that has materialized views which reference dictionary objects can result is lost data.

Materialized views are unusual, in that a materialized view artifact form a previous build, is running throughout any subsequent dbt build process (except from the period in which it is being rebuilt). This means that if a materialized view depends on another artifact that is built by another model, and that artifact is dropped (even temporarily) during its model build, then the currently running materialized view can fail to process a batch of rows and hence data can be lost.

Steps to reproduce

  1. Make a dbt project that contains a materialized view that depends on a dictionary, such that the materialized view processes a large continuous data flow.
  2. Build the project to set the materialized view running
  3. Build the project again (possibly multiple times)
  4. Data will be lost and exceptions will appear in the clickhouse logs

Expected behaviour

The materialized view should not drop data as a result of dictionary not being found

dbt and/or ClickHouse server logs

<Error> executeQuery: Code: 36. DB::Exception: Dictionary  ... not found

At a minimum dbt-clickhouse should use the DDL CREATE OR REPLACE DICTIONARY and not DROP DICTIONARY

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions