Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dbt test command recreates the whole database #235

Open
kmogilevskii opened this issue Jan 9, 2024 · 0 comments
Open

dbt test command recreates the whole database #235

kmogilevskii opened this issue Jan 9, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@kmogilevskii
Copy link

I have a dbt model that runs certain transformations incrementally on the ClickHouse table. Also, I wrote a generic dbt test to run after the model.

The model's config is as following:

{{ config (
    tags="my_tag",
    materialized = "incremental",
    order_by = ["col1", "col2", "col3"],
    partition_by = ['toYYYYMMDD("date_column")'],
)}}

The model's test defined in the schema.yml file like this

models:
  - name: model_name
    description: model description
    config:
      materialized: table
    columns:
      - name: date_column
        tests:
          - not_null:
              config:
                where: date_column > toStartOfMonth(today())
          - generic_test_name:
              arg1: 1
              arg2: MONTH
              config:
                severity: error
                error_if: ">100"
                warn_if: ">0"

dbt test command in case of the error (more than 100 records produced) saves the results in the separate database and table in it (ClickHouse doesn't have a notion of schema, just database and tables) My problem is that I can't find in the documentation how to prevent it to recreate a database and the tables in it for the results of the test run in case there were an actual error. It always recreates it on every run and my user that's gonna run it won't have the permissions to create a database. Plus, I want it to append the error records incrementally to the existing tables as well.

@kmogilevskii kmogilevskii added the bug Something isn't working label Jan 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant