Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LowCardinality/Nullable columns are incompatible with column constraints #274

Closed
rjoelnorgren opened this issue Apr 26, 2024 · 1 comment · Fixed by #276
Closed

LowCardinality/Nullable columns are incompatible with column constraints #274

rjoelnorgren opened this issue Apr 26, 2024 · 1 comment · Fixed by #276
Labels
bug Something isn't working

Comments

@rjoelnorgren
Copy link
Contributor

rjoelnorgren commented Apr 26, 2024

Describe the bug

Specifying a LowCardinality or Nullable type for a column results in a compilation error due to model contract failure.

Steps to reproduce

  1. Create a model with an an enforced contract
  2. Specify a column data_type as Nullable or LowCardinality

Expected behaviour

We can specify columns as Nullable or LowCardinality.

Code examples, such as models or profile settings

schema.yml

models:
  - name: nested_contract_model
    config:
      contract:
        enforced: true
      materialized: table
    columns:
      - name: low_card_column
        data_type: LowCardinality(String)
      - name: nullable_column
        data_type: Nullable(String)

low_card_model.sql

select toLowCardinality('hello') as low_card_column, null::Nullable(String) as nullable_column

dbt and/or ClickHouse server logs

dbt run -s nested_contract_model
20:32:45  Running with dbt=1.7.13
20:32:45  Registered adapter: clickhouse=1.7.6
20:32:45  Unable to do partial parsing because saved manifest not found. Starting full parse.
20:32:45  Found 3 models, 1 seed, 1 operation, 1 test, 3 sources, 0 exposures, 0 metrics, 431 macros, 0 groups, 0 semantic models
20:32:45  
20:32:46  
20:32:46  Running 1 on-run-start hook
20:32:46  1 of 1 START hook: jaffle_shop.on-run-start.0 .................................. [RUN]
20:32:46  1 of 1 OK hook: jaffle_shop.on-run-start.0 ..................................... [OK in 0.00s]
20:32:46  
20:32:46  Concurrency: 4 threads (target='dev')
20:32:46  
20:32:46  1 of 1 START sql table model `jaffle_shop`.`nested_contract_model` ............. [RUN]
20:32:46  1 of 1 ERROR creating sql table model `jaffle_shop`.`nested_contract_model` .... [ERROR in 0.03s]
20:32:46  
20:32:46  Finished running 1 table model, 1 hook in 0 hours 0 minutes and 0.18 seconds (0.18s).
20:32:46  
20:32:46  Completed with 1 error and 0 warnings:
20:32:46  
20:32:46    Compilation Error in model nested_contract_model (models/jaffle_shop/nested_contract_model.sql)
  This model has an enforced contract that failed.
  Please ensure the name, data_type, and number of columns in your contract match the columns in your model's definition.
  
  | column_name     | definition_type | contract_type          | mismatch_reason    |
  | --------------- | --------------- | ---------------------- | ------------------ |
  | low_card_column | String          | LowCardinality(String) | data type mismatch |
  | nullable_column | String          | Nullable(String)       | data type mismatch |
  
  
  > in macro clickhouse__get_assert_columns_equivalent (macros/column_spec_ddl.sql)
  > called by macro get_assert_columns_equivalent (macros/relations/column/columns_spec_ddl.sql)
  > called by macro create_table_or_empty (macros/materializations/table.sql)
  > called by macro clickhouse__create_table_as (macros/materializations/table.sql)
  > called by macro create_table_as (macros/relations/table/create.sql)
  > called by macro default__get_create_table_as_sql (macros/relations/table/create.sql)
  > called by macro get_create_table_as_sql (macros/relations/table/create.sql)
  > called by macro statement (macros/etc/statement.sql)
  > called by macro materialization_table_clickhouse (macros/materializations/table.sql)
  > called by model nested_contract_model (models/jaffle_shop/nested_contract_model.sql)
20:32:46  
20:32:46  Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1
@rjoelnorgren rjoelnorgren added the bug Something isn't working label Apr 26, 2024
@rjoelnorgren rjoelnorgren changed the title LowCardinality columns are incompatible with column constraints LowCardinality/Nullable columns are incompatible with column constraints Apr 26, 2024
@rjoelnorgren
Copy link
Contributor Author

A few way this could be resolved, but the underlying issues is related to correctly preserving the nesting of types (https://github.com/ClickHouse/dbt-clickhouse/blob/main/dbt/adapters/clickhouse/column.py#L123-L127). This is also important for #3 which would need to support cases such as AggregateFunction(uniq, Nullable(Int32)).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant