Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[staging] Throw error if model contract enforced #328

Merged
merged 1 commit into from
May 2, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
## dbt-databricks 1.5.0 (Release TBD)

### Under the hood
Throw an error if a model has an enforced contract. ([#322](https://github.com/databricks/dbt-databricks/pull/322))

## dbt-databricks 1.4.2 (February 17, 2023)

### Fixes
Expand Down
6 changes: 6 additions & 0 deletions dbt/include/databricks/macros/adapters.sql
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,12 @@
{% endmacro %}

{% macro databricks__persist_constraints(relation, model) %}
{# Model contracts are not currently supported. #}
{%- set contract_config = config.get('contract') -%}
{% if contract_config and contract_config.enforced %}
{{ exceptions.raise_compiler_error('Model contracts are not currently supported.') }}
{% endif %}

{% if config.get('persist_constraints', False) and config.get('file_format', 'delta') == 'delta' %}
{% do alter_table_add_constraints(relation, model.meta.constraints) %}
{% do alter_column_set_constraints(relation, model.columns) %}
Expand Down
98 changes: 98 additions & 0 deletions docs/databricks-dbt-constraints-vs-model-contracts.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
# dbt-databricks constraints vs DBT 1.5 model contracts

dbt-databricks constraints are enabled for a model by setting `persist_constraints: true` in the model configuration. Model contracts are enabled by setting `enforced: true` under the contract configuration.

```
models:
- name: table_model
config:
contract:
enforced: true
```

DBT model contracts enforce column names and datatypes. This means that **all** columns must be explicitly listed and have name and data_type properties.

dbt-databricks constraints list model level constraints under `meta: constraints:` while in DBT `constraints` is a property of the model.
```
dbt-databricks

models:
- name: incremental_model
meta:
constraints:
- name: id_greater_than_zero
condition: id > 0
```

```
model contract

models:
- name: table_model
constraints:
- type: check
columns: [id]
name: id_greater_than_zero
expression: "id > 0"
```

dbt-databricks constraints have a single column level constraint (currently limited to not_null) defined by the `meta: constraint:` property.
Model contracts have multiple column constraints listed under a columns `constraints` property.
```
dbt-databricks

columns:
- name: name
meta:
constraint: not_null
```

```
model contract

columns:
- name: name
data_type: string
constraints:
- type: not_null
```

Model contract constraint structure:
- **type** (required): dbt-databricks constraints do not have this property. DBT has not_null, check, primary_key, foreign_key, and custom types. dbt-databricks constraints currently support the equivalents of not_null and check.
- **expression**: Free text input to qualify the constraint. In dbt-databricks constraints this is the condition property. Note: in model contracts the expression text is contained by double quotes, the condition text in dbt-databricks constraints is not double quoted.
- **name** (optional in model contracts, required for check constraints in dbt-databricks constraints): Human-friendly name for this constraint.
- **columns** (model-level only): List of column names to apply the constraint over. dbt-databricks constraints do not have this property.


In a model contract a check constraint over a single column can be defined at either the model or the column level, but it is recommended that it be defined at the column level. Check constraints over multiple columns must be defined at the model level.
dbt-databricks check constraints are defined only at the model level.

```
dbt-databricks

models:
- name: my_model
meta:
constraints:
- name: id_greater_than_zero
condition: id > 0
columns:
- name: name
meta:
constraint: not_null
```

```
model contract

models:
- name: my_model
columns:
- name: name
data_type: integer
constraints:
- type: not_null
- type: check
name: id_greater_than_zero
expression: "id > 0"
```
12 changes: 12 additions & 0 deletions tests/integration/persist_constraints/models/schema.yml
Original file line number Diff line number Diff line change
Expand Up @@ -48,3 +48,15 @@ models:
meta:
constraint: not_null
- name: date

- name: table_model_contract
config:
contract:
enforced: true
columns:
- name: id
data_type: int
- name: name
data_type: string
- name: date
data_type: date
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{{config(materialized='table')}}

select * from {{ ref('seed') }}
25 changes: 25 additions & 0 deletions tests/integration/persist_constraints/test_persist_constraints.py
Original file line number Diff line number Diff line change
Expand Up @@ -259,3 +259,28 @@ def test_databricks_sql_endpoint(self):
@use_profile("databricks_uc_sql_endpoint")
def test_databricks_uc_sql_endpoint(self):
self.test_delta_constraints_disabled()


class TestModelContractNotSupported(TestConstraints):
def test_model_contract_not_supported(self):
model_name = "table_model_contract"
self.run_dbt(["seed"])
self.run_and_check_failure(
model_name, err_msg="Model contracts are not currently supported."
)

@use_profile("databricks_cluster")
def test_databricks_cluster(self):
self.test_model_contract_not_supported()

@use_profile("databricks_uc_cluster")
def test_databricks_uc_cluster(self):
self.test_model_contract_not_supported()

@use_profile("databricks_sql_endpoint")
def test_databricks_sql_endpoint(self):
self.test_model_contract_not_supported()

@use_profile("databricks_uc_sql_endpoint")
def test_databricks_uc_sql_endpoint(self):
self.test_model_contract_not_supported()