schema contract #594

sh-rp · 2023-08-27T15:54:39Z

issue #135
TODO:

migrate schemas to new version
add support for source level schema evolution control
add support for pipeline level overrides
add more tests for mixed schema evolution modes

netlify · 2023-08-27T15:54:44Z

✅ Deploy Preview for dlt-hub-docs canceled.

Name	Link
🔨 Latest commit	`340ed3d`
🔍 Latest deploy log	https://app.netlify.com/sites/dlt-hub-docs/deploys/6556a7a16f33c700080570e0

rudolfix · 2023-09-01T10:33:43Z

dlt/normalize/normalize.py

+                # coerce row of values into schema table, generating partial table with new columns if any
+                row, partial_table = schema.coerce_row(table_name, parent_table, row)
+                # check update
+                row, partial_table = schema.check_schema_update(table_name, row, partial_table, config.schema_update_mode)


don't call it if partial table is None! this is 99.99% of cases and this code is time critical

rudolfix · 2023-09-01T10:35:58Z

dlt/normalize/normalize.py

@@ -196,7 +203,7 @@ def map_parallel(self, schema: Schema, load_id: str, files: Sequence[str]) -> TM
        workers = self.pool._processes  # type: ignore
        chunk_files = self.group_worker_files(files, workers)
        schema_dict: TStoredSchema = schema.to_dict()
-        config_tuple = (self.normalize_storage.config, self.load_storage.config, self.config.destination_capabilities, schema_dict)
+        config_tuple = (self.config, self.normalize_storage.config, self.load_storage.config, self.config.destination_capabilities, schema_dict)


just FYI: this goes through process boundary and will be pickled. pay attention

rudolfix · 2023-09-01T10:37:36Z

dlt/common/schema/schema.py

@@ -174,7 +177,32 @@ def coerce_row(self, table_name: str, parent_table: str, row: StrAny) -> Tuple[D
                    updated_table_partial["columns"][new_col_name] = new_col_def

        return new_row, updated_table_partial
-
+
+    def check_schema_update(self, table_name: str, row: DictStrAny, partial_table: TPartialTableSchema, schema_update_mode: TSchemaUpdateMode) -> Tuple[DictStrAny, TPartialTableSchema]:


this is a solid implementation. but I'd like to describe it in the ticket (we can resue it for the docs) and maybe modify the requirements.

sh-rp · 2023-09-04T15:32:26Z

dlt/common/validation.py

            t = extract_optional_type(t)

-        if is_literal_type(t):
+        # TODO: support for union types?
+        if pk == "schema_evolution_settings":


we should have a look at this..

yes - and that should be trivial

# Conflicts: # dlt/common/schema/schema.py

…iltering

…cts pue to skip decoding

…ata filtering still missing

rudolfix · 2023-11-17T07:58:44Z

@sh-rp changes from previous version

extract is working on just one schema that is the most recent schema present in the pipeline. it updates this schema as the changes come
instead of filtering data in apply contracts in schema, I generate filters that get executed in extract and normalize phase. that's something we could do with Pydantic in the future
Pydantic is now tightly integrated with contracts - models are synthesized according to the settings.
I've improved arrow support. contracts now apply to it as well

sh-rp marked this pull request as ready for review August 28, 2023 12:44

sh-rp force-pushed the d#/data_contracts branch 3 times, most recently from 0daed01 to 3bbb8b1 Compare August 30, 2023 17:44

sh-rp added 5 commits August 31, 2023 09:43

basic schema freezing

78d6d71

small changes

09d8c63

temp

aec10b1

add new schema update mode

9441dcb

fix linting errors and one bug

edad4ad

sh-rp force-pushed the d#/data_contracts branch from 3bbb8b1 to edad4ad Compare August 31, 2023 07:47

move freeze code to schema

bef7ea4

rudolfix requested changes Sep 1, 2023

View reviewed changes

some work on schema evolution modes

fc6f083

sh-rp force-pushed the d#/data_contracts branch from 5e327c2 to fc6f083 Compare September 4, 2023 14:20

add tests

b36a74f

sh-rp commented Sep 4, 2023

View reviewed changes

small tests change

5659827

sh-rp force-pushed the d#/data_contracts branch from 4f575fd to 5659827 Compare September 4, 2023 15:41

sh-rp added 3 commits September 4, 2023 17:44

Merge branch 'devel' into d#/data_contracts

5343f4b

# Conflicts: # dlt/common/schema/schema.py

small fix

894875f

fix some tests

8bccfe5

sh-rp force-pushed the d#/data_contracts branch from 2481107 to 8bccfe5 Compare September 5, 2023 14:31

sh-rp added 3 commits September 5, 2023 16:59

add global override for schema evolution

cfd3f64

finish implemention of global override

e05855c

better tests

3f07127

sh-rp force-pushed the d#/data_contracts branch from 15cada5 to 05f4c1a Compare September 6, 2023 08:05

carry over schema settings on update

6308369

sh-rp force-pushed the d#/data_contracts branch from 05f4c1a to 6308369 Compare September 6, 2023 08:12

sh-rp changed the title ~~[wip] basic schema freezing~~ basic schema freezing Sep 6, 2023

rudolfix added 11 commits November 2, 2023 00:06

extracts on single pipeline schema

760cc43

allows to control relational normalizer descend with send

b645927

refactors data contract apply to generate filters instead of actual f…

8989a75

…iltering

detects if bytes string possibly contains pue characters

57842cc

applies schema contracts in item normalizer, uses binary stream, dete…

441299f

…cts pue to skip decoding

methods to remove and rename arrow columns, need arrow 12+

fc0eb47

implements contracts in extract, fixes issues in apply hints, arrow d…

9977227

…ata filtering still missing

always uses pipeline schema when extracting

d74242a

returns new items count from buffered write

e9344ee

bumps pyarrow to 12, temporary removes snowflake extra

e980396

Merge branch 'devel' into d#/data_contracts

5e2d131

rudolfix mentioned this pull request Nov 13, 2023

0.4.1 release notes and master merge #763

Closed

rudolfix added 8 commits November 13, 2023 16:10

fixes arrow imports and normalizer config

3dc4fa5

fixes normalizer config tests and pipeline state serialization

c76788a

normalizes arrow tables before saving

423a163

adds validation and model synth for contracts to pydantic helper

c1c32d6

splits extractor into files, improves pydantic validator

c24b643

runs tests on ci with minimal dependencies

2b97a4f

fixes deps in ci workflows

c35ec2d

re-adds snowflake connector

340ed3d

rudolfix added the sprint label Nov 17, 2023

rudolfix added 6 commits November 18, 2023 19:05

Merge branch 'devel' into d#/data_contracts

35c10b7

updates pydantic helper

2d260e8

improves contract violation exception

aa990f5

splits source and resource in extract, adds more tests

a6a782b

temp disable pydantic 1 tests

f4d2cac

fixes generic type parametrization on 3.8

f556584

rudolfix merged commit 02bac98 into devel Nov 21, 2023
37 of 40 checks passed

rudolfix deleted the d#/data_contracts branch November 21, 2023 09:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

schema contract #594

schema contract #594

sh-rp commented Aug 27, 2023 •

edited

Loading

netlify bot commented Aug 27, 2023 •

edited

Loading

rudolfix Sep 1, 2023

rudolfix Sep 1, 2023

rudolfix Sep 1, 2023

sh-rp Sep 4, 2023

rudolfix Sep 10, 2023

rudolfix commented Nov 17, 2023

schema contract #594

schema contract #594

Conversation

sh-rp commented Aug 27, 2023 • edited Loading

netlify bot commented Aug 27, 2023 • edited Loading

✅ Deploy Preview for dlt-hub-docs canceled.

rudolfix Sep 1, 2023

Choose a reason for hiding this comment

rudolfix Sep 1, 2023

Choose a reason for hiding this comment

rudolfix Sep 1, 2023

Choose a reason for hiding this comment

sh-rp Sep 4, 2023

Choose a reason for hiding this comment

rudolfix Sep 10, 2023

Choose a reason for hiding this comment

rudolfix commented Nov 17, 2023

sh-rp commented Aug 27, 2023 •

edited

Loading

netlify bot commented Aug 27, 2023 •

edited

Loading