Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

schema contract #594

Merged
merged 85 commits into from
Nov 21, 2023
Merged

schema contract #594

merged 85 commits into from
Nov 21, 2023

Commits on Aug 31, 2023

  1. basic schema freezing

    @sh-rp
    sh-rp committed Aug 31, 2023
    Configuration menu
    Copy the full SHA
    78d6d71 View commit details
    Browse the repository at this point in the history

  2. small changes

    @sh-rp
    sh-rp committed Aug 31, 2023
    Configuration menu
    Copy the full SHA
    09d8c63 View commit details
    Browse the repository at this point in the history

  3. temp

    @sh-rp
    sh-rp committed Aug 31, 2023
    Configuration menu
    Copy the full SHA
    aec10b1 View commit details
    Browse the repository at this point in the history

  4. add new schema update mode

    @sh-rp
    sh-rp committed Aug 31, 2023
    Configuration menu
    Copy the full SHA
    9441dcb View commit details
    Browse the repository at this point in the history

  5. fix linting errors and one bug

    @sh-rp
    sh-rp committed Aug 31, 2023
    Configuration menu
    Copy the full SHA
    edad4ad View commit details
    Browse the repository at this point in the history

  6. move freeze code to schema

    @sh-rp
    sh-rp committed Aug 31, 2023
    Configuration menu
    Copy the full SHA
    bef7ea4 View commit details
    Browse the repository at this point in the history

Commits on Sep 4, 2023

  1. some work on schema evolution modes

    @sh-rp
    sh-rp committed Sep 4, 2023
    Configuration menu
    Copy the full SHA
    fc6f083 View commit details
    Browse the repository at this point in the history

  2. add tests

    @sh-rp
    sh-rp committed Sep 4, 2023
    Configuration menu
    Copy the full SHA
    b36a74f View commit details
    Browse the repository at this point in the history

  3. small tests change

    @sh-rp
    sh-rp committed Sep 4, 2023
    Configuration menu
    Copy the full SHA
    5659827 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'devel' into d#/data_contracts 

    # Conflicts:
#	dlt/common/schema/schema.py
    @sh-rp
    sh-rp committed Sep 4, 2023
    Configuration menu
    Copy the full SHA
    5343f4b View commit details
    Browse the repository at this point in the history

Commits on Sep 5, 2023

  1. small fix

    @sh-rp
    sh-rp committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    894875f View commit details
    Browse the repository at this point in the history

  2. fix some tests

    @sh-rp
    sh-rp committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    8bccfe5 View commit details
    Browse the repository at this point in the history

  3. add global override for schema evolution

    @sh-rp
    sh-rp committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    cfd3f64 View commit details
    Browse the repository at this point in the history

  4. finish implemention of global override

    @sh-rp
    sh-rp committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    e05855c View commit details
    Browse the repository at this point in the history

Commits on Sep 6, 2023

  1. better tests

    @sh-rp
    sh-rp committed Sep 6, 2023
    Configuration menu
    Copy the full SHA
    3f07127 View commit details
    Browse the repository at this point in the history

  2. carry over schema settings on update

    @sh-rp
    sh-rp committed Sep 6, 2023
    Configuration menu
    Copy the full SHA
    6308369 View commit details
    Browse the repository at this point in the history

  3. add tests for single values

    @sh-rp
    sh-rp committed Sep 6, 2023
    Configuration menu
    Copy the full SHA
    ab0b8d7 View commit details
    Browse the repository at this point in the history

  4. small changes to tests and code

    @sh-rp
    sh-rp committed Sep 6, 2023
    Configuration menu
    Copy the full SHA
    6e99ed9 View commit details
    Browse the repository at this point in the history

  5. fix small error

    @sh-rp
    sh-rp committed Sep 6, 2023
    Configuration menu
    Copy the full SHA
    1a10ec4 View commit details
    Browse the repository at this point in the history

  6. add tests for data contract interaction

    @sh-rp
    sh-rp committed Sep 6, 2023
    Configuration menu
    Copy the full SHA
    175bee5 View commit details
    Browse the repository at this point in the history

Commits on Sep 7, 2023

  1. fix tests

    @sh-rp
    sh-rp committed Sep 7, 2023
    Configuration menu
    Copy the full SHA
    18b9341 View commit details
    Browse the repository at this point in the history

Commits on Sep 12, 2023

  1. Merge branch 'devel' into d#/data_contracts 

    # Conflicts:
#	dlt/extract/schema.py
#	dlt/pipeline/pipeline.py
#	tests/common/utils.py
    @sh-rp
    sh-rp committed Sep 12, 2023
    Configuration menu
    Copy the full SHA
    6dcaa7d View commit details
    Browse the repository at this point in the history

  2. some PR work

    @sh-rp
    sh-rp committed Sep 12, 2023
    Configuration menu
    Copy the full SHA
    881d79a View commit details
    Browse the repository at this point in the history

Commits on Sep 13, 2023

  1. update schema management

    @sh-rp
    sh-rp committed Sep 13, 2023
    Configuration menu
    Copy the full SHA
    e707580 View commit details
    Browse the repository at this point in the history

  2. fix schema related tests

    @sh-rp
    sh-rp committed Sep 13, 2023
    Configuration menu
    Copy the full SHA
    b3dc41d View commit details
    Browse the repository at this point in the history

  3. add nice schema tests

    @sh-rp
    sh-rp committed Sep 13, 2023
    Configuration menu
    Copy the full SHA
    ac8f766 View commit details
    Browse the repository at this point in the history

  4. add docs page

    @sh-rp
    sh-rp committed Sep 13, 2023
    Configuration menu
    Copy the full SHA
    869f278 View commit details
    Browse the repository at this point in the history

  5. small test fix

    @sh-rp
    sh-rp committed Sep 13, 2023
    Configuration menu
    Copy the full SHA
    0f22ba0 View commit details
    Browse the repository at this point in the history

Commits on Sep 14, 2023

  1. smaller PR fixes

    @sh-rp
    sh-rp committed Sep 14, 2023
    Configuration menu
    Copy the full SHA
    463c447 View commit details
    Browse the repository at this point in the history

Commits on Sep 17, 2023

  1. more work

    @sh-rp
    sh-rp committed Sep 17, 2023
    Configuration menu
    Copy the full SHA
    db3f447 View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2023

  1. tests update

    @rudolfix
    rudolfix committed Sep 18, 2023
    Configuration menu
    Copy the full SHA
    c17577a View commit details
    Browse the repository at this point in the history

Commits on Sep 25, 2023

  1. almost there

    @sh-rp
    sh-rp committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    fb1d224 View commit details
    Browse the repository at this point in the history

Commits on Sep 26, 2023

  1. tmp

    @sh-rp
    sh-rp committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    6a15fa2 View commit details
    Browse the repository at this point in the history

  2. fix freeze tests

    @sh-rp
    sh-rp committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    d66c2e6 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'devel' into d#/data_contracts 

    # Conflicts:
#	dlt/extract/decorators.py
#	dlt/extract/schema.py
#	dlt/pipeline/pipeline.py
#	docs/website/docs/reference/performance.md
    @sh-rp
    sh-rp committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    5e238c5 View commit details
    Browse the repository at this point in the history

  4. cleanup

    @sh-rp
    sh-rp committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    bf9da7e View commit details
    Browse the repository at this point in the history

  5. Merge branch 'devel' into d#/data_contracts 

    # Conflicts:
#	docs/website/docs/general-usage/schema.md
    @sh-rp
    sh-rp committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    9ba2496 View commit details
    Browse the repository at this point in the history

  6. create data contracts page

    @sh-rp
    sh-rp committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    038d03a View commit details
    Browse the repository at this point in the history

  7. small cleanup

    @sh-rp
    sh-rp committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    84384c3 View commit details
    Browse the repository at this point in the history

  8. add pydantic dep to destination tests

    @sh-rp
    sh-rp committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    44dfb69 View commit details
    Browse the repository at this point in the history

Commits on Sep 28, 2023

  1. Merge branch 'devel' into d#/data_contracts 

    # Conflicts:
#	dlt/common/schema/utils.py
    @sh-rp
    sh-rp committed Sep 28, 2023
    Configuration menu
    Copy the full SHA
    ece2bfc View commit details
    Browse the repository at this point in the history

Commits on Sep 29, 2023

  1. rename contract settings

    @sh-rp
    sh-rp committed Sep 29, 2023
    Configuration menu
    Copy the full SHA
    7b8f2d2 View commit details
    Browse the repository at this point in the history

  2. rename schema contract dict keys

    @sh-rp
    sh-rp committed Sep 29, 2023
    Configuration menu
    Copy the full SHA
    00c540b View commit details
    Browse the repository at this point in the history

  3. some work

    @sh-rp
    sh-rp committed Sep 29, 2023
    Configuration menu
    Copy the full SHA
    3ed4630 View commit details
    Browse the repository at this point in the history

  4. more work...

    @sh-rp
    sh-rp committed Sep 29, 2023
    Configuration menu
    Copy the full SHA
    333217c View commit details
    Browse the repository at this point in the history

Commits on Oct 1, 2023

  1. more work

    @sh-rp
    sh-rp committed Oct 1, 2023
    Configuration menu
    Copy the full SHA
    d69e54d View commit details
    Browse the repository at this point in the history

Commits on Oct 2, 2023

  1. move checking of new tables into extract function

    @sh-rp
    sh-rp committed Oct 2, 2023
    Configuration menu
    Copy the full SHA
    041da6d View commit details
    Browse the repository at this point in the history

  2. fix most tests

    @sh-rp
    sh-rp committed Oct 2, 2023
    Configuration menu
    Copy the full SHA
    b72a1a9 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'devel' into d#/data_contracts 

    # Conflicts:
#	tests/common/schema/test_filtering.py
#	tests/common/schema/test_versioning.py
#	tests/common/test_typing.py
#	tests/load/test_job_client.py
#	tests/load/utils.py
    @sh-rp
    sh-rp committed Oct 2, 2023
    Configuration menu
    Copy the full SHA
    f2abadf View commit details
    Browse the repository at this point in the history

  4. fix linter after merge

    @sh-rp
    sh-rp committed Oct 2, 2023
    Configuration menu
    Copy the full SHA
    2ae36e3 View commit details
    Browse the repository at this point in the history

  5. small cleanup

    @sh-rp
    sh-rp committed Oct 2, 2023
    Configuration menu
    Copy the full SHA
    d85e04f View commit details
    Browse the repository at this point in the history

Commits on Oct 10, 2023

  1. Merge branch 'devel' into d#/data_contracts

    @rudolfix
    rudolfix committed Oct 10, 2023
    Configuration menu
    Copy the full SHA
    1d7be25 View commit details
    Browse the repository at this point in the history

Commits on Oct 17, 2023

  1. Merge branch 'devel' into d#/data_contracts 

    # Conflicts:
#	dlt/common/schema/exceptions.py
#	dlt/common/schema/schema.py
#	dlt/common/schema/utils.py
#	dlt/extract/decorators.py
#	dlt/extract/extract.py
#	dlt/extract/schema.py
#	dlt/normalize/normalize.py
#	dlt/pipeline/pipeline.py
#	docs/website/sidebars.js
#	tests/load/test_job_client.py
    @sh-rp
    sh-rp committed Oct 17, 2023
    Configuration menu
    Copy the full SHA
    96785c4 View commit details
    Browse the repository at this point in the history

  2. post merge code updates

    @sh-rp
    sh-rp committed Oct 17, 2023
    Configuration menu
    Copy the full SHA
    302d909 View commit details
    Browse the repository at this point in the history

  3. small fixes

    @sh-rp
    sh-rp committed Oct 17, 2023
    Configuration menu
    Copy the full SHA
    4a1fab0 View commit details
    Browse the repository at this point in the history

Commits on Oct 18, 2023

  1. some cleanup

    @sh-rp
    sh-rp committed Oct 18, 2023
    Configuration menu
    Copy the full SHA
    903f000 View commit details
    Browse the repository at this point in the history

  2. update docs

    @sh-rp
    sh-rp committed Oct 18, 2023
    Configuration menu
    Copy the full SHA
    912dd8b View commit details
    Browse the repository at this point in the history

Commits on Oct 29, 2023

  1. Merge branch 'devel' into d#/data_contracts

    @rudolfix
    rudolfix committed Oct 29, 2023
    Configuration menu
    Copy the full SHA
    ef1b10f View commit details
    Browse the repository at this point in the history

Commits on Nov 1, 2023

  1. Merge branch 'devel' into d#/data_contracts

    @rudolfix
    rudolfix committed Nov 1, 2023
    Configuration menu
    Copy the full SHA
    256920e View commit details
    Browse the repository at this point in the history

  2. makes bumping version optional in Schema, preserves hashes on replace… 

    … schema content
    @rudolfix
    rudolfix committed Nov 1, 2023
    Configuration menu
    Copy the full SHA
    168f0da View commit details
    Browse the repository at this point in the history

  3. extracts on single pipeline schema

    @rudolfix
    rudolfix committed Nov 1, 2023
    Configuration menu
    Copy the full SHA
    760cc43 View commit details
    Browse the repository at this point in the history

Commits on Nov 12, 2023

  1. allows to control relational normalizer descend with send

    @rudolfix
    rudolfix committed Nov 12, 2023
    Configuration menu
    Copy the full SHA
    b645927 View commit details
    Browse the repository at this point in the history

  2. refactors data contract apply to generate filters instead of actual f… 

    …iltering
    @rudolfix
    rudolfix committed Nov 12, 2023
    Configuration menu
    Copy the full SHA
    8989a75 View commit details
    Browse the repository at this point in the history

  3. detects if bytes string possibly contains pue characters

    @rudolfix
    rudolfix committed Nov 12, 2023
    Configuration menu
    Copy the full SHA
    57842cc View commit details
    Browse the repository at this point in the history

  4. applies schema contracts in item normalizer, uses binary stream, dete… 

    …cts pue to skip decoding
    @rudolfix
    rudolfix committed Nov 12, 2023
    Configuration menu
    Copy the full SHA
    441299f View commit details
    Browse the repository at this point in the history

  5. methods to remove and rename arrow columns, need arrow 12+

    @rudolfix
    rudolfix committed Nov 12, 2023
    Configuration menu
    Copy the full SHA
    fc0eb47 View commit details
    Browse the repository at this point in the history

  6. implements contracts in extract, fixes issues in apply hints, arrow d… 

    …ata filtering still missing
    @rudolfix
    rudolfix committed Nov 12, 2023
    Configuration menu
    Copy the full SHA
    9977227 View commit details
    Browse the repository at this point in the history

  7. always uses pipeline schema when extracting

    @rudolfix
    rudolfix committed Nov 12, 2023
    Configuration menu
    Copy the full SHA
    d74242a View commit details
    Browse the repository at this point in the history

  8. returns new items count from buffered write

    @rudolfix
    rudolfix committed Nov 12, 2023
    Configuration menu
    Copy the full SHA
    e9344ee View commit details
    Browse the repository at this point in the history

  9. bumps pyarrow to 12, temporary removes snowflake extra

    @rudolfix
    rudolfix committed Nov 12, 2023
    Configuration menu
    Copy the full SHA
    e980396 View commit details
    Browse the repository at this point in the history

  10. Merge branch 'devel' into d#/data_contracts

    @rudolfix
    rudolfix committed Nov 12, 2023
    Configuration menu
    Copy the full SHA
    5e2d131 View commit details
    Browse the repository at this point in the history

Commits on Nov 13, 2023

  1. fixes arrow imports and normalizer config

    @rudolfix
    rudolfix committed Nov 13, 2023
    Configuration menu
    Copy the full SHA
    3dc4fa5 View commit details
    Browse the repository at this point in the history

  2. fixes normalizer config tests and pipeline state serialization

    @rudolfix
    rudolfix committed Nov 13, 2023
    Configuration menu
    Copy the full SHA
    c76788a View commit details
    Browse the repository at this point in the history

  3. normalizes arrow tables before saving

    @rudolfix
    rudolfix committed Nov 13, 2023
    Configuration menu
    Copy the full SHA
    423a163 View commit details
    Browse the repository at this point in the history

Commits on Nov 16, 2023

  1. adds validation and model synth for contracts to pydantic helper

    @rudolfix
    rudolfix committed Nov 16, 2023
    Configuration menu
    Copy the full SHA
    c1c32d6 View commit details
    Browse the repository at this point in the history

  2. splits extractor into files, improves pydantic validator

    @rudolfix
    rudolfix committed Nov 16, 2023
    Configuration menu
    Copy the full SHA
    c24b643 View commit details
    Browse the repository at this point in the history

  3. runs tests on ci with minimal dependencies

    @rudolfix
    rudolfix committed Nov 16, 2023
    Configuration menu
    Copy the full SHA
    2b97a4f View commit details
    Browse the repository at this point in the history

  4. fixes deps in ci workflows

    @rudolfix
    rudolfix committed Nov 16, 2023
    Configuration menu
    Copy the full SHA
    c35ec2d View commit details
    Browse the repository at this point in the history

  5. re-adds snowflake connector

    @rudolfix
    rudolfix committed Nov 16, 2023
    Configuration menu
    Copy the full SHA
    340ed3d View commit details
    Browse the repository at this point in the history

Commits on Nov 18, 2023

  1. Merge branch 'devel' into d#/data_contracts

    @rudolfix
    rudolfix committed Nov 18, 2023
    Configuration menu
    Copy the full SHA
    35c10b7 View commit details
    Browse the repository at this point in the history

Commits on Nov 19, 2023

  1. updates pydantic helper

    @rudolfix
    rudolfix committed Nov 19, 2023
    Configuration menu
    Copy the full SHA
    2d260e8 View commit details
    Browse the repository at this point in the history

  2. improves contract violation exception

    @rudolfix
    rudolfix committed Nov 19, 2023
    Configuration menu
    Copy the full SHA
    aa990f5 View commit details
    Browse the repository at this point in the history

  3. splits source and resource in extract, adds more tests

    @rudolfix
    rudolfix committed Nov 19, 2023
    Configuration menu
    Copy the full SHA
    a6a782b View commit details
    Browse the repository at this point in the history

  4. temp disable pydantic 1 tests

    @rudolfix
    rudolfix committed Nov 19, 2023
    Configuration menu
    Copy the full SHA
    f4d2cac View commit details
    Browse the repository at this point in the history

Commits on Nov 20, 2023

  1. fixes generic type parametrization on 3.8

    @rudolfix
    rudolfix committed Nov 20, 2023
    Configuration menu
    Copy the full SHA
    f556584 View commit details
    Browse the repository at this point in the history