Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Generic destination / sink decorator #1065

Merged
merged 51 commits into from
Mar 14, 2024
Merged

Generic destination / sink decorator #1065

merged 51 commits into from
Mar 14, 2024

Commits on Jan 30, 2024

  1. start sink

    @sh-rp
    sh-rp committed Jan 30, 2024
    Configuration menu
    Copy the full SHA
    8b6028b View commit details
    Browse the repository at this point in the history

  2. parquet sink prototype

    @sh-rp
    sh-rp committed Jan 30, 2024
    Configuration menu
    Copy the full SHA
    6a924d9 View commit details
    Browse the repository at this point in the history

  3. some more sink implementations

    @sh-rp
    sh-rp committed Jan 30, 2024
    Configuration menu
    Copy the full SHA
    83c1af6 View commit details
    Browse the repository at this point in the history

  4. finish first batch of helpers

    @sh-rp
    sh-rp committed Jan 30, 2024
    Configuration menu
    Copy the full SHA
    b972f23 View commit details
    Browse the repository at this point in the history

  5. add missing tests and fix linting

    @sh-rp
    sh-rp committed Jan 30, 2024
    Configuration menu
    Copy the full SHA
    9dabeff View commit details
    Browse the repository at this point in the history

  6. make configuratio more versatile

    @sh-rp
    sh-rp committed Jan 30, 2024
    Configuration menu
    Copy the full SHA
    af6defd View commit details
    Browse the repository at this point in the history

  7. implement sink function progress state

    @sh-rp
    sh-rp committed Jan 30, 2024
    Configuration menu
    Copy the full SHA
    4d730e8 View commit details
    Browse the repository at this point in the history

  8. move to iterator

    @sh-rp
    sh-rp committed Jan 30, 2024
    Configuration menu
    Copy the full SHA
    3b577b0 View commit details
    Browse the repository at this point in the history

  9. persist sink load state in pipeline state

    @sh-rp
    sh-rp committed Jan 30, 2024
    Configuration menu
    Copy the full SHA
    5527689 View commit details
    Browse the repository at this point in the history

  10. fix unrelated typo

    @sh-rp
    sh-rp committed Jan 30, 2024
    Configuration menu
    Copy the full SHA
    0657034 View commit details
    Browse the repository at this point in the history

  11. move sink state storage to loadpackage state

    @sh-rp
    sh-rp committed Jan 30, 2024
    Configuration menu
    Copy the full SHA
    b5db5b8 View commit details
    Browse the repository at this point in the history

  12. additional pr fixes

    @sh-rp
    sh-rp committed Jan 30, 2024
    Configuration menu
    Copy the full SHA
    189d24b View commit details
    Browse the repository at this point in the history

  13. disable creating empty state file on loadpackage init

    @sh-rp
    sh-rp committed Jan 30, 2024
    Configuration menu
    Copy the full SHA
    57ed090 View commit details
    Browse the repository at this point in the history

Commits on Jan 31, 2024

  1. add sink docs page

    @sh-rp
    sh-rp committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    4f53bc4 View commit details
    Browse the repository at this point in the history

Commits on Feb 1, 2024

  1. small changes

    @sh-rp
    sh-rp committed Feb 1, 2024
    Configuration menu
    Copy the full SHA
    c6c06ba View commit details
    Browse the repository at this point in the history

Commits on Feb 6, 2024

  1. Merge branch 'devel' into d#/data_sink_decorator 

    # Conflicts:
#	dlt/destinations/__init__.py
#	tests/utils.py
    @sh-rp
    sh-rp committed Feb 6, 2024
    Configuration menu
    Copy the full SHA
    2f6a15a View commit details
    Browse the repository at this point in the history

  2. make loadstorage state versioned and separate out common base functions

    @sh-rp
    sh-rp committed Feb 6, 2024
    Configuration menu
    Copy the full SHA
    374b267 View commit details
    Browse the repository at this point in the history

  3. restrict access of destinations to load package state in accessor fun… 

    …ctions
    @sh-rp
    sh-rp committed Feb 6, 2024
    Configuration menu
    Copy the full SHA
    daae33e View commit details
    Browse the repository at this point in the history

  4. fix tests

    @sh-rp
    sh-rp committed Feb 6, 2024
    Configuration menu
    Copy the full SHA
    644e6f3 View commit details
    Browse the repository at this point in the history

  5. add tests for state and new injectable context

    @sh-rp
    sh-rp committed Feb 6, 2024
    Configuration menu
    Copy the full SHA
    9930ad6 View commit details
    Browse the repository at this point in the history

  6. fix linter

    @sh-rp
    sh-rp committed Feb 6, 2024
    Configuration menu
    Copy the full SHA
    678d187 View commit details
    Browse the repository at this point in the history

Commits on Feb 7, 2024

  1. fix linter error

    @sh-rp
    sh-rp committed Feb 7, 2024
    Configuration menu
    Copy the full SHA
    376832d View commit details
    Browse the repository at this point in the history

  2. some pr fixes

    @sh-rp
    sh-rp committed Feb 7, 2024
    Configuration menu
    Copy the full SHA
    79fce9e View commit details
    Browse the repository at this point in the history

Commits on Feb 8, 2024

  1. Merge commit '17aea98527eab19f3300ee161114541f7eb2b5b5' into d#/data_… 

    …sink_decorator

# Conflicts:
#	dlt/pipeline/current.py
    @sh-rp
    sh-rp committed Feb 8, 2024
    Configuration menu
    Copy the full SHA
    8eb3e1a View commit details
    Browse the repository at this point in the history

  2. more pr fixes

    @sh-rp
    sh-rp committed Feb 8, 2024
    Configuration menu
    Copy the full SHA
    105569a View commit details
    Browse the repository at this point in the history

  3. small readme changes

    @sh-rp
    sh-rp committed Feb 8, 2024
    Configuration menu
    Copy the full SHA
    27b8b2c View commit details
    Browse the repository at this point in the history

Commits on Mar 4, 2024

  1. Merge branch 'devel' into d#/data_sink_decorator 

    # Conflicts:
#	dlt/common/storages/load_package.py
#	dlt/destinations/impl/athena/athena.py
#	dlt/destinations/impl/bigquery/bigquery.py
#	dlt/load/load.py
    @sh-rp
    sh-rp committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    e60f2f1 View commit details
    Browse the repository at this point in the history

  2. add load id to loadpackage info in current

    @sh-rp
    sh-rp committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    3229745 View commit details
    Browse the repository at this point in the history

  3. add support for directly passing through the naming convention to the… 

    … sink
    @sh-rp
    sh-rp committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    dbbbe7c View commit details
    Browse the repository at this point in the history

  4. add support for batch size zero (filepath passthrouh)

    @sh-rp
    sh-rp committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    db9b488 View commit details
    Browse the repository at this point in the history

Commits on Mar 5, 2024

  1. use patched version of flak8 encoding

    @sh-rp
    sh-rp committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    3c39f41 View commit details
    Browse the repository at this point in the history

  2. fix tests

    @sh-rp
    sh-rp committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    3dfcf39 View commit details
    Browse the repository at this point in the history

  3. add support for secrets and config in sink

    @sh-rp
    sh-rp committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    bc44618 View commit details
    Browse the repository at this point in the history

  4. update sink docs

    @sh-rp
    sh-rp committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    db8d0ed View commit details
    Browse the repository at this point in the history

  5. revert encodings branch

    @sh-rp
    sh-rp committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    d8719c1 View commit details
    Browse the repository at this point in the history

  6. fix small linting problem

    @sh-rp
    sh-rp committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    d7eb19d View commit details
    Browse the repository at this point in the history

Commits on Mar 6, 2024

  1. add support for config specs

    @sh-rp
    sh-rp committed Mar 6, 2024
    Configuration menu
    Copy the full SHA
    ef35502 View commit details
    Browse the repository at this point in the history

Commits on Mar 7, 2024

  1. add possibility to create a resolved partial

    @sh-rp
    sh-rp committed Mar 7, 2024
    Configuration menu
    Copy the full SHA
    2db3430 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'devel' into d#/data_sink_decorator 

    # Conflicts:
#	dlt/extract/incremental/__init__.py
    @sh-rp
    sh-rp committed Mar 7, 2024
    Configuration menu
    Copy the full SHA
    9582b9f View commit details
    Browse the repository at this point in the history

  3. add lock for resolving config 

    add test for nested configs
    @sh-rp
    sh-rp committed Mar 7, 2024
    Configuration menu
    Copy the full SHA
    f488210 View commit details
    Browse the repository at this point in the history

  4. change resolved partial method to dedicated function

    @sh-rp
    sh-rp committed Mar 7, 2024
    Configuration menu
    Copy the full SHA
    4d2563b View commit details
    Browse the repository at this point in the history

Commits on Mar 12, 2024

  1. change signatures in decorator 

    lock injection context for wrapped functions
small pr fixes
    @sh-rp
    sh-rp committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    12b5cbd View commit details
    Browse the repository at this point in the history

  2. fixes bug in inject wrapper refactor

    @sh-rp
    sh-rp committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    3e5f808 View commit details
    Browse the repository at this point in the history

  3. mark destination decorator as experimental in the docs

    @sh-rp
    sh-rp committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    53dc2f9 View commit details
    Browse the repository at this point in the history

Commits on Mar 13, 2024

  1. change injection context locking strategy 

    forward generic destiation call params into config
small fixes
    @sh-rp
    sh-rp committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    d1e1e38 View commit details
    Browse the repository at this point in the history

  2. make tests independent from gcp imports

    @sh-rp
    sh-rp committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    943ac3e View commit details
    Browse the repository at this point in the history

  3. move generic destination tests into common tests section destinations

    @sh-rp
    sh-rp committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    1f47a05 View commit details
    Browse the repository at this point in the history

  4. fix global instantiation test after file move

    @sh-rp
    sh-rp committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    b66c653 View commit details
    Browse the repository at this point in the history

  5. add tests for locking injection context

    @sh-rp
    sh-rp committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    31b661b View commit details
    Browse the repository at this point in the history

  6. make inject test a bit better 

    make simple test for loading load package without state
    @sh-rp
    sh-rp committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    97ed409 View commit details
    Browse the repository at this point in the history

Commits on Mar 14, 2024

  1. skip generic destination in init test

    @sh-rp
    sh-rp committed Mar 14, 2024
    Configuration menu
    Copy the full SHA
    246df72 View commit details
    Browse the repository at this point in the history