Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

allows naming conventions to be changed #998

Draft
wants to merge 91 commits into
base: devel
Choose a base branch
from
Draft

allows naming conventions to be changed #998

wants to merge 91 commits into from

Commits on Feb 20, 2024

  1. allows to decorate async function with dlt.source

    @rudolfix
    rudolfix committed Feb 20, 2024
    Configuration menu
    Copy the full SHA
    93995b9 View commit details
    Browse the repository at this point in the history

  2. adds pytest-async and updates pytest to 7.x

    @rudolfix
    rudolfix committed Feb 20, 2024
    Configuration menu
    Copy the full SHA
    4444878 View commit details
    Browse the repository at this point in the history

  3. fixes forked teardown issue 7.x

    @rudolfix
    rudolfix committed Feb 20, 2024
    Configuration menu
    Copy the full SHA
    b3b70f6 View commit details
    Browse the repository at this point in the history

Commits on Feb 21, 2024

  1. bumps deps for py 3.12

    @rudolfix
    rudolfix committed Feb 21, 2024
    Configuration menu
    Copy the full SHA
    f5d7a0a View commit details
    Browse the repository at this point in the history

  2. adds py 12 common tests

    @rudolfix
    rudolfix committed Feb 21, 2024
    Configuration menu
    Copy the full SHA
    83dc38a View commit details
    Browse the repository at this point in the history

  3. fixes typings after deps bump

    @rudolfix
    rudolfix committed Feb 21, 2024
    Configuration menu
    Copy the full SHA
    21ebfee View commit details
    Browse the repository at this point in the history

  4. bumps airflow, yanks duckdb to 0.9.2

    @rudolfix
    rudolfix committed Feb 21, 2024
    Configuration menu
    Copy the full SHA
    7985f9d View commit details
    Browse the repository at this point in the history

  5. fixes tests

    @rudolfix
    rudolfix committed Feb 21, 2024
    Configuration menu
    Copy the full SHA
    07f285e View commit details
    Browse the repository at this point in the history

  6. fixes pandas version

    @rudolfix
    rudolfix committed Feb 21, 2024
    Configuration menu
    Copy the full SHA
    06e441e View commit details
    Browse the repository at this point in the history

Commits on Feb 22, 2024

  1. adds 3.12 duckdb dep

    @rudolfix
    rudolfix committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    3e846a1 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'devel' into rfix/enables-async-source

    @rudolfix
    rudolfix committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    37b4a31 View commit details
    Browse the repository at this point in the history

Commits on Feb 24, 2024

  1. Merge branch 'devel' into rfix/enables-async-source

    @rudolfix
    rudolfix committed Feb 24, 2024
    Configuration menu
    Copy the full SHA
    934c167 View commit details
    Browse the repository at this point in the history

  2. adds right hand pipe operator

    @rudolfix
    rudolfix committed Feb 24, 2024
    Configuration menu
    Copy the full SHA
    7fa574d View commit details
    Browse the repository at this point in the history

  3. fixes docker ci build

    @rudolfix
    rudolfix committed Feb 24, 2024
    Configuration menu
    Copy the full SHA
    8c7942d View commit details
    Browse the repository at this point in the history

  4. adds docs on async sources and resources

    @rudolfix
    rudolfix committed Feb 24, 2024
    Configuration menu
    Copy the full SHA
    f951fc0 View commit details
    Browse the repository at this point in the history

Commits on Feb 25, 2024

  1. normalizes default hints and preferred types in schema

    @rudolfix
    rudolfix committed Feb 25, 2024
    Configuration menu
    Copy the full SHA
    387a7c7 View commit details
    Browse the repository at this point in the history

  2. defines pipeline state table in utils, column normalization in simple… 

    … regex
    @rudolfix
    rudolfix committed Feb 25, 2024
    Configuration menu
    Copy the full SHA
    88728e1 View commit details
    Browse the repository at this point in the history

  3. normalizes all identifiers used by relational normalizer, fixes other… 

    … modules
    @rudolfix
    rudolfix committed Feb 25, 2024
    Configuration menu
    Copy the full SHA
    1a53425 View commit details
    Browse the repository at this point in the history

  4. fixes sql job client to use normalized identifiers in queries

    @rudolfix
    rudolfix committed Feb 25, 2024
    Configuration menu
    Copy the full SHA
    8835023 View commit details
    Browse the repository at this point in the history

  5. runs state sync tests for lower and upper case naming conventions

    @rudolfix
    rudolfix committed Feb 25, 2024
    Configuration menu
    Copy the full SHA
    f4c504f View commit details
    Browse the repository at this point in the history

  6. fixes weaviate to use normalized identifiers in queries

    @rudolfix
    rudolfix committed Feb 25, 2024
    Configuration menu
    Copy the full SHA
    874cc29 View commit details
    Browse the repository at this point in the history

  7. partially fixes qdrant incorrect state and version retrieval queries

    @rudolfix
    rudolfix committed Feb 25, 2024
    Configuration menu
    Copy the full SHA
    c4e9f35 View commit details
    Browse the repository at this point in the history

  8. initial sql uppercase naming convention

    @rudolfix
    rudolfix committed Feb 25, 2024
    Configuration menu
    Copy the full SHA
    6345377 View commit details
    Browse the repository at this point in the history

Commits on Mar 8, 2024

  1. Merge branch 'devel' into rfix/allows-naming-conventions

    @rudolfix
    rudolfix committed Mar 8, 2024
    Configuration menu
    Copy the full SHA
    96a02ff View commit details
    Browse the repository at this point in the history

Commits on Mar 9, 2024

  1. adds native df readers to databricks and bigquery

    @rudolfix
    rudolfix committed Mar 9, 2024
    Configuration menu
    Copy the full SHA
    aef8cc2 View commit details
    Browse the repository at this point in the history

  2. adds casing identifier capability to support different casing in nami… 

    …ng conventions, fixes how identifiers are normalized in destinations
    @rudolfix
    rudolfix committed Mar 9, 2024
    Configuration menu
    Copy the full SHA
    a53c00b View commit details
    Browse the repository at this point in the history

  3. cleans typing for relational normalizer

    @rudolfix
    rudolfix committed Mar 9, 2024
    Configuration menu
    Copy the full SHA
    91f5780 View commit details
    Browse the repository at this point in the history

Commits on Mar 18, 2024

  1. renames escape functions

    @rudolfix
    rudolfix committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    5984824 View commit details
    Browse the repository at this point in the history

  2. destination capabilities for case fold and case sensitivity

    @rudolfix
    rudolfix committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    3458441 View commit details
    Browse the repository at this point in the history

  3. drops supports naming module and allows naming to be instance in conf… 

    …ig and schema
    @rudolfix
    rudolfix committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    55362b0 View commit details
    Browse the repository at this point in the history

  4. checks all tables in information schema in one go, observes case fold… 

    …ing and sensitivity in sql destinations
    @rudolfix
    rudolfix committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    b836dfe View commit details
    Browse the repository at this point in the history

  5. moves schema verification to destination utils

    @rudolfix
    rudolfix committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    e50bfaa View commit details
    Browse the repository at this point in the history

  6. adds method to remove processing hints from schema, helper functions … 

    …for schema settings, refactor, tests
    @rudolfix
    rudolfix committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    42d149f View commit details
    Browse the repository at this point in the history

  7. accepts naming convention instances when resolving configs

    @rudolfix
    rudolfix committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    c53808f View commit details
    Browse the repository at this point in the history

  8. fixes the cloning of schema in decorator, removes processing hints

    @rudolfix
    rudolfix committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    b97ae53 View commit details
    Browse the repository at this point in the history

  9. removes processing hints when saving imported schema

    @rudolfix
    rudolfix committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    0132c2f View commit details
    Browse the repository at this point in the history

  10. adds docs on naming conventions, removes technical docs

    @rudolfix
    rudolfix committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    2a7c5dd View commit details
    Browse the repository at this point in the history

Commits on Jun 4, 2024

  1. Merge branch 'devel' into rfix/allows-naming-conventions

    @rudolfix
    rudolfix committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    d502c7c View commit details
    Browse the repository at this point in the history

Commits on Jun 6, 2024

  1. Merge branch 'devel' into rfix/allows-naming-conventions

    @rudolfix
    rudolfix committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    3e7504b View commit details
    Browse the repository at this point in the history

Commits on Jun 11, 2024

  1. adds casing info to databrick caps, makes caps an instance attr

    @rudolfix
    rudolfix committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    3bb929f View commit details
    Browse the repository at this point in the history

  2. Merge branch 'devel' into rfix/allows-naming-conventions

    @rudolfix
    rudolfix committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    9f0920c View commit details
    Browse the repository at this point in the history

  3. adjusts destination casing in caps from schema naming and config

    @rudolfix
    rudolfix committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    724dc15 View commit details
    Browse the repository at this point in the history

  4. raises detailed schema identifier clash exceptions

    @rudolfix
    rudolfix committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    b58a118 View commit details
    Browse the repository at this point in the history

  5. adds is_case_sensitive and name to NamingConvention

    @rudolfix
    rudolfix committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    d190ea1 View commit details
    Browse the repository at this point in the history

  6. adds sanity check if _dlt prefix is preserved

    @rudolfix
    rudolfix committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    b445654 View commit details
    Browse the repository at this point in the history

  7. finds genric types in non generic classes deriving from generic

    @rudolfix
    rudolfix committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    ee8a95b View commit details
    Browse the repository at this point in the history

  8. uses casefold INSERT VALUES job column names

    @rudolfix
    rudolfix committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    eb30838 View commit details
    Browse the repository at this point in the history

  9. adds a method make_qualified_table_name_path that calculates componen… 

    …ts of fully qualified table name and uses it to query INFO SCHEMA
    @rudolfix
    rudolfix committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    558db91 View commit details
    Browse the repository at this point in the history

  10. adds casing info to destinations, caps as instance attrs, custom tabl… 

    …e name paths
    @rudolfix
    rudolfix committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    dea9669 View commit details
    Browse the repository at this point in the history

  11. adds naming convention to restore state tests, make them essential

    @rudolfix
    rudolfix committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    b1e2b09 View commit details
    Browse the repository at this point in the history

  12. fixes table builder tests

    @rudolfix
    rudolfix committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    210be70 View commit details
    Browse the repository at this point in the history

Commits on Jun 12, 2024

  1. removes processing hints when exporting schema to import folder, warn… 

    …s on schema import overriding local schema, warns on processing hints present
    @rudolfix
    rudolfix committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    95b703d View commit details
    Browse the repository at this point in the history

  2. allows to subclass INFO SCHEMA query generation and uses specialized … 

    …big query override
    @rudolfix
    rudolfix committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    4b72b77 View commit details
    Browse the repository at this point in the history

  3. uses correct schema escaping function in sql jobs

    @rudolfix
    rudolfix committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    ab39e06 View commit details
    Browse the repository at this point in the history

  4. passes pipeline state to package state via extract

    @rudolfix
    rudolfix committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    2ae3ad2 View commit details
    Browse the repository at this point in the history

  5. fixes optional normalizers module

    @rudolfix
    rudolfix committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    09b7731 View commit details
    Browse the repository at this point in the history

  6. excludes version_hash from pipeline state SELECT

    @rudolfix
    rudolfix committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    cfd3e5f View commit details
    Browse the repository at this point in the history

  7. passes pipeline state to package state pt.2

    @rudolfix
    rudolfix committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    0edbbfd View commit details
    Browse the repository at this point in the history

  8. re-enables sentry tests

    @rudolfix
    rudolfix committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    5769ba1 View commit details
    Browse the repository at this point in the history

  9. bumps qdrant client, makes test running for local version

    @rudolfix
    rudolfix committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    1f17a44 View commit details
    Browse the repository at this point in the history

  10. makes weaviate running

    @rudolfix
    rudolfix committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    71e418b View commit details
    Browse the repository at this point in the history

Commits on Jun 13, 2024

  1. uses schemata to find databases on athena

    @rudolfix
    rudolfix committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    ce414e1 View commit details
    Browse the repository at this point in the history

  2. uses api get_table for hidden dataset on bigquery to reflect schemas,… 

    … support case insensitive datasets
    @rudolfix
    rudolfix committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    bde61a9 View commit details
    Browse the repository at this point in the history

  3. adds naming conventions to two restore state tests

    @rudolfix
    rudolfix committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    036e3dd View commit details
    Browse the repository at this point in the history

  4. fixes escape identifiers to column escape

    @rudolfix
    rudolfix committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    8546763 View commit details
    Browse the repository at this point in the history

  5. fix conflicts in docs

    @rudolfix
    rudolfix committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    f57e286 View commit details
    Browse the repository at this point in the history

Commits on Jun 15, 2024

  1. adjusts capabilities in capabilities() method, uses config and naming… 

    … optionally
    @rudolfix
    rudolfix committed Jun 15, 2024
    Configuration menu
    Copy the full SHA
    cf50bd4 View commit details
    Browse the repository at this point in the history

  2. allows to add props to classes without vectorizer in weaviate

    @rudolfix
    rudolfix committed Jun 15, 2024
    Configuration menu
    Copy the full SHA
    72969ce View commit details
    Browse the repository at this point in the history

  3. moves caps function into factories, cleansup adapters and custom dest… 

    …ination
    @rudolfix
    rudolfix committed Jun 15, 2024
    Configuration menu
    Copy the full SHA
    656d5fc View commit details
    Browse the repository at this point in the history

  4. sentry_dsn

    @rudolfix
    rudolfix committed Jun 15, 2024
    Configuration menu
    Copy the full SHA
    bbd7fe6 View commit details
    Browse the repository at this point in the history

  5. adds basic destination reference tests

    @rudolfix
    rudolfix committed Jun 15, 2024
    Configuration menu
    Copy the full SHA
    a671508 View commit details
    Browse the repository at this point in the history

  6. fixes table builder tests

    @rudolfix
    rudolfix committed Jun 15, 2024
    Configuration menu
    Copy the full SHA
    81e0db9 View commit details
    Browse the repository at this point in the history

  7. fix deps and docs

    @rudolfix
    rudolfix committed Jun 15, 2024
    Configuration menu
    Copy the full SHA
    8a32793 View commit details
    Browse the repository at this point in the history

Commits on Jun 16, 2024

  1. fixes more tests

    @rudolfix
    rudolfix committed Jun 16, 2024
    Configuration menu
    Copy the full SHA
    0dc6dc8 View commit details
    Browse the repository at this point in the history

Commits on Jun 17, 2024

  1. case sensitivity docs stubs

    @rudolfix
    rudolfix committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    4a39795 View commit details
    Browse the repository at this point in the history

  2. fixes drop_pipeline fixture

    @rudolfix
    rudolfix committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    43d6d5f View commit details
    Browse the repository at this point in the history

  3. improves partial config generation for capabilities

    @rudolfix
    rudolfix committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    e3d998c View commit details
    Browse the repository at this point in the history

  4. adds snowflake csv support

    @rudolfix
    rudolfix committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    3aef3fd View commit details
    Browse the repository at this point in the history

  5. creates separate csv tests

    @rudolfix
    rudolfix committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    6df7a34 View commit details
    Browse the repository at this point in the history

Commits on Jun 19, 2024

  1. allows to import files into extract storage, adds import file writer … 

    …and spec
    @rudolfix
    rudolfix committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    57aec2e View commit details
    Browse the repository at this point in the history

  2. handles ImportFileMeta in extractor

    @rudolfix
    rudolfix committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    fee7af5 View commit details
    Browse the repository at this point in the history

  3. adds import file item normalizer and router to normalize

    @rudolfix
    rudolfix committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    96c7222 View commit details
    Browse the repository at this point in the history

  4. supports csv format config for snowflake

    @rudolfix
    rudolfix committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    116add0 View commit details
    Browse the repository at this point in the history

Commits on Jun 20, 2024

  1. removes realpath wherever possible and adds fast make_full_path to Fi… 

    …leStorage
    @rudolfix
    rudolfix committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    42eacaf View commit details
    Browse the repository at this point in the history

  2. adds additional methods to load_package storage to make listings faster

    @rudolfix
    rudolfix committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    3793d06 View commit details
    Browse the repository at this point in the history

Commits on Jun 21, 2024

  1. adds file_format to dlt.resource, uses preferred file format for dlt … 

    …state table
    @rudolfix
    rudolfix committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    88eec9c View commit details
    Browse the repository at this point in the history

  2. docs for importing files, file_format

    @rudolfix
    rudolfix committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    8e0f0a8 View commit details
    Browse the repository at this point in the history

  3. code improvements and tests

    @rudolfix
    rudolfix committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    b1c095c View commit details
    Browse the repository at this point in the history

  4. docs hard links note

    @rudolfix
    rudolfix committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    46ec732 View commit details
    Browse the repository at this point in the history

  5. Merge pull request #1479 from dlt-hub/feat/snowflake-csv-support 

    Feat/snowflake csv support
    @rudolfix
    rudolfix committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    2194b18 View commit details
    Browse the repository at this point in the history

  6. Merge branch 'devel' into rfix/allows-naming-conventions

    @rudolfix
    rudolfix committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    1384ed3 View commit details
    Browse the repository at this point in the history