Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(ingest): redshift - Redshift rework #6906

Merged
merged 83 commits into from
Apr 12, 2023
Merged

feat(ingest): redshift - Redshift rework #6906

merged 83 commits into from
Apr 12, 2023

Commits on Dec 28, 2022

  1. Refactoring container creation 

    Using auto-state handler in bigquery, sql-common and snowflake
    @treff7es
    treff7es committed Dec 28, 2022
    Configuration menu
    Copy the full SHA
    58ab3cf View commit details
    Browse the repository at this point in the history

  2. Adding back accidentally removed imports

    @treff7es
    treff7es committed Dec 28, 2022
    Configuration menu
    Copy the full SHA
    c809aa8 View commit details
    Browse the repository at this point in the history

  3. Adding back line breaks as well

    @treff7es
    treff7es committed Dec 28, 2022
    Configuration menu
    Copy the full SHA
    fae7ec1 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'master' into sql_common_refactor

    @treff7es
    treff7es committed Dec 28, 2022
    Configuration menu
    Copy the full SHA
    a71f8dc View commit details
    Browse the repository at this point in the history

  5. Remove unneeded line

    @treff7es
    treff7es committed Dec 28, 2022
    Configuration menu
    Copy the full SHA
    cb219bd View commit details
    Browse the repository at this point in the history

  6. Black formatting

    @treff7es
    treff7es committed Dec 28, 2022
    Configuration menu
    Copy the full SHA
    d3e6407 View commit details
    Browse the repository at this point in the history

  7. Isorting

    @treff7es
    treff7es committed Dec 28, 2022
    Configuration menu
    Copy the full SHA
    93df8fc View commit details
    Browse the repository at this point in the history

  8. Fixing return types

    @treff7es
    treff7es committed Dec 28, 2022
    Configuration menu
    Copy the full SHA
    cf074f0 View commit details
    Browse the repository at this point in the history

  9. Black formatting

    @treff7es
    treff7es committed Dec 28, 2022
    Configuration menu
    Copy the full SHA
    42004e5 View commit details
    Browse the repository at this point in the history

  10. Add option to set container name differently than the key name

    @treff7es
    treff7es committed Dec 28, 2022
    Configuration menu
    Copy the full SHA
    8fcdb7a View commit details
    Browse the repository at this point in the history

Commits on Dec 29, 2022

  1. Fix snowflake container generation

    @treff7es
    treff7es committed Dec 29, 2022
    Configuration menu
    Copy the full SHA
    e985a20 View commit details
    Browse the repository at this point in the history

  2. Fixing 2 tier container generation

    @treff7es
    treff7es committed Dec 29, 2022
    Configuration menu
    Copy the full SHA
    9af84c9 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' into sql_common_refactor

    @treff7es
    treff7es committed Dec 29, 2022
    Configuration menu
    Copy the full SHA
    be580ee View commit details
    Browse the repository at this point in the history

  4. isorting

    @treff7es
    treff7es committed Dec 29, 2022
    Configuration menu
    Copy the full SHA
    a626a9e View commit details
    Browse the repository at this point in the history

  5. Removing unused import

    @treff7es
    treff7es committed Dec 29, 2022
    Configuration menu
    Copy the full SHA
    ade9fa2 View commit details
    Browse the repository at this point in the history

  6. Fixing presto container generation

    @treff7es
    treff7es committed Dec 29, 2022
    Configuration menu
    Copy the full SHA
    0da0c55 View commit details
    Browse the repository at this point in the history

  7. Athena inherits from two tier db 

    Fix presto on hive container generation
    @treff7es
    treff7es committed Dec 29, 2022
    Configuration menu
    Copy the full SHA
    2268cad View commit details
    Browse the repository at this point in the history

  8. Reverting Athena changes

    @treff7es
    treff7es committed Dec 29, 2022
    Configuration menu
    Copy the full SHA
    480497f View commit details
    Browse the repository at this point in the history

  9. Merge branch 'master' into sql_common_refactor

    @treff7es
    treff7es committed Dec 29, 2022
    Configuration menu
    Copy the full SHA
    e017f44 View commit details
    Browse the repository at this point in the history

Commits on Dec 30, 2022

  1. Merge branch 'master' into sql_common_refactor 

    # Conflicts:
#	metadata-ingestion/src/datahub/ingestion/source/snowflake/snowflake_v2.py
#	metadata-ingestion/src/datahub/ingestion/source/sql/bigquery.py
    @treff7es
    treff7es committed Dec 30, 2022
    Configuration menu
    Copy the full SHA
    d6c1b7d View commit details
    Browse the repository at this point in the history

  2. Initial commit

    @treff7es
    treff7es committed Dec 30, 2022
    Configuration menu
    Copy the full SHA
    defaff6 View commit details
    Browse the repository at this point in the history

  3. Extracting out common methods, fixign mypy issues

    @treff7es
    treff7es committed Dec 30, 2022
    Configuration menu
    Copy the full SHA
    1f825e4 View commit details
    Browse the repository at this point in the history

  4. - Adding connection test 

    - Adding more detailed reporting
    @treff7es
    treff7es committed Dec 30, 2022
    Configuration menu
    Copy the full SHA
    b522948 View commit details
    Browse the repository at this point in the history

  5. Redshift rework

    @treff7es
    treff7es committed Dec 30, 2022
    Configuration menu
    Copy the full SHA
    0b56fbc View commit details
    Browse the repository at this point in the history

  6. Fixing issues after merge

    @treff7es
    treff7es committed Dec 30, 2022
    Configuration menu
    Copy the full SHA
    7b5b420 View commit details
    Browse the repository at this point in the history

  7. Fixing accidental change

    @treff7es
    treff7es committed Dec 30, 2022
    Configuration menu
    Copy the full SHA
    d3cb087 View commit details
    Browse the repository at this point in the history

  8. Flake, black, mypy fixes

    @treff7es
    treff7es committed Dec 30, 2022
    Configuration menu
    Copy the full SHA
    df55103 View commit details
    Browse the repository at this point in the history

Commits on Jan 9, 2023

  1. Adding fixes for Redshift source

    @treff7es
    treff7es committed Jan 9, 2023
    Configuration menu
    Copy the full SHA
    2945d27 View commit details
    Browse the repository at this point in the history

Commits on Jan 24, 2023

  1. Merge branch 'master' into redshift-rework 

    # Conflicts:
#	metadata-ingestion/src/datahub/ingestion/source/bigquery_v2/bigquery.py
#	metadata-ingestion/src/datahub/ingestion/source/bigquery_v2/profiler.py
#	metadata-ingestion/src/datahub/ingestion/source/snowflake/snowflake_v2.py
#	metadata-ingestion/src/datahub/ingestion/source/sql/athena.py
#	metadata-ingestion/src/datahub/ingestion/source/sql/presto_on_hive.py
#	metadata-ingestion/src/datahub/ingestion/source/sql/sql_common.py
#	metadata-ingestion/src/datahub/ingestion/source/sql/sql_utils.py
#	metadata-ingestion/src/datahub/ingestion/source/sql/two_tier_sql_source.py
    @treff7es
    treff7es committed Jan 24, 2023
    Configuration menu
    Copy the full SHA
    c4106f4 View commit details
    Browse the repository at this point in the history

  2. Resolving merge conflicts

    @treff7es
    treff7es committed Jan 24, 2023
    Configuration menu
    Copy the full SHA
    aecafda View commit details
    Browse the repository at this point in the history

  3. Fixing wrap_aspect_as_workunit

    @treff7es
    treff7es committed Jan 24, 2023
    Configuration menu
    Copy the full SHA
    7753ffb View commit details
    Browse the repository at this point in the history

  4. Merge branch 'master' into redshift-rework

    @treff7es
    treff7es committed Jan 24, 2023
    Configuration menu
    Copy the full SHA
    ceb0b37 View commit details
    Browse the repository at this point in the history

Commits on Jan 25, 2023

  1. Fixing usage test

    @treff7es
    treff7es committed Jan 25, 2023
    Configuration menu
    Copy the full SHA
    12625b3 View commit details
    Browse the repository at this point in the history

  2. Fixing linter error

    @treff7es
    treff7es committed Jan 25, 2023
    Configuration menu
    Copy the full SHA
    442c8e0 View commit details
    Browse the repository at this point in the history

  3. Adding missing test file

    @treff7es
    treff7es committed Jan 25, 2023
    Configuration menu
    Copy the full SHA
    9535790 View commit details
    Browse the repository at this point in the history

Commits on Jan 26, 2023

  1. various fixes

    @treff7es
    treff7es committed Jan 26, 2023
    Configuration menu
    Copy the full SHA
    4332e07 View commit details
    Browse the repository at this point in the history

  2. Adding unload lineage code as well

    @treff7es
    treff7es committed Jan 26, 2023
    Configuration menu
    Copy the full SHA
    dae7284 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' into redshift-rework

    @treff7es
    treff7es committed Jan 26, 2023
    Configuration menu
    Copy the full SHA
    5b6881b View commit details
    Browse the repository at this point in the history

  4. Adding quoting to patch path

    @treff7es
    treff7es committed Jan 26, 2023
    Configuration menu
    Copy the full SHA
    4efc105 View commit details
    Browse the repository at this point in the history

Commits on Jan 31, 2023

  1. Merge branch 'master' into redshift-rework

    @treff7es
    treff7es committed Jan 31, 2023
    Configuration menu
    Copy the full SHA
    22a1f6a View commit details
    Browse the repository at this point in the history

  2. PR review fixes

    @treff7es
    treff7es committed Jan 31, 2023
    Configuration menu
    Copy the full SHA
    d29dd53 View commit details
    Browse the repository at this point in the history

Commits on Feb 6, 2023

  1. Adding support for complex types in Redshift

    @treff7es
    treff7es committed Feb 6, 2023
    Configuration menu
    Copy the full SHA
    920648f View commit details
    Browse the repository at this point in the history

  2. Renaming non-binding lineage parser to generic view ddl sql parser

    @treff7es
    treff7es committed Feb 6, 2023
    Configuration menu
    Copy the full SHA
    da73a74 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' into redshift-rework

    @treff7es
    treff7es committed Feb 6, 2023
    Configuration menu
    Copy the full SHA
    9264bfa View commit details
    Browse the repository at this point in the history

Commits on Feb 7, 2023

  1. Updating docs 

    Moving queries into a separate class
    @treff7es
    treff7es committed Feb 7, 2023
    Configuration menu
    Copy the full SHA
    c0d4461 View commit details
    Browse the repository at this point in the history

Commits on Feb 28, 2023

  1. Merge branch 'master' into redshift-rework

    @treff7es
    treff7es committed Feb 28, 2023
    Configuration menu
    Copy the full SHA
    99ae34f View commit details
    Browse the repository at this point in the history

  2. Fixing build issues

    @treff7es
    treff7es committed Feb 28, 2023
    Configuration menu
    Copy the full SHA
    7234509 View commit details
    Browse the repository at this point in the history

  3. Addressing some pr review comments

    @treff7es
    treff7es committed Feb 28, 2023
    Configuration menu
    Copy the full SHA
    26fe103 View commit details
    Browse the repository at this point in the history

  4. Addressing even more pr review comments

    @treff7es
    treff7es committed Feb 28, 2023
    Configuration menu
    Copy the full SHA
    54556a0 View commit details
    Browse the repository at this point in the history

  5. Additional pr review fixes

    @treff7es
    treff7es committed Feb 28, 2023
    Configuration menu
    Copy the full SHA
    a497a25 View commit details
    Browse the repository at this point in the history

  6. Additional pr review fixes

    @treff7es
    treff7es committed Feb 28, 2023
    Configuration menu
    Copy the full SHA
    c2fa727 View commit details
    Browse the repository at this point in the history

Commits on Mar 1, 2023

  1. Merge branch 'master' into redshift-rework

    @treff7es
    treff7es committed Mar 1, 2023
    Configuration menu
    Copy the full SHA
    a44b598 View commit details
    Browse the repository at this point in the history

Commits on Mar 9, 2023

  1. Addressing more pr review comment

    @treff7es
    treff7es committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    df05369 View commit details
    Browse the repository at this point in the history

Commits on Mar 14, 2023

  1. Merge branch 'master' into redshift-rework 

    # Conflicts:
#	metadata-ingestion/src/datahub/ingestion/source/redshift/usage.py
    @treff7es
    treff7es committed Mar 14, 2023
    Configuration menu
    Copy the full SHA
    b3cd9a6 View commit details
    Browse the repository at this point in the history

  2. Mergin master

    @treff7es
    treff7es committed Mar 14, 2023
    Configuration menu
    Copy the full SHA
    5b304fa View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' into redshift-rework

    @treff7es
    treff7es committed Mar 14, 2023
    Configuration menu
    Copy the full SHA
    281b78d View commit details
    Browse the repository at this point in the history

Commits on Mar 16, 2023

  1. Removing redshift-beta

    @treff7es
    treff7es committed Mar 16, 2023
    Configuration menu
    Copy the full SHA
    ea0a6cf View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into redshift-rework

    @treff7es
    treff7es committed Mar 16, 2023
    Configuration menu
    Copy the full SHA
    d41f776 View commit details
    Browse the repository at this point in the history

  3. Fixing linter error

    @treff7es
    treff7es committed Mar 16, 2023
    Configuration menu
    Copy the full SHA
    82160ed View commit details
    Browse the repository at this point in the history

Commits on Mar 20, 2023

  1. Merge branch 'master' into redshift-rework

    @treff7es
    treff7es committed Mar 20, 2023
    Configuration menu
    Copy the full SHA
    3e7409a View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into redshift-rework

    @jjoyce0510
    jjoyce0510 committed Mar 20, 2023
    Configuration menu
    Copy the full SHA
    5cc4be1 View commit details
    Browse the repository at this point in the history

Commits on Mar 21, 2023

  1. Merge branch 'master' into redshift-rework

    @treff7es
    treff7es committed Mar 21, 2023
    Configuration menu
    Copy the full SHA
    298f0d9 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into redshift-rework

    @treff7es
    treff7es committed Mar 21, 2023
    Configuration menu
    Copy the full SHA
    bce47ac View commit details
    Browse the repository at this point in the history

  3. Addressing pr review comments

    @treff7es
    treff7es committed Mar 21, 2023
    Configuration menu
    Copy the full SHA
    7f67047 View commit details
    Browse the repository at this point in the history

  4. fixing import order

    @treff7es
    treff7es committed Mar 21, 2023
    Configuration menu
    Copy the full SHA
    6c9585c View commit details
    Browse the repository at this point in the history

  5. Merge branch 'master' into redshift-rework

    @treff7es
    treff7es committed Mar 21, 2023
    Configuration menu
    Copy the full SHA
    5d98eb9 View commit details
    Browse the repository at this point in the history

Commits on Mar 23, 2023

  1. Updating dbt golden files

    @treff7es
    treff7es committed Mar 23, 2023
    Configuration menu
    Copy the full SHA
    6c3cf64 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into redshift-rework

    @treff7es
    treff7es committed Mar 23, 2023
    Configuration menu
    Copy the full SHA
    ea78f9f View commit details
    Browse the repository at this point in the history

  3. update to redshift-usage-legacy

    @hsheth2
    hsheth2 committed Mar 23, 2023
    Configuration menu
    Copy the full SHA
    903e0a7 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'master' into redshift-rework

    @hsheth2
    hsheth2 committed Mar 23, 2023
    Configuration menu
    Copy the full SHA
    d0e2d42 View commit details
    Browse the repository at this point in the history

  5. update dbt golden files

    @hsheth2
    hsheth2 committed Mar 23, 2023
    Configuration menu
    Copy the full SHA
    975274b View commit details
    Browse the repository at this point in the history

  6. Fixing usgage-legacy

    @treff7es
    treff7es committed Mar 23, 2023
    Configuration menu
    Copy the full SHA
    7f9bffb View commit details
    Browse the repository at this point in the history

Commits on Mar 24, 2023

  1. Recovering legacy redshift-usage

    @treff7es
    treff7es committed Mar 24, 2023
    Configuration menu
    Copy the full SHA
    aa4c6d4 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into redshift-rework

    @treff7es
    treff7es committed Mar 24, 2023
    Configuration menu
    Copy the full SHA
    3968cf5 View commit details
    Browse the repository at this point in the history

Commits on Mar 27, 2023

  1. fixing lineage query

    @treff7es
    treff7es committed Mar 27, 2023
    Configuration menu
    Copy the full SHA
    af2d718 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into redshift-rework

    @treff7es
    treff7es committed Mar 27, 2023
    Configuration menu
    Copy the full SHA
    1eb15d1 View commit details
    Browse the repository at this point in the history

Commits on Apr 7, 2023

  1. Merge branch 'master' into redshift-rework

    @hsheth2
    hsheth2 committed Apr 7, 2023
    Configuration menu
    Copy the full SHA
    c778093 View commit details
    Browse the repository at this point in the history

Commits on Apr 11, 2023

  1. Merge branch 'master' into redshift-rework

    @treff7es
    treff7es committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    5539139 View commit details
    Browse the repository at this point in the history

  2. Fixing linter error

    @treff7es
    treff7es committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    88bb936 View commit details
    Browse the repository at this point in the history

  3. Fixing linter issues

    @treff7es
    treff7es committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    25fce12 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'master' into redshift-rework

    @treff7es
    treff7es committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    bc9480f View commit details
    Browse the repository at this point in the history

  5. fixing bugbear linter issue

    @treff7es
    treff7es committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    99dbd43 View commit details
    Browse the repository at this point in the history

Commits on Apr 12, 2023

  1. Merge branch 'master' into redshift-rework

    @treff7es
    treff7es committed Apr 12, 2023
    Configuration menu
    Copy the full SHA
    06cdcd4 View commit details
    Browse the repository at this point in the history