Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(ingest): schema-aware SQL parsing for column-level lineage #8334

Merged
merged 39 commits into from
Jul 7, 2023
Merged

feat(ingest): schema-aware SQL parsing for column-level lineage #8334

merged 39 commits into from
Jul 7, 2023

Commits on Jun 5, 2023

  1. start playing with sqlglot

    @hsheth2
    hsheth2 committed Jun 5, 2023
    Configuration menu
    Copy the full SHA
    10c1467 View commit details
    Browse the repository at this point in the history

  2. more stuff for testing

    @hsheth2
    hsheth2 committed Jun 5, 2023
    Configuration menu
    Copy the full SHA
    906c24d View commit details
    Browse the repository at this point in the history

  3. freeze a bunch of stuff

    @hsheth2
    hsheth2 committed Jun 5, 2023
    Configuration menu
    Copy the full SHA
    8dbdaef View commit details
    Browse the repository at this point in the history

Commits on Jun 7, 2023

  1. tweak lineage

    @hsheth2
    hsheth2 committed Jun 7, 2023
    Configuration menu
    Copy the full SHA
    afbc7b6 View commit details
    Browse the repository at this point in the history

Commits on Jun 8, 2023

  1. start hacking schema awareness

    @hsheth2
    hsheth2 committed Jun 8, 2023
    Configuration menu
    Copy the full SHA
    1d34187 View commit details
    Browse the repository at this point in the history

  2. enable cross-run persist in FileBackedDict

    @hsheth2
    hsheth2 committed Jun 8, 2023
    Configuration menu
    Copy the full SHA
    1bb874a View commit details
    Browse the repository at this point in the history

Commits on Jun 9, 2023

  1. start cleaning up

    @hsheth2
    hsheth2 committed Jun 9, 2023
    Configuration menu
    Copy the full SHA
    ff148a9 View commit details
    Browse the repository at this point in the history

  2. fix schema resolution bugs

    @hsheth2
    hsheth2 committed Jun 9, 2023
    Configuration menu
    Copy the full SHA
    44f0adf View commit details
    Browse the repository at this point in the history

  3. handle table normalization

    @hsheth2
    hsheth2 committed Jun 9, 2023
    Configuration menu
    Copy the full SHA
    80002a8 View commit details
    Browse the repository at this point in the history

  4. add todos

    @hsheth2
    hsheth2 committed Jun 9, 2023
    Configuration menu
    Copy the full SHA
    1ca83c0 View commit details
    Browse the repository at this point in the history

Commits on Jun 12, 2023

  1. fix lowercasing test

    @hsheth2
    hsheth2 committed Jun 12, 2023
    Configuration menu
    Copy the full SHA
    1624d06 View commit details
    Browse the repository at this point in the history

  2. switch to pydantic

    @hsheth2
    hsheth2 committed Jun 12, 2023
    Configuration menu
    Copy the full SHA
    27c812f View commit details
    Browse the repository at this point in the history

Commits on Jun 13, 2023

  1. slightly better bq _partitiontime handling

    @hsheth2
    hsheth2 committed Jun 13, 2023
    Configuration menu
    Copy the full SHA
    d4c8017 View commit details
    Browse the repository at this point in the history

  2. add identify call

    @hsheth2
    hsheth2 committed Jun 13, 2023
    Configuration menu
    Copy the full SHA
    0c7eb5c View commit details
    Browse the repository at this point in the history

Commits on Jun 14, 2023

  1. fix for bq/snowflake schema

    @hsheth2
    hsheth2 committed Jun 14, 2023
    Configuration menu
    Copy the full SHA
    09dbc24 View commit details
    Browse the repository at this point in the history

Commits on Jun 15, 2023

  1. query types + bug fixes

    @hsheth2
    hsheth2 committed Jun 15, 2023
    Configuration menu
    Copy the full SHA
    fc28e8b View commit details
    Browse the repository at this point in the history

  2. better table qualification

    @hsheth2
    hsheth2 committed Jun 15, 2023
    Configuration menu
    Copy the full SHA
    b58a06e View commit details
    Browse the repository at this point in the history

  3. better debug info

    @hsheth2
    hsheth2 committed Jun 15, 2023
    Configuration menu
    Copy the full SHA
    fca4718 View commit details
    Browse the repository at this point in the history

  4. resolve table urns

    @hsheth2
    hsheth2 committed Jun 15, 2023
    Configuration menu
    Copy the full SHA
    603fa7f View commit details
    Browse the repository at this point in the history

Commits on Jun 16, 2023

  1. convert remaining urns

    @hsheth2
    hsheth2 committed Jun 16, 2023
    Configuration menu
    Copy the full SHA
    15856bc View commit details
    Browse the repository at this point in the history

  2. mark more things as internal

    @hsheth2
    hsheth2 committed Jun 16, 2023
    Configuration menu
    Copy the full SHA
    b43b575 View commit details
    Browse the repository at this point in the history

  3. tweak logging + bug fixes

    @hsheth2
    hsheth2 committed Jun 16, 2023
    Configuration menu
    Copy the full SHA
    862275b View commit details
    Browse the repository at this point in the history

Commits on Jun 28, 2023

  1. fixup lint

    @hsheth2
    hsheth2 committed Jun 28, 2023
    Configuration menu
    Copy the full SHA
    a2bcfbb View commit details
    Browse the repository at this point in the history

  2. start testing setup

    @hsheth2
    hsheth2 committed Jun 28, 2023
    Configuration menu
    Copy the full SHA
    6aee383 View commit details
    Browse the repository at this point in the history

  3. add some test cases

    @hsheth2
    hsheth2 committed Jun 28, 2023
    Configuration menu
    Copy the full SHA
    b4e3893 View commit details
    Browse the repository at this point in the history

  4. cleanup exceptions

    @hsheth2
    hsheth2 committed Jun 28, 2023
    Configuration menu
    Copy the full SHA
    1c01456 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'master' into sqlglot

    @hsheth2
    hsheth2 committed Jun 28, 2023
    Configuration menu
    Copy the full SHA
    ea97fc7 View commit details
    Browse the repository at this point in the history

Commits on Jun 29, 2023

  1. add extra subqueries test

    @hsheth2
    hsheth2 committed Jun 29, 2023
    Configuration menu
    Copy the full SHA
    05949af View commit details
    Browse the repository at this point in the history

  2. lint

    @hsheth2
    hsheth2 committed Jun 29, 2023
    Configuration menu
    Copy the full SHA
    d1dafaa View commit details
    Browse the repository at this point in the history

  3. add a cli tool + graph method for sql parsing

    @hsheth2
    hsheth2 committed Jun 29, 2023
    Configuration menu
    Copy the full SHA
    e314540 View commit details
    Browse the repository at this point in the history

  4. fix build

    @hsheth2
    hsheth2 committed Jun 29, 2023
    Configuration menu
    Copy the full SHA
    7ff2af8 View commit details
    Browse the repository at this point in the history

  5. add bq table normalization test

    @hsheth2
    hsheth2 committed Jun 29, 2023
    Configuration menu
    Copy the full SHA
    8aa1697 View commit details
    Browse the repository at this point in the history

  6. tweak normalization in snowflake

    @hsheth2
    hsheth2 committed Jun 29, 2023
    Configuration menu
    Copy the full SHA
    2d7264d View commit details
    Browse the repository at this point in the history

  7. tweak comment

    @hsheth2
    hsheth2 committed Jun 29, 2023
    Configuration menu
    Copy the full SHA
    7b105c3 View commit details
    Browse the repository at this point in the history

  8. add todo

    @hsheth2
    hsheth2 committed Jun 29, 2023
    Configuration menu
    Copy the full SHA
    1c1bc30 View commit details
    Browse the repository at this point in the history

  9. add pip extra

    @hsheth2
    hsheth2 committed Jun 29, 2023
    Configuration menu
    Copy the full SHA
    349d481 View commit details
    Browse the repository at this point in the history

Commits on Jul 5, 2023

  1. fix test

    @hsheth2
    hsheth2 committed Jul 5, 2023
    Configuration menu
    Copy the full SHA
    791c203 View commit details
    Browse the repository at this point in the history

Commits on Jul 6, 2023

  1. handle bigquery's monkeypatching

    @hsheth2
    hsheth2 committed Jul 6, 2023
    Configuration menu
    Copy the full SHA
    4cf2f03 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into sqlglot

    @hsheth2
    hsheth2 authored Jul 6, 2023
    Configuration menu
    Copy the full SHA
    1c0f186 View commit details
    Browse the repository at this point in the history