Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add parquet data writer #403

Merged
merged 33 commits into from
Jun 21, 2023
Merged

add parquet data writer #403

merged 33 commits into from
Jun 21, 2023

Commits on Jun 19, 2023

  1. add simple parquet data writer with test

    @sh-rp
    sh-rp committed Jun 19, 2023
    Configuration menu
    Copy the full SHA
    8353095 View commit details
    Browse the repository at this point in the history

  2. more work on parquet writer

    @sh-rp
    sh-rp committed Jun 19, 2023
    Configuration menu
    Copy the full SHA
    7a5c27a View commit details
    Browse the repository at this point in the history

  3. enable parquet for big query

    @sh-rp
    sh-rp committed Jun 19, 2023
    Configuration menu
    Copy the full SHA
    12431c7 View commit details
    Browse the repository at this point in the history

  4. move tests and pyarrow imports

    @sh-rp
    sh-rp committed Jun 19, 2023
    Configuration menu
    Copy the full SHA
    e80cb4d View commit details
    Browse the repository at this point in the history

  5. start rotation tests

    @sh-rp
    sh-rp committed Jun 19, 2023
    Configuration menu
    Copy the full SHA
    654c977 View commit details
    Browse the repository at this point in the history

  6. test file rotation

    @sh-rp
    sh-rp committed Jun 19, 2023
    Configuration menu
    Copy the full SHA
    e1bb088 View commit details
    Browse the repository at this point in the history

  7. fix linting

    @sh-rp
    sh-rp committed Jun 19, 2023
    Configuration menu
    Copy the full SHA
    3d5073c View commit details
    Browse the repository at this point in the history

  8. propagate file loader type to normalize methods

    @sh-rp
    sh-rp committed Jun 19, 2023
    Configuration menu
    Copy the full SHA
    ce8ab9f View commit details
    Browse the repository at this point in the history

  9. bigquery parquet loading

    @sh-rp
    sh-rp committed Jun 19, 2023
    Configuration menu
    Copy the full SHA
    c6dbb8f View commit details
    Browse the repository at this point in the history

  10. fix comments

    @sh-rp
    sh-rp committed Jun 19, 2023
    Configuration menu
    Copy the full SHA
    ff4a8ed View commit details
    Browse the repository at this point in the history

  11. move bigquery default back to jsonl

    @sh-rp
    sh-rp committed Jun 19, 2023
    Configuration menu
    Copy the full SHA
    9a3e07f View commit details
    Browse the repository at this point in the history

  12. fixes jsonl default in filesystem: normalize tests

    @rudolfix @sh-rp
    rudolfix authored and sh-rp committed Jun 19, 2023
    Configuration menu
    Copy the full SHA
    c723469 View commit details
    Browse the repository at this point in the history

  13. fixes the pipeline info cli command output

    @rudolfix @sh-rp
    rudolfix authored and sh-rp committed Jun 19, 2023
    Configuration menu
    Copy the full SHA
    1cccb14 View commit details
    Browse the repository at this point in the history

  14. put sql back into bigquer caps

    @sh-rp
    sh-rp committed Jun 19, 2023
    Configuration menu
    Copy the full SHA
    aa61349 View commit details
    Browse the repository at this point in the history

  15. add biquery parquet result testing

    @sh-rp
    sh-rp committed Jun 19, 2023
    Configuration menu
    Copy the full SHA
    351324b View commit details
    Browse the repository at this point in the history

  16. add config spec

    @sh-rp
    sh-rp committed Jun 19, 2023
    Configuration menu
    Copy the full SHA
    9f03539 View commit details
    Browse the repository at this point in the history

  17. some pr fixes

    @sh-rp
    sh-rp committed Jun 19, 2023
    Configuration menu
    Copy the full SHA
    fcf2820 View commit details
    Browse the repository at this point in the history

  18. make non compression support explicit in parquet writer

    @sh-rp
    sh-rp committed Jun 19, 2023
    Configuration menu
    Copy the full SHA
    ad73d6b View commit details
    Browse the repository at this point in the history

  19. further pr fixes

    @sh-rp
    sh-rp committed Jun 19, 2023
    Configuration menu
    Copy the full SHA
    3ed0e2a View commit details
    Browse the repository at this point in the history

  20. fix tests

    @sh-rp
    sh-rp committed Jun 19, 2023
    Configuration menu
    Copy the full SHA
    8227652 View commit details
    Browse the repository at this point in the history

  21. temporarily remove extra flag from pyarrow in pyproject toml

    @sh-rp
    sh-rp committed Jun 19, 2023
    Configuration menu
    Copy the full SHA
    e735971 View commit details
    Browse the repository at this point in the history

  22. add bigquery extra

    @sh-rp
    sh-rp committed Jun 19, 2023
    Configuration menu
    Copy the full SHA
    ac1338c View commit details
    Browse the repository at this point in the history

  23. some dependency fixes for tests

    @sh-rp
    sh-rp committed Jun 19, 2023
    Configuration menu
    Copy the full SHA
    5cd8f5f View commit details
    Browse the repository at this point in the history

  24. create proper extra

    @sh-rp
    sh-rp committed Jun 19, 2023
    Configuration menu
    Copy the full SHA
    ca8334f View commit details
    Browse the repository at this point in the history

  25. move default destination caps for bigquery back to jsonl

    @sh-rp
    sh-rp committed Jun 19, 2023
    Configuration menu
    Copy the full SHA
    98cf9dd View commit details
    Browse the repository at this point in the history

  26. fix source format detection in bigquery loader

    @sh-rp
    sh-rp committed Jun 19, 2023
    Configuration menu
    Copy the full SHA
    fc3ac52 View commit details
    Browse the repository at this point in the history

Commits on Jun 20, 2023

  1. pr fixes

    @sh-rp
    sh-rp committed Jun 20, 2023
    Configuration menu
    Copy the full SHA
    ed90517 View commit details
    Browse the repository at this point in the history

  2. small pr fix

    @sh-rp
    sh-rp committed Jun 20, 2023
    Configuration menu
    Copy the full SHA
    afe16e5 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'devel' into d#/389-add-parquet-data-writer

    @rudolfix
    rudolfix committed Jun 20, 2023
    Configuration menu
    Copy the full SHA
    c5692c2 View commit details
    Browse the repository at this point in the history

  4. updates poetry.lock

    @rudolfix
    rudolfix committed Jun 20, 2023
    Configuration menu
    Copy the full SHA
    0f62cb3 View commit details
    Browse the repository at this point in the history

  5. makes random dataset name in compression test to run in parallel

    @rudolfix
    rudolfix committed Jun 20, 2023
    Configuration menu
    Copy the full SHA
    fb1b82d View commit details
    Browse the repository at this point in the history

  6. resolves the potgres/redshift/snowflake credentials when parsing nati… 

    …ve value and completed
    @rudolfix
    rudolfix committed Jun 20, 2023
    Configuration menu
    Copy the full SHA
    a84c7ad View commit details
    Browse the repository at this point in the history

  7. adds proper handling of password vs private key fields in snowflake c… 

    …reds + tests
    @rudolfix
    rudolfix committed Jun 20, 2023
    Configuration menu
    Copy the full SHA
    2732ca3 View commit details
    Browse the repository at this point in the history