Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[MAINTENANCE] Splitter cleanup and enhancements #4984

Merged
merged 48 commits into from Apr 29, 2022
Merged

[MAINTENANCE] Splitter cleanup and enhancements #4984

merged 48 commits into from Apr 29, 2022

Commits on Apr 25, 2022

  1. WIP starting to pull together tests

    @anthonyburdi
    anthonyburdi committed Apr 25, 2022
    Configuration menu
    Copy the full SHA
    d66f95c View commit details
    Browse the repository at this point in the history

  2. Merge branch 'develop' into FEATURE/GREAT-727/GREAT-733/splitting_dat… 

    …a_assets_into_batches_using_datetime_columns_in_spark
    @anthonyburdi
    anthonyburdi committed Apr 25, 2022
    Configuration menu
    Copy the full SHA
    c12676d View commit details
    Browse the repository at this point in the history

Commits on Apr 26, 2022

  1. Remove integration test case that is handled in unit tests.

    @anthonyburdi
    anthonyburdi committed Apr 26, 2022
    Configuration menu
    Copy the full SHA
    5bedcec View commit details
    Browse the repository at this point in the history

  2. Combine removed test into existing test case.

    @anthonyburdi
    anthonyburdi committed Apr 26, 2022
    Configuration menu
    Copy the full SHA
    36a724b View commit details
    Browse the repository at this point in the history

  3. Partial stub of end to end test.

    @anthonyburdi
    anthonyburdi committed Apr 26, 2022
    Configuration menu
    Copy the full SHA
    0e43d82 View commit details
    Browse the repository at this point in the history

  4. First pass spark split_on_date_parts, new methods to be DRY between s… 

    …park + sql.
    @anthonyburdi
    anthonyburdi committed Apr 26, 2022
    Configuration menu
    Copy the full SHA
    2d5de06 View commit details
    Browse the repository at this point in the history

  5. Use SparkDataSplitter in SparkDFExecutionEngine, other cleanup.

    @anthonyburdi
    anthonyburdi committed Apr 26, 2022
    Configuration menu
    Copy the full SHA
    3d128d3 View commit details
    Browse the repository at this point in the history

  6. WIP Spark specific integration tests

    @anthonyburdi
    anthonyburdi committed Apr 26, 2022
    Configuration menu
    Copy the full SHA
    02bcd2c View commit details
    Browse the repository at this point in the history

Commits on Apr 27, 2022

  1. Move all splitting methods to SparkDataSplitter

    @anthonyburdi
    anthonyburdi committed Apr 27, 2022
    Configuration menu
    Copy the full SHA
    d31002c View commit details
    Browse the repository at this point in the history

  2. Move shared test cases to separate file, move splitter test to separa… 

    …te directory
    @anthonyburdi
    anthonyburdi committed Apr 27, 2022
    Configuration menu
    Copy the full SHA
    849a8e6 View commit details
    Browse the repository at this point in the history

  3. Code consolidation into parameterized test since it will be reused fo… 

    …r spark version.
    @anthonyburdi
    anthonyburdi committed Apr 27, 2022
    Configuration menu
    Copy the full SHA
    7054d04 View commit details
    Browse the repository at this point in the history

  4. Adding unit tests for spark splitters

    @anthonyburdi
    anthonyburdi committed Apr 27, 2022
    Configuration menu
    Copy the full SHA
    1cbd142 View commit details
    Browse the repository at this point in the history

  5. Adding unit tests for spark splitters, single and multiple date parts… 

    … tests
    @anthonyburdi
    anthonyburdi committed Apr 27, 2022
    Configuration menu
    Copy the full SHA
    7419a09 View commit details
    Browse the repository at this point in the history

  6. Unit tests for unsupported batch identifiers in splitter_kwargs (not … 

    …DatePart)
    @anthonyburdi
    anthonyburdi committed Apr 27, 2022
    Configuration menu
    Copy the full SHA
    abeddc7 View commit details
    Browse the repository at this point in the history

  7. Use pre-built fixture to speed up creation of 10 trips from each mont… 

    …h sample
    @anthonyburdi
    anthonyburdi committed Apr 27, 2022
    Configuration menu
    Copy the full SHA
    51891a0 View commit details
    Browse the repository at this point in the history

  8. Add missing docstring.

    @anthonyburdi
    anthonyburdi committed Apr 27, 2022
    Configuration menu
    Copy the full SHA
    8c977ad View commit details
    Browse the repository at this point in the history

  9. Cleanup and add test cases for SparkDataSplitter.date_part access

    @anthonyburdi
    anthonyburdi committed Apr 27, 2022
    Configuration menu
    Copy the full SHA
    c2044fc View commit details
    Browse the repository at this point in the history

  10. Remove spark integration test from test_script_runner.py

    @anthonyburdi
    anthonyburdi committed Apr 27, 2022
    Configuration menu
    Copy the full SHA
    6c31869 View commit details
    Browse the repository at this point in the history

  11. Merge branch 'develop' into FEATURE/GREAT-727/GREAT-733/splitting_dat… 

    …a_assets_into_batches_using_datetime_columns_in_spark
    @anthonyburdi
    anthonyburdi committed Apr 27, 2022
    Configuration menu
    Copy the full SHA
    0d47bf0 View commit details
    Browse the repository at this point in the history

  12. Merge branch 'develop' into FEATURE/GREAT-727/GREAT-733/splitting_dat… 

    …a_assets_into_batches_using_datetime_columns_in_spark
    @anthonyburdi
    anthonyburdi committed Apr 27, 2022
    Configuration menu
    Copy the full SHA
    2b7d182 View commit details
    Browse the repository at this point in the history

  13. Clarify comment from PR review

    @anthonyburdi
    anthonyburdi committed Apr 27, 2022
    Configuration menu
    Copy the full SHA
    d1253fa View commit details
    Browse the repository at this point in the history

  14. Fix items from PR Review

    @anthonyburdi
    anthonyburdi committed Apr 27, 2022
    Configuration menu
    Copy the full SHA
    02785c9 View commit details
    Browse the repository at this point in the history

  15. Add annotations to new dunder methods

    @anthonyburdi
    anthonyburdi committed Apr 27, 2022
    Configuration menu
    Copy the full SHA
    e1a7c15 View commit details
    Browse the repository at this point in the history

  16. Merge branch 'develop' into FEATURE/GREAT-727/GREAT-733/splitting_dat… 

    …a_assets_into_batches_using_datetime_columns_in_spark
    @anthonyburdi
    anthonyburdi committed Apr 27, 2022
    Configuration menu
    Copy the full SHA
    7713f04 View commit details
    Browse the repository at this point in the history

  17. Add docstrings to moved methods.

    @anthonyburdi
    anthonyburdi committed Apr 27, 2022
    Configuration menu
    Copy the full SHA
    457a547 View commit details
    Browse the repository at this point in the history

  18. Remove index from fixture

    @anthonyburdi
    anthonyburdi committed Apr 27, 2022
    Configuration menu
    Copy the full SHA
    b3d2645 View commit details
    Browse the repository at this point in the history

  19. Merge branch 'develop' into FEATURE/GREAT-727/GREAT-733/splitting_dat… 

    …a_assets_into_batches_using_datetime_columns_in_spark
    @anthonyburdi
    anthonyburdi committed Apr 27, 2022
    Configuration menu
    Copy the full SHA
    0441538 View commit details
    Browse the repository at this point in the history

  20. Merge branch 'develop' into FEATURE/GREAT-727/GREAT-733/splitting_dat… 

    …a_assets_into_batches_using_datetime_columns_in_spark
    @anthonyburdi
    anthonyburdi committed Apr 27, 2022
    Configuration menu
    Copy the full SHA
    3722ca2 View commit details
    Browse the repository at this point in the history

  21. Typo in docstring

    @anthonyburdi
    anthonyburdi committed Apr 27, 2022
    Configuration menu
    Copy the full SHA
    8be75e5 View commit details
    Browse the repository at this point in the history

  22. Use imported DataFrame in test

    @anthonyburdi
    anthonyburdi committed Apr 27, 2022
    Configuration menu
    Copy the full SHA
    526ff57 View commit details
    Browse the repository at this point in the history

Commits on Apr 28, 2022

  1. Merge branch 'develop' into FEATURE/GREAT-727/GREAT-733/splitting_dat… 

    …a_assets_into_batches_using_datetime_columns_in_spark
    @anthonyburdi
    anthonyburdi committed Apr 28, 2022
    Configuration menu
    Copy the full SHA
    eb50727 View commit details
    Browse the repository at this point in the history

  2. Initial implementation and some tests.

    @anthonyburdi
    anthonyburdi committed Apr 28, 2022
    Configuration menu
    Copy the full SHA
    bd118c4 View commit details
    Browse the repository at this point in the history

  3. Unit and integration tests.

    @anthonyburdi
    anthonyburdi committed Apr 28, 2022
    Configuration menu
    Copy the full SHA
    03545c6 View commit details
    Browse the repository at this point in the history

  4. Cleanup

    @anthonyburdi
    anthonyburdi committed Apr 28, 2022
    Configuration menu
    Copy the full SHA
    dd7d081 View commit details
    Browse the repository at this point in the history

  5. Update docs

    @anthonyburdi
    anthonyburdi committed Apr 28, 2022
    Configuration menu
    Copy the full SHA
    3e0b899 View commit details
    Browse the repository at this point in the history

  6. Merge branch 'develop' into FEATURE/GREAT-727/GREAT-734/splitting_dat… 

    …a_assets_into_batches_using_datetime_columns_in_pandas
    @anthonyburdi
    anthonyburdi committed Apr 28, 2022
    Configuration menu
    Copy the full SHA
    8c06815 View commit details
    Browse the repository at this point in the history

  7. Update docstrings

    @anthonyburdi
    anthonyburdi committed Apr 28, 2022
    Configuration menu
    Copy the full SHA
    ea44295 View commit details
    Browse the repository at this point in the history

  8. Merge branch 'develop' into FEATURE/GREAT-727/GREAT-734/splitting_dat… 

    …a_assets_into_batches_using_datetime_columns_in_pandas
    @anthonyburdi
    anthonyburdi committed Apr 28, 2022
    Configuration menu
    Copy the full SHA
    5a7e9ba View commit details
    Browse the repository at this point in the history

  9. Add type hints

    @anthonyburdi
    anthonyburdi committed Apr 28, 2022
    Configuration menu
    Copy the full SHA
    d3f6a91 View commit details
    Browse the repository at this point in the history

  10. Use 'multi' method in to_sql for redshift

    @anthonyburdi
    anthonyburdi committed Apr 28, 2022
    Configuration menu
    Copy the full SHA
    43d6afe View commit details
    Browse the repository at this point in the history

  11. Move sqlalchemy_data_splitter.py into execution_engine/split_and_samp… 

    …le directory alongside spark and pandas versions
    @anthonyburdi
    anthonyburdi committed Apr 28, 2022
    Configuration menu
    Copy the full SHA
    c0e7a47 View commit details
    Browse the repository at this point in the history

  12. Remove underscore prefix from examples

    @anthonyburdi
    anthonyburdi committed Apr 28, 2022
    Configuration menu
    Copy the full SHA
    1dbaff7 View commit details
    Browse the repository at this point in the history

  13. Merge branch 'develop' into MAINTENANCE/GREAT-727/none/cleanup_and_en… 

    …hancements
    @anthonyburdi
    anthonyburdi committed Apr 28, 2022
    Configuration menu
    Copy the full SHA
    7db6fd6 View commit details
    Browse the repository at this point in the history

Commits on Apr 29, 2022

  1. Possible temporary fix for failing integration test.

    @anthonyburdi
    anthonyburdi committed Apr 29, 2022
    Configuration menu
    Copy the full SHA
    ae98465 View commit details
    Browse the repository at this point in the history

  2. Possible permanent fix for failing integration test. Will need follow… 

    …up commit to clean up.
    @anthonyburdi
    anthonyburdi committed Apr 29, 2022
    Configuration menu
    Copy the full SHA
    73388a4 View commit details
    Browse the repository at this point in the history

  3. Clean up interface by adding / removing relevant methods and moving m… 

    …ore split logic to SqlAlchemyDataSplitter.
    @anthonyburdi
    anthonyburdi committed Apr 29, 2022
    Configuration menu
    Copy the full SHA
    bd18cb6 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'develop' into MAINTENANCE/GREAT-727/none/cleanup_and_en… 

    …hancements
    @anthonyburdi
    anthonyburdi committed Apr 29, 2022
    Configuration menu
    Copy the full SHA
    a6c8b55 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'develop' into MAINTENANCE/GREAT-727/none/cleanup_and_en… 

    …hancements
    @anthonyburdi
    anthonyburdi committed Apr 29, 2022
    Configuration menu
    Copy the full SHA
    fac48b9 View commit details
    Browse the repository at this point in the history