Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor read_metadata in fastparquet engine #8092

Merged
merged 60 commits into from Oct 19, 2021
Merged

Refactor read_metadata in fastparquet engine #8092

merged 60 commits into from Oct 19, 2021

Commits on Aug 18, 2021

  1. initial pyarrow-dataset experimental refactor

    @rjzamora
    rjzamora committed Aug 18, 2021
    Copy the full SHA
    8b23c6e View commit details
    Browse the repository at this point in the history

Commits on Aug 19, 2021

  1. inital refactor of both arrow engines

    @rjzamora
    rjzamora committed Aug 19, 2021
    Copy the full SHA
    a0e0fec View commit details
    Browse the repository at this point in the history

  2. move private methods to correct location

    @rjzamora
    rjzamora committed Aug 19, 2021
    Copy the full SHA
    697e4f2 View commit details
    Browse the repository at this point in the history

  3. save state

    @rjzamora
    rjzamora committed Aug 19, 2021
    Copy the full SHA
    08ea5ad View commit details
    Browse the repository at this point in the history

Commits on Aug 20, 2021

  1. ready to convert _metadata-free case to a dask graph

    @rjzamora
    rjzamora committed Aug 20, 2021
    Copy the full SHA
    87845b9 View commit details
    Browse the repository at this point in the history

  2. parallel collect parts

    @rjzamora
    rjzamora committed Aug 20, 2021
    Copy the full SHA
    a44bc39 View commit details
    Browse the repository at this point in the history

  3. remove stale/unused code and move legacy-specific code into legacy class

    @rjzamora
    rjzamora committed Aug 20, 2021
    Copy the full SHA
    5086583 View commit details
    Browse the repository at this point in the history

  4. Merge remote-tracking branch 'upstream/main' into read-metadata-refactor

    @rjzamora
    rjzamora committed Aug 20, 2021
    Copy the full SHA
    a9c36ba View commit details
    Browse the repository at this point in the history

  5. ensure correct hive-partition order

    @rjzamora
    rjzamora committed Aug 20, 2021
    Copy the full SHA
    b978de5 View commit details
    Browse the repository at this point in the history

  6. add files_per_metadata_task to docstring

    @rjzamora
    rjzamora committed Aug 20, 2021
    Copy the full SHA
    6c819ea View commit details
    Browse the repository at this point in the history

  7. change default files_per_metadata_task to 32

    @rjzamora
    rjzamora committed Aug 20, 2021
    Copy the full SHA
    54db532 View commit details
    Browse the repository at this point in the history

  8. adding basic test coverage

    @rjzamora
    rjzamora committed Aug 20, 2021
    Copy the full SHA
    4b0b4d3 View commit details
    Browse the repository at this point in the history

Commits on Aug 24, 2021

  1. Merge branch 'main' of https://github.com/dask/dask into read-metadat… 

    …a-refactor
    @jrbourbeau
    jrbourbeau committed Aug 24, 2021
    Copy the full SHA
    c51e970 View commit details
    Browse the repository at this point in the history

Commits on Aug 25, 2021

  1. minor fix

    @rjzamora
    rjzamora committed Aug 25, 2021
    Copy the full SHA
    52f6c30 View commit details
    Browse the repository at this point in the history

  2. change name to metadata_task_size

    @rjzamora
    rjzamora committed Aug 25, 2021
    Copy the full SHA
    1110330 View commit details
    Browse the repository at this point in the history

  3. port _determine_pf_parts to _collect_dataset_info

    @rjzamora
    rjzamora committed Aug 25, 2021
    Copy the full SHA
    e34c48b View commit details
    Browse the repository at this point in the history

  4. ported _generate_dd_meta to _create_dd_meta

    @rjzamora
    rjzamora committed Aug 25, 2021
    Copy the full SHA
    0cd35b4 View commit details
    Browse the repository at this point in the history

  5. Move _construct_parts into _construct_collection_plan

    @rjzamora
    rjzamora committed Aug 25, 2021
    Copy the full SHA
    1354cf7 View commit details
    Browse the repository at this point in the history

  6. remove unused (stale) code

    @rjzamora
    rjzamora committed Aug 25, 2021
    Copy the full SHA
    b4c8a45 View commit details
    Browse the repository at this point in the history

  7. minor paths tweak

    @rjzamora
    rjzamora committed Aug 25, 2021
    Copy the full SHA
    d9e48c5 View commit details
    Browse the repository at this point in the history

  8. resolve bug for partitioned data with gather_statistics=False

    @rjzamora
    rjzamora committed Aug 25, 2021
    Copy the full SHA
    bb57a2a View commit details
    Browse the repository at this point in the history

  9. improve pyarrow 'fast-path'

    @rjzamora
    rjzamora committed Aug 25, 2021
    Copy the full SHA
    322af53 View commit details
    Browse the repository at this point in the history

  10. enable faster partitioned reads with pyarrow

    @rjzamora
    rjzamora committed Aug 25, 2021
    Copy the full SHA
    7e8b8bd View commit details
    Browse the repository at this point in the history

Commits on Aug 26, 2021

  1. fix pyarrow partitioning behaviour

    @rjzamora
    rjzamora committed Aug 26, 2021
    Copy the full SHA
    9d88830 View commit details
    Browse the repository at this point in the history

  2. basic parallel fastparquet metadata support

    @rjzamora
    rjzamora committed Aug 26, 2021
    Copy the full SHA
    18f316e View commit details
    Browse the repository at this point in the history

  3. remove stale code

    @rjzamora
    rjzamora committed Aug 26, 2021
    Copy the full SHA
    b4ca448 View commit details
    Browse the repository at this point in the history

  4. remove sync compute

    @rjzamora
    rjzamora committed Aug 26, 2021
    Copy the full SHA
    ef59025 View commit details
    Browse the repository at this point in the history

Commits on Aug 31, 2021

  1. update arrow changes

    @rjzamora
    rjzamora committed Aug 31, 2021
    Copy the full SHA
    a468872 View commit details
    Browse the repository at this point in the history

Commits on Sep 8, 2021

  1. add arrow_filesystem option for 'pyarrow-dataset' engine

    @rjzamora
    rjzamora committed Sep 8, 2021
    Copy the full SHA
    ef3bfe9 View commit details
    Browse the repository at this point in the history

Commits on Sep 14, 2021

  1. avoid default caching in fs.open

    @rjzamora
    rjzamora committed Sep 14, 2021
    Copy the full SHA
    355739e View commit details
    Browse the repository at this point in the history

  2. roll back arrow_filesystem= option - non-default caching provides goo… 

    …d performance
    @rjzamora
    rjzamora committed Sep 14, 2021
    Copy the full SHA
    884cd38 View commit details
    Browse the repository at this point in the history

  3. remove arrow_filesystsem test

    @rjzamora
    rjzamora committed Sep 14, 2021
    Copy the full SHA
    abc561c View commit details
    Browse the repository at this point in the history

  4. address missing pandas metadata issue

    @rjzamora
    rjzamora committed Sep 14, 2021
    Copy the full SHA
    ae3fc23 View commit details
    Browse the repository at this point in the history

Commits on Sep 27, 2021

  1. Merge remote-tracking branch 'upstream/main' into read-metadata-refactor

    @rjzamora
    rjzamora committed Sep 27, 2021
    Copy the full SHA
    7a9086b View commit details
    Browse the repository at this point in the history

  2. Merge branch 'read-metadata-refactor' into read-metadata-refactor-fas… 

    …tparquet
    @rjzamora
    rjzamora committed Sep 27, 2021
    Copy the full SHA
    44bf0f5 View commit details
    Browse the repository at this point in the history

  3. remove stale comment

    @rjzamora
    rjzamora committed Sep 27, 2021
    Copy the full SHA
    5fe0fe0 View commit details
    Browse the repository at this point in the history

Commits on Sep 28, 2021

  1. re-order check for 'simple' partitioning plan

    @rjzamora
    rjzamora committed Sep 28, 2021
    Copy the full SHA
    6f7fab5 View commit details
    Browse the repository at this point in the history

  2. update arrow.py

    @rjzamora
    rjzamora committed Sep 28, 2021
    Copy the full SHA
    ff82935 View commit details
    Browse the repository at this point in the history

Commits on Sep 29, 2021

  1. code review part 1

    @rjzamora
    rjzamora committed Sep 29, 2021
    Copy the full SHA
    9b940f2 View commit details
    Browse the repository at this point in the history

  2. roll back partition_method name change and fix metadata_task_size test

    @rjzamora
    rjzamora committed Sep 29, 2021
    Copy the full SHA
    a18223c View commit details
    Browse the repository at this point in the history

  3. avoid regex

    @rjzamora
    rjzamora committed Sep 29, 2021
    Copy the full SHA
    b2474d2 View commit details
    Browse the repository at this point in the history

  4. move metadata_task_size to config

    @rjzamora
    rjzamora committed Sep 29, 2021
    Copy the full SHA
    05b544c View commit details
    Browse the repository at this point in the history

  5. docstring fix

    @rjzamora
    rjzamora committed Sep 29, 2021
    Copy the full SHA
    68f6490 View commit details
    Browse the repository at this point in the history

  6. update dask/dask/dask-schema.yaml

    @rjzamora
    rjzamora committed Sep 29, 2021
    Copy the full SHA
    ec8139b View commit details
    Browse the repository at this point in the history

  7. Merge branch 'read-metadata-refactor' into read-metadata-refactor-fas… 

    …tparquet
    @rjzamora
    rjzamora committed Sep 29, 2021
    Copy the full SHA
    b39588c View commit details
    Browse the repository at this point in the history

  8. improve defaults

    @rjzamora
    rjzamora committed Sep 29, 2021
    Copy the full SHA
    284736c View commit details
    Browse the repository at this point in the history

  9. Merge branch 'read-metadata-refactor' into read-metadata-refactor-fas… 

    …tparquet
    @rjzamora
    rjzamora committed Sep 29, 2021
    Copy the full SHA
    6cab46c View commit details
    Browse the repository at this point in the history

  10. bugfix in _set_metadata_task_size

    @rjzamora
    rjzamora committed Sep 29, 2021
    Copy the full SHA
    5ff9d14 View commit details
    Browse the repository at this point in the history

  11. Merge branch 'read-metadata-refactor' into read-metadata-refactor-fas… 

    …tparquet
    @rjzamora
    rjzamora committed Sep 29, 2021
    Copy the full SHA
    c79906c View commit details
    Browse the repository at this point in the history

  12. avoid parallel metadata collection when the file count is less than m… 

    …etadata_task_size
    @rjzamora
    rjzamora committed Sep 29, 2021
    Copy the full SHA
    6ea87b3 View commit details
    Browse the repository at this point in the history

  13. test_metadata_task_size fix

    @rjzamora
    rjzamora committed Sep 29, 2021
    Copy the full SHA
    9a444fe View commit details
    Browse the repository at this point in the history

Commits on Oct 1, 2021

  1. Merge remote-tracking branch 'upstream/main' into read-metadata-refac… 

    …tor-fastparquet
    @rjzamora
    rjzamora committed Oct 1, 2021
    Copy the full SHA
    4587494 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'read-metadata-refactor-fastparquet' of https://github.c… 

    …om/rjzamora/dask into read-metadata-refactor-fastparquet
    @rjzamora
    rjzamora committed Oct 1, 2021
    Copy the full SHA
    2231875 View commit details
    Browse the repository at this point in the history

Commits on Oct 4, 2021

  1. address 8201

    @rjzamora
    rjzamora committed Oct 4, 2021
    Copy the full SHA
    47b07b9 View commit details
    Browse the repository at this point in the history

  2. remove debug compute call in test

    @rjzamora
    rjzamora committed Oct 4, 2021
    Copy the full SHA
    c7401de View commit details
    Browse the repository at this point in the history

  3. roll back base_path name change to reduce diff

    @rjzamora
    rjzamora committed Oct 4, 2021
    Copy the full SHA
    6deed66 View commit details
    Browse the repository at this point in the history

Commits on Oct 14, 2021

  1. fix undefined piece mistake

    @rjzamora
    rjzamora committed Oct 14, 2021
    Copy the full SHA
    859c107 View commit details
    Browse the repository at this point in the history

  2. remove _common_metadata_exists

    @rjzamora
    rjzamora committed Oct 14, 2021
    Copy the full SHA
    37d59e5 View commit details
    Browse the repository at this point in the history

  3. update to OSError

    @rjzamora
    rjzamora committed Oct 14, 2021
    Copy the full SHA
    d09c76c View commit details
    Browse the repository at this point in the history

  4. Merge remote-tracking branch 'upstream/main' into read-metadata-refac… 

    …tor-fastparquet
    @rjzamora
    rjzamora committed Oct 14, 2021
    Copy the full SHA
    b9edc41 View commit details
    Browse the repository at this point in the history