Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support Direct Recursive Hive File Listings #12443

Merged
merged 5 commits into from Jun 23, 2022
Merged

Support Direct Recursive Hive File Listings #12443

merged 5 commits into from Jun 23, 2022

Commits on Jun 17, 2022

  1. Support direct recursive directory file listing 

    Adds support for directly listing all recursive files in directory
listing and associated caches.
    @pettyjamesm
    pettyjamesm committed Jun 17, 2022
    Copy the full SHA
    2ab5c24 View commit details
    Browse the repository at this point in the history

  2. Use recursive listing in HiveFileIterator

    @pettyjamesm
    pettyjamesm committed Jun 17, 2022
    Copy the full SHA
    69963ce View commit details
    Browse the repository at this point in the history

  3. Remove unnecessary FileSystem.exists check 

    Removes a pre-emptive FileSystem#exists(Path) check before attempting
to list path contents when absent partition errors are configured to
be ignored. Instead, ignoring absent partitions can be done as part
of a check only in the case where partition listing actually fails.

For file systems like S3, checking the existence of a "directory"
already incurs an S3 listing call, which are relatively expensive in
terms of API rate limits and latency.
    @pettyjamesm
    pettyjamesm committed Jun 17, 2022
    Copy the full SHA
    75e3b36 View commit details
    Browse the repository at this point in the history

  4. Add Hive FileSystem listing test

    @pettyjamesm
    pettyjamesm committed Jun 17, 2022
    Copy the full SHA
    d0e4ffa View commit details
    Browse the repository at this point in the history

  5. Add test for recursive caching directory listings

    @pettyjamesm
    pettyjamesm committed Jun 17, 2022
    Copy the full SHA
    9ac2278 View commit details
    Browse the repository at this point in the history