Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DRILL-4530: Optimize partition pruning with metadata caching for the … #519

Closed
wants to merge 7 commits into from
Closed

DRILL-4530: Optimize partition pruning with metadata caching for the … #519

wants to merge 7 commits into from
+624 −104

Commits on Jul 16, 2016

  1. DRILL-4530: Optimize partition pruning with metadata caching for the … 

    …single partition case.

 - Enhance PruneScanRule to detect single partitions based on referenced dirs in the filter.
 - Keep a new status of EXPANDED_PARTIAL for FileSelection.
 - Create separate .directories metadata file to prune directories first before files.
 - Introduce cacheFileRoot attribute to keep track of the parent directory of the cache file after partition pruning.
    Aman Sinha committed Jul 16, 2016
    Configuration menu
    Copy the full SHA
    d7939b7 View commit details
    Browse the repository at this point in the history

  2. Check if prefix components are non-null the very first time single pa… 

    …rtition info is initialized.
    Aman Sinha committed Jul 16, 2016
    Configuration menu
    Copy the full SHA
    e80c65b View commit details
    Browse the repository at this point in the history

  3. Add separate interface method to create scan using a cacheFileRoot.

    Aman Sinha committed Jul 16, 2016
    Configuration menu
    Copy the full SHA
    4f77799 View commit details
    Browse the repository at this point in the history

  4. Create filenames list with unique names using fileSet if available. A… 

    …dd several unit tests.
    Aman Sinha committed Jul 16, 2016
    Configuration menu
    Copy the full SHA
    2bc7fbc View commit details
    Browse the repository at this point in the history

  5. Populate only fileSet when expanding using the metadata cache.

    Aman Sinha committed Jul 16, 2016
    Configuration menu
    Copy the full SHA
    ef37b77 View commit details
    Browse the repository at this point in the history

  6. Remove cacheFileRoot parameter from FileGroupScan's clone() method an… 

    …d instead leverage it from FileSelection.
    Aman Sinha committed Jul 16, 2016
    Configuration menu
    Copy the full SHA
    7ae8f7a View commit details
    Browse the repository at this point in the history

  7. Keep track of whether all partitions were previously pruned and proce… 

    …ss this state where needed.
    Aman Sinha committed Jul 16, 2016
    Configuration menu
    Copy the full SHA
    ce509af View commit details
    Browse the repository at this point in the history