Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Performance Optimizations for Non-Bulk Indexing #26

Merged
merged 11 commits into from
Mar 15, 2021
Merged

Performance Optimizations for Non-Bulk Indexing #26

merged 11 commits into from
Mar 15, 2021

Commits on Mar 10, 2021

  1. by default, don't patch (refactor --no-patch -> --patch)

    @ric-evans
    ric-evans committed Mar 10, 2021
    Configuration menu
    Copy the full SHA
    2a7c3ed View commit details
    Browse the repository at this point in the history

  2. check if file is already in the FC before collecting its metadata

    @ric-evans
    ric-evans committed Mar 10, 2021
    Configuration menu
    Copy the full SHA
    acff252 View commit details
    Browse the repository at this point in the history

  3. make single-processed/multi-processed fork at top of pipeline

    @ric-evans
    ric-evans committed Mar 10, 2021
    Configuration menu
    Copy the full SHA
    d746936 View commit details
    Browse the repository at this point in the history

  4. misc

    @ric-evans
    ric-evans committed Mar 10, 2021
    Configuration menu
    Copy the full SHA
    0b1b5cb View commit details
    Browse the repository at this point in the history

  5. refactor with descriptive names and defined responsibilities

    @ric-evans
    ric-evans committed Mar 10, 2021
    Configuration menu
    Copy the full SHA
    1db1778 View commit details
    Browse the repository at this point in the history

  6. add -n/--non-recursive for not recursively indexing

    @ric-evans
    ric-evans committed Mar 10, 2021
    Configuration menu
    Copy the full SHA
    2e6815f View commit details
    Browse the repository at this point in the history

  7. get absolute paths for each blacklisted filepath

    @ric-evans
    ric-evans committed Mar 10, 2021
    Configuration menu
    Copy the full SHA
    75aa101 View commit details
    Browse the repository at this point in the history

  8. indexer_make_dag.py: remove any 'patch' options (there's no need)

    @ric-evans
    ric-evans committed Mar 10, 2021
    Configuration menu
    Copy the full SHA
    9f373b2 View commit details
    Browse the repository at this point in the history

Commits on Mar 12, 2021

  1. add --blacklist option for shorter blacklists

    @ric-evans
    ric-evans committed Mar 12, 2021
    Configuration menu
    Copy the full SHA
    864a95c View commit details
    Browse the repository at this point in the history

  2. sort & standardize git-pips

    @ric-evans
    ric-evans committed Mar 12, 2021
    Configuration menu
    Copy the full SHA
    9eab40d View commit details
    Browse the repository at this point in the history

Commits on Mar 15, 2021

  1. install iceprod@v2.5.7

    @ric-evans
    ric-evans committed Mar 15, 2021
    Configuration menu
    Copy the full SHA
    135f7f4 View commit details
    Browse the repository at this point in the history