Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DM-25806: Support parallel ap_verify ingestion in Gen 3 #102

Merged
merged 7 commits into from
Sep 21, 2020
Merged

DM-25806: Support parallel ap_verify ingestion in Gen 3 #102

merged 7 commits into from
Sep 21, 2020

Commits on Aug 27, 2020

  1. Define __eq__ and __repr__ for Dataset.

    @kfindeisen
    kfindeisen committed Aug 27, 2020
    Configuration menu
    Copy the full SHA
    d9ada0f View commit details
    Browse the repository at this point in the history

  2. Define __eq__ and __repr__ for Workspace.

    @kfindeisen
    kfindeisen committed Aug 27, 2020
    Configuration menu
    Copy the full SHA
    3ac583c View commit details
    Browse the repository at this point in the history

  3. Add new sub-parser for ApPipe and IngestDatasets. 

    The parser will hold common arguments that aren't related to I/O.
    @kfindeisen
    kfindeisen committed Aug 27, 2020
    Configuration menu
    Copy the full SHA
    3b3007d View commit details
    Browse the repository at this point in the history

  4. Move --processes to common parser. 

    To keep parser use in same module as parser definitions, processes
are now passed by main program to pipeline driver.
    @kfindeisen
    kfindeisen committed Aug 27, 2020
    Configuration menu
    Copy the full SHA
    253066d View commit details
    Browse the repository at this point in the history

Commits on Sep 21, 2020

  1. Fix pickling for Gen3DatasetIngestTask. 

    Pickle support is needed to be able to run ingestion in parallel.
    @kfindeisen
    kfindeisen committed Sep 21, 2020
    Configuration menu
    Copy the full SHA
    5a7e776 View commit details
    Browse the repository at this point in the history

  2. Implement -j flag for Gen 3 ingestion. 

    This passes the argument to all ingestion tasks that support the
processes keyword.
    @kfindeisen
    kfindeisen committed Sep 21, 2020
    Configuration menu
    Copy the full SHA
    5859710 View commit details
    Browse the repository at this point in the history

  3. Fix path inconsistencies on Jenkins.

    @kfindeisen
    kfindeisen committed Sep 21, 2020
    Configuration menu
    Copy the full SHA
    7638c16 View commit details
    Browse the repository at this point in the history