Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DM-13163: Refactor ap_pipe to use CmdLineTask primitives #22

Merged
merged 6 commits into from
Mar 22, 2018

Commits on Mar 12, 2018

  1. Configuration menu
    Copy the full SHA
    8663e1e View commit details
    Browse the repository at this point in the history
  2. Move doIngestTemplates to ap_verify.

    This hack has been replaced with a multi-input Butler in ap_pipe.
    ap_verify still needs it in order to simulate operations repositories.
    kfindeisen committed Mar 12, 2018
    Configuration menu
    Copy the full SHA
    178d0be View commit details
    Browse the repository at this point in the history
  3. Add butlers to Workspace.

    kfindeisen committed Mar 12, 2018
    Configuration menu
    Copy the full SHA
    574b377 View commit details
    Browse the repository at this point in the history
  4. Move dataId parser from ap_verify.

    These will likely be removed in the future, depending on how much
    interaction is desired between ap_verify and the details of ap_pipe.
    Unfortunately, there is no clean way to parse a dataId string unless
    in a command-line task or calling one.
    kfindeisen committed Mar 12, 2018
    Configuration menu
    Copy the full SHA
    e454d60 View commit details
    Browse the repository at this point in the history

Commits on Mar 22, 2018

  1. Call new ap_pipe API.

    These are the minimal changes needed to make ap_verify work with
    ap_pipe again. Optimization of pipeline_driver will happen on a
    different ticket.
    kfindeisen committed Mar 22, 2018
    Configuration menu
    Copy the full SHA
    58e041f View commit details
    Browse the repository at this point in the history
  2. Modify metrics and error handling.

    The new code imports metrics on exit or failure instead of after each
    pipeline step, allowing us to, for the moment, query the task object
    for metadata. This may need to be reworked depending on how parallel
    processing is handled.
    
    This commit makes one behavior change, in that it now tries to extract
    metrics from the failed pipeline step. If this process raises an
    exception, it will suppress any exception raised by the pipeline. This
    is an acceptable cost for the time being.
    kfindeisen committed Mar 22, 2018
    Configuration menu
    Copy the full SHA
    f723aaa View commit details
    Browse the repository at this point in the history