Skip to content

[Provenance] Store the execution metadata, including command line, when running a job #84

Closed

Description

@hmenager

Maybe also include the generated files.

Activity

  1. mr-c commented on Oct 26, 2015

    @mr-c
    Member

    +1 for the reference implementation to do so, moved to common-workflow-language/cwltool#9

    For the spec, I agree it that it would be useful to expose the metadata to the tools and require a specific report available to the user.

  2. hmenager commented on Oct 26, 2015

    @hmenager
    MemberAuthor

    +1 on moving this to ref implementation ;)

  3. added this to the milestone on Feb 16, 2016
  4. changed the title [-]Store the execution metadata, including command line, when running a job[/-] [+][Provenance] Store the execution metadata, including command line, when running a job[/+] on Mar 11, 2016
  5. mr-c commented on Mar 11, 2016

    @mr-c
    Member

    From @stain in chat

    I would like to work on how we can have a somewhat common view of provenance from a CWL worflow run - we can use what we've done in Taverna https://github.com/apache/incubator-taverna-engine/tree/master/taverna-prov#structure-of-exported-provenanceand https://w3id.org/ro/2016-01-28/wfprov as a starting point.
    I think something like our RO Bundle is needed so you can gather all the files made and used, and include copies of the CWL workflow and tool descriptions at the time of running.
    [...]
    in tavernaprov we added some more technical things that might not be needed in CWL as they could be engine specific
    like we included a way to show a snippet of the value within the graph
    but obviously you don't want to do that with larger values

  6. stain commented on Mar 15, 2016

    @stain
    Member

    Great draft, @ntijanic!

    It's OK that numbers are also wfprov:Artifact - they don't have to be files. I guess just value (rdf:value?) would make sense. In Taverna's workflow traces there's also UUID-like identifiers on the outputs so they can be cited elsewhere - but I don't think we need to require this. (This would make it easier to see that a workflow output value was the same ITEM as the one at a process run output, not just accidentally the same bytes).

    We have to be careful about how we reference the workflow steps so we get the identifiers straight, so how you do process: wf.yaml#s1 should be correct - each ProcessRun is an enactment of the step defined in the workflow (wfdesc:Process).

    I guess some confusion comes with the term.. in early wfdesc drafts we used to call all the wfdesc terms *Description to say that they are the plans for execution, so a wfdesc:Workflow is a workflow plan/description, and equivalent for wfdesc:Process. (It was dropped as the namespace already says desc :)

    I'll go through the scatter considerations in a separate comment.

  7. stain commented on Mar 15, 2016

    @stain
    Member

    First of all, perhaps it's useful to look at this recent release candidate for updating the wfdesc and wfprov ontologies. They should be made live some time later this month when we have finished reviewing them - feel free to help!)

    cwl:CommandLineTool should probably not be a wfdesc:Process as it can be used in several steps, but a wfdesc:ProcessImplementation, referred to with wfdesc:hasImplementation. (this is equivalent to Taverna's Activity concept)

    For a wfdesc:Workflow sometimes the equivalent wfdesc:WorkflowDefinition is the same (e.g. the top level Workflow in your example) - but in theory (and in practice for Taverna) the Workflow as a structure can have a common (UUID) identifier no matter where it is saved or included - hence the possibility to keep those distinct. I'm OK with CWL using the same identifier here as it's not a file-format where it's necessarily easy to remember to update such a UUID (as we do in Taverna's GUI).

    So the special case is when a workflow is used as a nested workflow twice - for CWL I think the easiest is to make the identifier for each the inner Workflow instances to match the outer cwl:Step - and then both can have wfdesc:hasWorkflowDefinition to the same file. That would mean a bit of trouble in the provenance of the inner workflow runs as there would be two runs with the same parameter names appearing.

    If a cwl:Step is a nested workflow, then the equivalent wfdesc:ProcessRuns gets 'upgraded' to wfdesc:WorkflowRuns which would have inner ProcessRuns linked to it with wfprov:wasPartOfWorkflowRun. So you can climb back out from that to find which process run (and in your model then eventually also which IterationRun) that particular workflow run was part of.

    But on the other hand, if you wanted to know "all values seen at a given port" across all scatter runs, this would still be easy as they would all be having the same describedByParameter link.

  8. stain commented on Mar 15, 2016

    @stain
    Member

    I quite like the IterationRun idea! I see how it messes up the ports as it doesn't have its own ports (actually internally in Taverna's engine the iteration strategy DO have its own ports, but we don't assign those identifiers in the provenance) - but I don't see a big problem with saying this is an intermediate step as long as you have distinct identifiers for each inner run that it wraps.

    So I would just use the same port parameters on both - it's not very different from the issue with nested workflow ports - where a workflow input port is both an outer receiver port and inner sender port.

  9. stain commented on Mar 16, 2016

    @stain
    Member

    Here's a bit of motivation for wfprov based on https://gist.github.com/stain/fa63a3527bd09be2a42d

    Here a particular run 6a6ffc3d of processor join_cd used two values at ports first and second:

    <run/process/6a6ffc3d-47cd-48c4-9a6e-8451481f15e3/>     a prov:Activity;
         prov:qualifiedAssociation  [ 
               prov:agent <workflowrun.prov.ttl#taverna-engine>; 
              # which step was run
               prov:hadPlan <wf/workflow/Hello_abcd/processor/join_cd/> ];
    
         prov:qualifiedUsage  
              [  # data 293661a3 used as input 'first'
                prov:entity <data/ref/293661a3-3e4e-4ff2-aff8-f8513a19a04a>;
                prov:hadRole <wf/workflow/Hello_abcd/processor/join_cd/in/first> ],
              [ # data 50d03488 used as input 'second'
                prov:entity <data/ref/50d03488-50e2-4a5b-9dc6-28579b24fcef>;
                prov:hadRole <wf/workflow/Hello_abcd/processor/join_cd/in/second> ];

    So yes, it's clear here which value was used in which port, but it's a bit elaborate as we have to invoke prov:hadRole to qualify the prov:used relation.

    wfprov is more descriptive with shortcuts like describedByProcess and usedInput:

    <run/process/6a6ffc3d-47cd-48c4-9a6e-8451481f15e3/>     a wfprov:WorkflowRun;
        wfprov:describedByProcess <wf/workflow/Hello_abcd/processor/join_cd/>;
        wfprov:usedInput <data/ref/293661a3-3e4e-4ff2-aff8-f8513a19a04a>,
                <data/ref/50d03488-50e2-4a5b-9dc6-28579b24fcef>;

    However if you care about at which input port, you would have to combine the wfdesc Workflow listing of its hasInput:

    <wf/workflow/Hello_abcd/processor/join_cd/> a wfdesc:Workflow ;
          rdfs:label "join_cd" ;
        wfdesc:hasInput <processor/join_cd/in/first> , <processor/join_cd/in/second> ;
        wfdesc:hasOutput <processor/join_cd/out/joint>  .

    and the ports the artifact appeared at:

    <data/ref/50d03488-50e2-4a5b-9dc6-28579b24fcef>     a wfprov:Artifact ;
      wfprov:describedByParameter <wf/workflow/Hello_abcd/processor/join_cd/in/second>,
                <wf/workflow/join_ab/processor/concatenate/in/string2> .
  10. stain commented on Mar 16, 2016

    @stain
    Member

    See https://gist.github.com/stain/fa63a3527bd09be2a42d#file-helloabcd-wfprov-ttl for an abbreviated example of wfprov from Taverna of a workflow that runs the same inserted nested workflow twice, iterating over it twice, which should mean 4 distinct calls to the inner "concatenate" process.

    I've shortened the URLs and removed the "duplicate" statements in pure PROV - see the rest of the gist for the gory details - or download the run bundle ZIP for the actual provenance output.

    I don't think the CWL output need to enforce a distinction between the data 'item' and its file with tavernaprov:content as we did in Taverna - but one reason we did it is that it would be wrong to claim that the serialized file was passed along in the workflow when the value only existed in memory as a data structure. (In Taverna provenance, those intermediate value files are saved out afterwards).

  11. stain commented on Mar 16, 2016

    @stain
    Member

    I did a quick translate of the helloabcd wfprov as JSON-LD.

    Can be shortened further!

  12. stain commented on Mar 16, 2016

    @stain
    Member

    No, we used describedByParameter to say it was the actual final value at the port - in earlier approaches we had "seen at port (possibly part of a list)" - but this made it very difficult to untangle the list and its members when trying to show the data afterwards as you would have to query for the "highest level list that doesn't directly or indirectly contain the others".

    (or as happened at one point - we didn't track the lists at all, which meant downstream consumers of the list seem to get the list out of thin air)

    But if you have two different parameters, then it would not longer be tangled - so you would then come back to the issue of minting separate identifiers for the inner and outer ports of the IterationRun.

    So instead we ended up declaring the list members using the cumbersome prov:Dictionary approach:

    <http://ns.taverna.org.uk/2011/data/60bce245-9f15-4391-902f-134bc699c583/list/71f5d67b-195d-49bb-94c0-1b800d18f72a/false/1>     
        a wfprov:Artifact, prov:Collection, prov:Dictionary,prov:Entity;
        # The individual values are **not** described by this parameter
         wfprov:describedByParameter <http://ns.taverna.org.uk/2010/workflowBundle/f5fddb99-9677-43cb-aac5-4d14e1ad1b46/workflow/Hello_abcd/processor/a_b/out/list>;
         wfprov:wasOutputFrom <http://ns.taverna.org.uk/2011/run/60bce245-9f15-4391-902f-134bc699c583/process/6680bce1-8b12-411f-841c-84df7b25cefb/>;
    # Short form - no order:
         prov:hadMember <http://ns.taverna.org.uk/2011/data/60bce245-9f15-4391-902f-134bc699c583/ref/0f086ff4-c030-42c3-ab05-ca5fca72e432>,
                <http://ns.taverna.org.uk/2011/data/60bce245-9f15-4391-902f-134bc699c583/ref/32eba706-636a-47c3-9607-a7712d7a0a49>;
    # Longer form - index as an `xsd:long`
         prov:hadDictionaryMember
            [ a prov:KeyEntityPair;
               prov:pairEntity <http://ns.taverna.org.uk/2011/data/60bce245-9f15-4391-902f-134bc699c583/ref/0f086ff4-c030-42c3-ab05-ca5fca72e432>;
               prov:pairKey "1"^^xsd:long ],
            [ a prov:KeyEntityPair;
               prov:pairEntity <http://ns.taverna.org.uk/2011/data/60bce245-9f15-4391-902f-134bc699c583/ref/32eba706-636a-47c3-9607-a7712d7a0a49>;
               prov:pairKey "0"^^xsd:long ];

    A list of lists (depth 2) would be done by another such prov:Dictionary listed as the prov:pairEntity - which become quite deep.

    With JSON-LD you can use native JSON lists and make them into a proper RDF List - however in PROV-O there was resistance against RDF lists as they could be tricky to support in the other PROV serializations. So to do that you would need a new property - perhaps it would be appropriate to include in wfdesc as I don't like the above.

    But there could be an argument for allowing describedByParameter also on the list items, such as when there is pipelining/item streaming and values are delivered before the final list - then the provenance could look inconsistent with a list being generated after its values have been consumed downstream.

  13. modified the milestones: unscheduled, on Jun 7, 2016
  14. added a commit that references this issue on Jul 20, 2017
  15. added a commit that references this issue on Mar 21, 2018
  16. stain commented on Apr 9, 2018

    @stain
    Member

    See how @FarahZKhan and myself implemented this in common-workflow-language/cwltool#676 - described (and example research object) in https://doi.org/10.5281/zenodo.1208478

  17. added a commit that references this issue on Jul 5, 2018
  18. mr-c commented on Jun 29, 2019

    @mr-c
    Member

    Do we want to officially recommend or suggest use of CWLProv ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions