[Provenance] Store the execution metadata, including command line, when running a job

[hmenager]

Maybe also include the generated files.

[ntijanic]

Is this for reference implementation, or do you think the spec should cover how to do this consistently?

If latter, perhaps we could just specify a few properties to extend wfprov. Some regular metadata is already handled by PROV-O, but might be missing stuff like command line and environment setup used. It might be worth adding these properties elsewhere in generic form (EDAM?).

[mr-c]

+1 for the reference implementation to do so, moved to common-workflow-language/cwltool#9

For the spec, I agree it that it would be useful to expose the metadata to the tools and require a specific report available to the user.

[hmenager]

+1 on moving this to ref implementation ;)

[mr-c]

From @stain in chat

I would like to work on how we can have a somewhat common view of provenance from a CWL worflow run - we can use what we've done in Taverna https://github.com/apache/incubator-taverna-engine/tree/master/taverna-prov#structure-of-exported-provenanceand https://w3id.org/ro/2016-01-28/wfprov as a starting point.
I think something like our RO Bundle is needed so you can gather all the files made and used, and include copies of the CWL workflow and tool descriptions at the time of running.
[...]
in tavernaprov we added some more technical things that might not be needed in CWL as they could be engine specific
like we included a way to show a snippet of the value within the graph
but obviously you don't want to do that with larger values

[ntijanic]

Also from @stain:

perhaps come up with a decent JSON-LD profile for wfprov

+1.

[ntijanic]

Playing around with structure of run snapshots. Starting with the simplest workflow:

# wf.yaml
class: Workflow
inputs:
  - id: i1
outputs:
  - id: o1
    source: s1.o1
steps:
  - id: s1
    run: tool.yaml
    inputs:
      - id: s1.i1
        source: i1

This one just has a single step - a tool that e.g. increments its input integer.
After running (i1=1), we can get a wfprov:WorkflowRun that might be represented in JSON-LD like:

class: WorkflowRun                          # type
id: "#run1"
process: wf.yaml                            # describedByProcess
enactedBy: http://example.org/engine
inputs:                                     # reverse of usedInput
  - {port: wf.yaml#i1, value: 1}            # port = describedByParameter
outputs:                                    # reverse of wasOutputFrom
  - {port: wf.yaml#o1, value: 2}
runs:                                       # reverse of wasPartOfWorkflowRun
  - class: ProcessRun
    id: "#run1.s1"
    process: wf.yaml#s1                     # Step, not tool.
    enactedBy: http://example.org/engine
    inputs:
      - {port: wf.yaml#s1.i1, value: 1}
    outputs:
      - {port: wf.yaml#s1.o1, value: 2}

Not quite sure how to represent wfprov:Artifacts that are just integers, so here they are objects with value property.
Also not sure if this is even a valid wfprov:WorkflowRun.

The trickiness comes when we nest this workflow in an outer workflow and scatter it:

# wf-outer.yaml
class: Workflow
outputs:
  - id: o1
    source: s1.o1
steps:
  - id: s1
    run: wf.yaml
    scatter: s1.i1
    inputs:
      - id: s1.i1
        source: i1

Suppose we run this with [1, 2] as input.
Should do a scattered increment and get [2, 3] as result.
Without any RDF restrictions, I would encode the runs/jobs like this:

- class: WorkflowJob
  id: run1
  app: wf-outer.yaml
  parent: null
  inputs: {i1: [1, 2]}
  outputs: {o1: [2, 3]}
- class: ScatterJob
  id: run1.s1
  app: null
  parent: run1
  inputs: {i1: [1, 2]}
  outputs: {o1: [2, 3]}
- class: WorkflowJob
  id: run1.s1.1
  app: wf.yaml
  parent: run1.s1
  inputs: {i1: 1}
  outputs: {o1: 2}
- class: WorkflowJob
  id: run1.s1.2
  app: wf.yaml
  parent: run1.s1
  inputs: {i1: 2}
  outputs: {o1: 3}
- class: CommandLineJob
  id: run1.s1.1.s1
  app: tool.yaml
  parent: run1.s1.1
  inputs: {i1: 1}
  outputs: {o1: 2}
- class: CommandLineJob
  id: run1.s1.2.s1
  app: tool.yaml
  parent: run1.s1.2
  inputs: {i1: 2}
  outputs: {o1: 3}

There's a root job (run1), a ScatterJob which is a container for iterations,
two nested WorkflowJobs, each with own CommandLineJob.

Job identifiers are built semantically: <parent_id>.<step_id> or <parent_id>.<iteration_index>.
This should work since CWL local identifiers should match the [A-Za-z][A-Za-z0-9]* pattern.
An improvement would be to have IRIs (e.g. http://example.org/jobs/run1.s1).
It's only a nicety - IDs can be anything (hopefully URLs though).
Likewise, app references should be URLs.

Putting this into wfprov is tricky. Best I could come up is:

class: WorkflowRun
id: "#run1"
process: wf-outer.yaml
inputs:
  - {port: wf-outer.yaml#i1, valueJson: "[1, 2]"}
outputs:
  - {port: wf-outer.yaml#o1, valueJson: "[2, 3]"}
runs:
  - class: IterationRun
    id: "#run1.s1"
    process: wf-outer.yaml#s1
    inputs:
      - {port: wf-outer.yaml#s1.i1, valueJson: "[1, 2]"}
    outputs:
      - {port: wf-outer.yaml#s1.o1, valueJson: "[2, 3]"}
    runs:
      - class: WorkflowRun
        id: "#run1.s1.1"
        process: wf-outer.yaml#s1
        inputs: [{port: wf.yaml#i1, value: 1}]
        outputs: [{port: wf.yaml#o1, value: 2}]
        runs:
          - class: ProcessRun
            id: "#run1.s1.1.s1"
            process: wf.yaml#s1
            inputs: [{port: wf.yaml#s1.i1, value: 1}]
            outputs: [{port: wf.yaml#s1.o1, value: 2}]
      - class: WorkflowRun
        id: "#run1.s1.2"
        process: wf-outer.yaml#s1
        inputs: [{port: wf.yaml#i1, value: 2}]
        outputs: [{port: wf.yaml#o1, value: 3}]
        runs:
          - class: ProcessRun
            id: "#run1.s1.2.s1"
            process: wf.yaml#s1
            inputs: [{port: wf.yaml#s1.i1, value: 2}]
            outputs: [{port: wf.yaml#s1.o1, value: 3}]

Few issues here:

• Needed to encode the arrays in a valueJson property. This is a known issue with CWL though (encoding non-primitive values in RDF).
• IterationRun (not part of wfprov) is a specialization of a ProcessRun, similar to the previously mentioned ScatterJob. Not sure how I would encode the iterations without it.
• The wfprov:wasPartOfWorkflowRun property doesn't quite fit with IterationRuns.
• For the nested WorkflowRuns, there's an inconsistency with port properties.
  If this was a non-scattered workflow component, those would point to inputs and outputs of the step in the outer workflow
  (in the previous example ProcessRuns had e.g. {port: wf.yaml#s1.i1, value: 1} rather than {port: tool.yaml#i1, value: 1}).

I'm still confused about wfdesc:Process and how it maps to CWL.
It seems to map to cwl:Step nicely, but it also maps to cwl:Workflow and cwl:CommandLineTool.
Perhaps if I found an example wfdesc:Workflow that has two instances of the same workflow as components the confusion would be cleared :)

[stain]

Great draft, @ntijanic!

It's OK that numbers are also wfprov:Artifact - they don't have to be files. I guess just value (rdf:value?) would make sense. In Taverna's workflow traces there's also UUID-like identifiers on the outputs so they can be cited elsewhere - but I don't think we need to require this. (This would make it easier to see that a workflow output value was the same ITEM as the one at a process run output, not just accidentally the same bytes).

We have to be careful about how we reference the workflow steps so we get the identifiers straight, so how you do process: wf.yaml#s1 should be correct - each ProcessRun is an enactment of the step defined in the workflow (wfdesc:Process).

I guess some confusion comes with the term.. in early wfdesc drafts we used to call all the wfdesc terms *Description to say that they are the plans for execution, so a wfdesc:Workflow is a workflow plan/description, and equivalent for wfdesc:Process. (It was dropped as the namespace already says desc :)

I'll go through the scatter considerations in a separate comment.

[stain]

First of all, perhaps it's useful to look at this recent release candidate for updating the wfdesc and wfprov ontologies. They should be made live some time later this month when we have finished reviewing them - feel free to help!)

cwl:CommandLineTool should probably not be a wfdesc:Process as it can be used in several steps, but a wfdesc:ProcessImplementation, referred to with wfdesc:hasImplementation. (this is equivalent to Taverna's Activity concept)

For a wfdesc:Workflow sometimes the equivalent wfdesc:WorkflowDefinition is the same (e.g. the top level Workflow in your example) - but in theory (and in practice for Taverna) the Workflow as a structure can have a common (UUID) identifier no matter where it is saved or included - hence the possibility to keep those distinct. I'm OK with CWL using the same identifier here as it's not a file-format where it's necessarily easy to remember to update such a UUID (as we do in Taverna's GUI).

So the special case is when a workflow is used as a nested workflow twice - for CWL I think the easiest is to make the identifier for each the inner Workflow instances to match the outer cwl:Step - and then both can have wfdesc:hasWorkflowDefinition to the same file. That would mean a bit of trouble in the provenance of the inner workflow runs as there would be two runs with the same parameter names appearing.

If a cwl:Step is a nested workflow, then the equivalent wfdesc:ProcessRuns gets 'upgraded' to wfdesc:WorkflowRuns which would have inner ProcessRuns linked to it with wfprov:wasPartOfWorkflowRun. So you can climb back out from that to find which process run (and in your model then eventually also which IterationRun) that particular workflow run was part of.

But on the other hand, if you wanted to know "all values seen at a given port" across all scatter runs, this would still be easy as they would all be having the same describedByParameter link.

[stain]

I quite like the IterationRun idea! I see how it messes up the ports as it doesn't have its own ports (actually internally in Taverna's engine the iteration strategy DO have its own ports, but we don't assign those identifiers in the provenance) - but I don't see a big problem with saying this is an intermediate step as long as you have distinct identifiers for each inner run that it wraps.

So I would just use the same port parameters on both - it's not very different from the issue with nested workflow ports - where a workflow input port is both an outer receiver port and inner sender port.

[stain]

Here's a bit of motivation for wfprov based on https://gist.github.com/stain/fa63a3527bd09be2a42d

Here a particular run 6a6ffc3d of processor join_cd used two values at ports first and second:

<run/process/6a6ffc3d-47cd-48c4-9a6e-8451481f15e3/>     a prov:Activity;
     prov:qualifiedAssociation  [ 
           prov:agent <workflowrun.prov.ttl#taverna-engine>; 
          # which step was run
           prov:hadPlan <wf/workflow/Hello_abcd/processor/join_cd/> ];

     prov:qualifiedUsage  
          [  # data 293661a3 used as input 'first'
            prov:entity <data/ref/293661a3-3e4e-4ff2-aff8-f8513a19a04a>;
            prov:hadRole <wf/workflow/Hello_abcd/processor/join_cd/in/first> ],
          [ # data 50d03488 used as input 'second'
            prov:entity <data/ref/50d03488-50e2-4a5b-9dc6-28579b24fcef>;
            prov:hadRole <wf/workflow/Hello_abcd/processor/join_cd/in/second> ];

So yes, it's clear here which value was used in which port, but it's a bit elaborate as we have to invoke prov:hadRole to qualify the prov:used relation.

wfprov is more descriptive with shortcuts like describedByProcess and usedInput:

<run/process/6a6ffc3d-47cd-48c4-9a6e-8451481f15e3/>     a wfprov:WorkflowRun;
    wfprov:describedByProcess <wf/workflow/Hello_abcd/processor/join_cd/>;
    wfprov:usedInput <data/ref/293661a3-3e4e-4ff2-aff8-f8513a19a04a>,
            <data/ref/50d03488-50e2-4a5b-9dc6-28579b24fcef>;

However if you care about at which input port, you would have to combine the wfdesc Workflow listing of its hasInput:

<wf/workflow/Hello_abcd/processor/join_cd/> a wfdesc:Workflow ;
      rdfs:label "join_cd" ;
    wfdesc:hasInput <processor/join_cd/in/first> , <processor/join_cd/in/second> ;
    wfdesc:hasOutput <processor/join_cd/out/joint>  .

and the ports the artifact appeared at:

<data/ref/50d03488-50e2-4a5b-9dc6-28579b24fcef>     a wfprov:Artifact ;
  wfprov:describedByParameter <wf/workflow/Hello_abcd/processor/join_cd/in/second>,
            <wf/workflow/join_ab/processor/concatenate/in/string2> .

[stain]

See https://gist.github.com/stain/fa63a3527bd09be2a42d#file-helloabcd-wfprov-ttl for an abbreviated example of wfprov from Taverna of a workflow that runs the same inserted nested workflow twice, iterating over it twice, which should mean 4 distinct calls to the inner "concatenate" process.

I've shortened the URLs and removed the "duplicate" statements in pure PROV - see the rest of the gist for the gory details - or download the run bundle ZIP for the actual provenance output.

I don't think the CWL output need to enforce a distinction between the data 'item' and its file with tavernaprov:content as we did in Taverna - but one reason we did it is that it would be wrong to claim that the serialized file was passed along in the workflow when the value only existed in memory as a data structure. (In Taverna provenance, those intermediate value files are saved out afterwards).

[stain]

I did a quick translate of the helloabcd wfprov as JSON-LD.

Can be shortened further!

[ntijanic]

Wow. Thanks @stain, this clarifies a lot for me.

I'll need time to go over it in detail; some questions/comments meanwhile:

But on the other hand, if you wanted to know "all values seen at a given port" across all scatter runs, this would still be easy as they would all be having the same describedByParameter link.

Ah, so describedByParameter, usedInput and wasOutputFrom don't necessarily mean that it was the exact value on that port - it could mean they were part of value on that port? Similarly, data/list/Hello-ab is a list and references its members through hadMember. But where do you encode the actual list value (preserving the ordering of list items)?

I don't think the CWL output need to enforce a distinction between the data 'item' and its file

Similar issue to above, since CWL's artifacts can be arbitrary dicts-and-lists trees which can contain files as leaves. Perhaps we can somehow store the tree itself, plus a flat set of all contained files/artifacts through hadMember.

I really like the idea of wrapping everything in an Artifact to signify that it's not just any 42, it's the same 42 that was the output from previous step.

[stain]

No, we used describedByParameter to say it was the actual final value at the port - in earlier approaches we had "seen at port (possibly part of a list)" - but this made it very difficult to untangle the list and its members when trying to show the data afterwards as you would have to query for the "highest level list that doesn't directly or indirectly contain the others".

(or as happened at one point - we didn't track the lists at all, which meant downstream consumers of the list seem to get the list out of thin air)

But if you have two different parameters, then it would not longer be tangled - so you would then come back to the issue of minting separate identifiers for the inner and outer ports of the IterationRun.

So instead we ended up declaring the list members using the cumbersome prov:Dictionary approach:

<http://ns.taverna.org.uk/2011/data/60bce245-9f15-4391-902f-134bc699c583/list/71f5d67b-195d-49bb-94c0-1b800d18f72a/false/1>     
    a wfprov:Artifact, prov:Collection, prov:Dictionary,prov:Entity;
    # The individual values are **not** described by this parameter
     wfprov:describedByParameter <http://ns.taverna.org.uk/2010/workflowBundle/f5fddb99-9677-43cb-aac5-4d14e1ad1b46/workflow/Hello_abcd/processor/a_b/out/list>;
     wfprov:wasOutputFrom <http://ns.taverna.org.uk/2011/run/60bce245-9f15-4391-902f-134bc699c583/process/6680bce1-8b12-411f-841c-84df7b25cefb/>;
# Short form - no order:
     prov:hadMember <http://ns.taverna.org.uk/2011/data/60bce245-9f15-4391-902f-134bc699c583/ref/0f086ff4-c030-42c3-ab05-ca5fca72e432>,
            <http://ns.taverna.org.uk/2011/data/60bce245-9f15-4391-902f-134bc699c583/ref/32eba706-636a-47c3-9607-a7712d7a0a49>;
# Longer form - index as an `xsd:long`
     prov:hadDictionaryMember
        [ a prov:KeyEntityPair;
           prov:pairEntity <http://ns.taverna.org.uk/2011/data/60bce245-9f15-4391-902f-134bc699c583/ref/0f086ff4-c030-42c3-ab05-ca5fca72e432>;
           prov:pairKey "1"^^xsd:long ],
        [ a prov:KeyEntityPair;
           prov:pairEntity <http://ns.taverna.org.uk/2011/data/60bce245-9f15-4391-902f-134bc699c583/ref/32eba706-636a-47c3-9607-a7712d7a0a49>;
           prov:pairKey "0"^^xsd:long ];

A list of lists (depth 2) would be done by another such prov:Dictionary listed as the prov:pairEntity - which become quite deep.

With JSON-LD you can use native JSON lists and make them into a proper RDF List - however in PROV-O there was resistance against RDF lists as they could be tricky to support in the other PROV serializations. So to do that you would need a new property - perhaps it would be appropriate to include in wfdesc as I don't like the above.

But there could be an argument for allowing describedByParameter also on the list items, such as when there is pipelining/item streaming and values are delivered before the final list - then the provenance could look inconsistent with a list being generated after its values have been consumed downstream.

[stain]

See how @FarahZKhan and myself implemented this in common-workflow-language/cwltool#676 - described (and example research object) in https://doi.org/10.5281/zenodo.1208478

[mr-c]

Do we want to officially recommend or suggest use of CWLProv ?