You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
[Provenance] Store the execution metadata, including command line, when running a job #84
changed the title [-]Store the execution metadata, including command line, when running a job[/-][+][Provenance] Store the execution metadata, including command line, when running a job[/+]on Mar 11, 2016
I would like to work on how we can have a somewhat common view of provenance from a CWL worflow run - we can use what we've done in Taverna https://github.com/apache/incubator-taverna-engine/tree/master/taverna-prov#structure-of-exported-provenanceandhttps://w3id.org/ro/2016-01-28/wfprov as a starting point.
I think something like our RO Bundle is needed so you can gather all the files made and used, and include copies of the CWL workflow and tool descriptions at the time of running.
[...]
in tavernaprov we added some more technical things that might not be needed in CWL as they could be engine specific
like we included a way to show a snippet of the value within the graph
but obviously you don't want to do that with larger values
It's OK that numbers are also wfprov:Artifact - they don't have to be files. I guess just value (rdf:value?) would make sense. In Taverna's workflow traces there's also UUID-like identifiers on the outputs so they can be cited elsewhere - but I don't think we need to require this. (This would make it easier to see that a workflow output value was the same ITEM as the one at a process run output, not just accidentally the same bytes).
We have to be careful about how we reference the workflow steps so we get the identifiers straight, so how you do process: wf.yaml#s1 should be correct - each ProcessRun is an enactment of the step defined in the workflow (wfdesc:Process).
I guess some confusion comes with the term.. in early wfdesc drafts we used to call all the wfdesc terms *Description to say that they are the plans for execution, so a wfdesc:Workflow is a workflow plan/description, and equivalent for wfdesc:Process. (It was dropped as the namespace already says desc :)
I'll go through the scatter considerations in a separate comment.
First of all, perhaps it's useful to look at this recent release candidate for updating the wfdesc and wfprov ontologies. They should be made live some time later this month when we have finished reviewing them - feel free to help!)
cwl:CommandLineTool should probably not be a wfdesc:Process as it can be used in several steps, but a wfdesc:ProcessImplementation, referred to with wfdesc:hasImplementation. (this is equivalent to Taverna's Activity concept)
For a wfdesc:Workflow sometimes the equivalent wfdesc:WorkflowDefinition is the same (e.g. the top level Workflow in your example) - but in theory (and in practice for Taverna) the Workflow as a structure can have a common (UUID) identifier no matter where it is saved or included - hence the possibility to keep those distinct. I'm OK with CWL using the same identifier here as it's not a file-format where it's necessarily easy to remember to update such a UUID (as we do in Taverna's GUI).
So the special case is when a workflow is used as a nested workflow twice - for CWL I think the easiest is to make the identifier for each the inner Workflow instances to match the outer cwl:Step - and then both can have wfdesc:hasWorkflowDefinition to the same file. That would mean a bit of trouble in the provenance of the inner workflow runs as there would be two runs with the same parameter names appearing.
If a cwl:Step is a nested workflow, then the equivalent wfdesc:ProcessRuns gets 'upgraded' to wfdesc:WorkflowRuns which would have inner ProcessRuns linked to it with wfprov:wasPartOfWorkflowRun. So you can climb back out from that to find which process run (and in your model then eventually also which IterationRun) that particular workflow run was part of.
But on the other hand, if you wanted to know "all values seen at a given port" across all scatter runs, this would still be easy as they would all be having the same describedByParameter link.
I quite like the IterationRun idea! I see how it messes up the ports as it doesn't have its own ports (actually internally in Taverna's engine the iteration strategy DO have its own ports, but we don't assign those identifiers in the provenance) - but I don't see a big problem with saying this is an intermediate step as long as you have distinct identifiers for each inner run that it wraps.
So I would just use the same port parameters on both - it's not very different from the issue with nested workflow ports - where a workflow input port is both an outer receiver port and inner sender port.
Here a particular run 6a6ffc3d of processor join_cd used two values at ports first and second:
<run/process/6a6ffc3d-47cd-48c4-9a6e-8451481f15e3/>a prov:Activity;
prov:qualifiedAssociation [
prov:agent <workflowrun.prov.ttl#taverna-engine>;
# which step was run
prov:hadPlan <wf/workflow/Hello_abcd/processor/join_cd/> ];
prov:qualifiedUsage
[ # data 293661a3 used as input 'first'
prov:entity <data/ref/293661a3-3e4e-4ff2-aff8-f8513a19a04a>;
prov:hadRole <wf/workflow/Hello_abcd/processor/join_cd/in/first> ],
[ # data 50d03488 used as input 'second'
prov:entity <data/ref/50d03488-50e2-4a5b-9dc6-28579b24fcef>;
prov:hadRole <wf/workflow/Hello_abcd/processor/join_cd/in/second> ];
So yes, it's clear here which value was used in which port, but it's a bit elaborate as we have to invoke prov:hadRole to qualify the prov:used relation.
wfprov is more descriptive with shortcuts like describedByProcess and usedInput:
I've shortened the URLs and removed the "duplicate" statements in pure PROV - see the rest of the gist for the gory details - or download the run bundle ZIP for the actual provenance output.
I don't think the CWL output need to enforce a distinction between the data 'item' and its file with tavernaprov:content as we did in Taverna - but one reason we did it is that it would be wrong to claim that the serialized file was passed along in the workflow when the value only existed in memory as a data structure. (In Taverna provenance, those intermediate value files are saved out afterwards).
No, we used describedByParameter to say it was the actual final value at the port - in earlier approaches we had "seen at port (possibly part of a list)" - but this made it very difficult to untangle the list and its members when trying to show the data afterwards as you would have to query for the "highest level list that doesn't directly or indirectly contain the others".
(or as happened at one point - we didn't track the lists at all, which meant downstream consumers of the list seem to get the list out of thin air)
But if you have two different parameters, then it would not longer be tangled - so you would then come back to the issue of minting separate identifiers for the inner and outer ports of the IterationRun.
So instead we ended up declaring the list members using the cumbersome prov:Dictionary approach:
<http://ns.taverna.org.uk/2011/data/60bce245-9f15-4391-902f-134bc699c583/list/71f5d67b-195d-49bb-94c0-1b800d18f72a/false/1>a wfprov:Artifact, prov:Collection, prov:Dictionary,prov:Entity;
# The individual values are **not** described by this parameter
wfprov:describedByParameter <http://ns.taverna.org.uk/2010/workflowBundle/f5fddb99-9677-43cb-aac5-4d14e1ad1b46/workflow/Hello_abcd/processor/a_b/out/list>;
wfprov:wasOutputFrom <http://ns.taverna.org.uk/2011/run/60bce245-9f15-4391-902f-134bc699c583/process/6680bce1-8b12-411f-841c-84df7b25cefb/>;
# Short form - no order:
prov:hadMember <http://ns.taverna.org.uk/2011/data/60bce245-9f15-4391-902f-134bc699c583/ref/0f086ff4-c030-42c3-ab05-ca5fca72e432>,
<http://ns.taverna.org.uk/2011/data/60bce245-9f15-4391-902f-134bc699c583/ref/32eba706-636a-47c3-9607-a7712d7a0a49>;
# Longer form - index as an `xsd:long`
prov:hadDictionaryMember
[ a prov:KeyEntityPair;
prov:pairEntity <http://ns.taverna.org.uk/2011/data/60bce245-9f15-4391-902f-134bc699c583/ref/0f086ff4-c030-42c3-ab05-ca5fca72e432>;
prov:pairKey "1"^^xsd:long ],
[ a prov:KeyEntityPair;
prov:pairEntity <http://ns.taverna.org.uk/2011/data/60bce245-9f15-4391-902f-134bc699c583/ref/32eba706-636a-47c3-9607-a7712d7a0a49>;
prov:pairKey "0"^^xsd:long ];
A list of lists (depth 2) would be done by another such prov:Dictionary listed as the prov:pairEntity - which become quite deep.
With JSON-LD you can use native JSON lists and make them into a proper RDF List - however in PROV-O there was resistance against RDF lists as they could be tricky to support in the other PROV serializations. So to do that you would need a new property - perhaps it would be appropriate to include in wfdesc as I don't like the above.
But there could be an argument for allowing describedByParameter also on the list items, such as when there is pipelining/item streaming and values are delivered before the final list - then the provenance could look inconsistent with a list being generated after its values have been consumed downstream.
Activity
mr-c commentedon Oct 26, 2015
+1 for the reference implementation to do so, moved to common-workflow-language/cwltool#9
For the spec, I agree it that it would be useful to expose the metadata to the tools and require a specific report available to the user.
hmenager commentedon Oct 26, 2015
+1 on moving this to ref implementation ;)
[-]Store the execution metadata, including command line, when running a job[/-][+][Provenance] Store the execution metadata, including command line, when running a job[/+]mr-c commentedon Mar 11, 2016
From @stain in chat
stain commentedon Mar 15, 2016
Great draft, @ntijanic!
It's OK that numbers are also
wfprov:Artifact- they don't have to be files. I guess justvalue(rdf:value?) would make sense. In Taverna's workflow traces there's also UUID-like identifiers on the outputs so they can be cited elsewhere - but I don't think we need to require this. (This would make it easier to see that a workflow output value was the same ITEM as the one at a process run output, not just accidentally the same bytes).We have to be careful about how we reference the workflow steps so we get the identifiers straight, so how you do
process: wf.yaml#s1should be correct - eachProcessRunis an enactment of the step defined in the workflow (wfdesc:Process).I guess some confusion comes with the term.. in early wfdesc drafts we used to call all the wfdesc terms
*Descriptionto say that they are the plans for execution, so a wfdesc:Workflow is a workflow plan/description, and equivalent forwfdesc:Process. (It was dropped as the namespace already saysdesc:)I'll go through the scatter considerations in a separate comment.
stain commentedon Mar 15, 2016
First of all, perhaps it's useful to look at this recent release candidate for updating the wfdesc and wfprov ontologies. They should be made live some time later this month when we have finished reviewing them - feel free to help!)
cwl:CommandLineToolshould probably not be awfdesc:Processas it can be used in several steps, but a wfdesc:ProcessImplementation, referred to with wfdesc:hasImplementation. (this is equivalent to Taverna'sActivityconcept)For a
wfdesc:Workflowsometimes the equivalent wfdesc:WorkflowDefinition is the same (e.g. the top level Workflow in your example) - but in theory (and in practice for Taverna) the Workflow as a structure can have a common (UUID) identifier no matter where it is saved or included - hence the possibility to keep those distinct. I'm OK with CWL using the same identifier here as it's not a file-format where it's necessarily easy to remember to update such a UUID (as we do in Taverna's GUI).So the special case is when a workflow is used as a nested workflow twice - for CWL I think the easiest is to make the identifier for each the inner
Workflowinstances to match the outercwl:Step- and then both can havewfdesc:hasWorkflowDefinitionto the same file. That would mean a bit of trouble in the provenance of the inner workflow runs as there would be two runs with the same parameter names appearing.If a
cwl:Stepis a nested workflow, then the equivalent wfdesc:ProcessRuns gets 'upgraded' to wfdesc:WorkflowRuns which would have innerProcessRuns linked to it with wfprov:wasPartOfWorkflowRun. So you can climb back out from that to find which process run (and in your model then eventually also whichIterationRun) that particular workflow run was part of.But on the other hand, if you wanted to know "all values seen at a given port" across all scatter runs, this would still be easy as they would all be having the same
describedByParameterlink.stain commentedon Mar 15, 2016
I quite like the
IterationRunidea! I see how it messes up the ports as it doesn't have its own ports (actually internally in Taverna's engine the iteration strategy DO have its own ports, but we don't assign those identifiers in the provenance) - but I don't see a big problem with saying this is an intermediate step as long as you have distinct identifiers for each inner run that it wraps.So I would just use the same port parameters on both - it's not very different from the issue with nested workflow ports - where a workflow input port is both an outer receiver port and inner sender port.
stain commentedon Mar 16, 2016
Here's a bit of motivation for wfprov based on https://gist.github.com/stain/fa63a3527bd09be2a42d
Here a particular run
6a6ffc3dof processorjoin_cdused two values at portsfirstandsecond:So yes, it's clear here which value was used in which port, but it's a bit elaborate as we have to invoke
prov:hadRoleto qualify theprov:usedrelation.wfprov is more descriptive with shortcuts like
describedByProcessandusedInput:However if you care about at which input port, you would have to combine the wfdesc
Workflowlisting of itshasInput:and the ports the artifact appeared at:
stain commentedon Mar 16, 2016
See https://gist.github.com/stain/fa63a3527bd09be2a42d#file-helloabcd-wfprov-ttl for an abbreviated example of wfprov from Taverna of a workflow that runs the same inserted nested workflow twice, iterating over it twice, which should mean 4 distinct calls to the inner "concatenate" process.
I've shortened the URLs and removed the "duplicate" statements in pure PROV - see the rest of the gist for the gory details - or download the run bundle ZIP for the actual provenance output.
I don't think the CWL output need to enforce a distinction between the data 'item' and its file with
tavernaprov:contentas we did in Taverna - but one reason we did it is that it would be wrong to claim that the serialized file was passed along in the workflow when the value only existed in memory as a data structure. (In Taverna provenance, those intermediate value files are saved out afterwards).stain commentedon Mar 16, 2016
I did a quick translate of the helloabcd wfprov as JSON-LD.
Can be shortened further!
stain commentedon Mar 16, 2016
No, we used
describedByParameterto say it was the actual final value at the port - in earlier approaches we had "seen at port (possibly part of a list)" - but this made it very difficult to untangle the list and its members when trying to show the data afterwards as you would have to query for the "highest level list that doesn't directly or indirectly contain the others".(or as happened at one point - we didn't track the lists at all, which meant downstream consumers of the list seem to get the list out of thin air)
But if you have two different parameters, then it would not longer be tangled - so you would then come back to the issue of minting separate identifiers for the inner and outer ports of the
IterationRun.So instead we ended up declaring the list members using the cumbersome prov:Dictionary approach:
A list of lists (depth 2) would be done by another such
prov:Dictionarylisted as theprov:pairEntity- which become quite deep.With JSON-LD you can use native JSON lists and make them into a proper RDF List - however in PROV-O there was resistance against RDF lists as they could be tricky to support in the other PROV serializations. So to do that you would need a new property - perhaps it would be appropriate to include in wfdesc as I don't like the above.
But there could be an argument for allowing describedByParameter also on the list items, such as when there is pipelining/item streaming and values are delivered before the final list - then the provenance could look inconsistent with a list being generated after its values have been consumed downstream.
Merge pull request #84 from common-workflow-language/mypy0.470
Squashed 'v1.1.0-dev1/salad/' changes from 98a5bea..27c20e8
stain commentedon Apr 9, 2018
See how @FarahZKhan and myself implemented this in common-workflow-language/cwltool#676 - described (and example research object) in https://doi.org/10.5281/zenodo.1208478
NOTICE with copyright
mr-c commentedon Jun 29, 2019
Do we want to officially recommend or suggest use of CWLProv ?