Store provenance in NIFTI files using niprov

[chrisgorgo]

NIprov https://github.com/ilogue/niprov is a library for storing provenance for neuroimaging analyses. We should consider tweaking Nipype to automatically save the provenance in the NIFTI files produced in each step. This would be incredibly useful! @satra @ilogue

[JasperVanDenBosch]

I'd love to help with this. Let me know what you guys need.

[satra]

hey jasper, what would it take to make niprov w3c prov compliant? since nipype itself has a provenance generator, i'm less inclined to adopt niprov to replace it. however, i'm happy to use to provide richer details, etc.,. since nipype's provenance tracker is quite basic.

regarding embedding provenance into nifti files, i'm willing to embed an external prov link, but i think it's a little silly to embed the same provenance into every 3d file that gets generated, when that happens. i have always said that it is the provenance that is the information or data flow, and the files are pieces of the provenance graph. i'd like to continue thinking that way.

[inati]

Be careful that every package that uses the nifti meta data does it differently and that there is no guarantee that your embedding of provenance will survive a processing step.

[JasperVanDenBosch]

Which w3c standard did you have in mind Satra? That would be a great thing.

Niprov optionally stores provenance for the file in question in the nifti
file header, but I'm more focused on the central (mongodb) storage. One of
the reasons for this is what Souheil mentions.
On Dec 7, 2015 7:05 PM, "Souheil Inati" notifications@github.com wrote:

Be careful that every package that uses the nifti meta data does it
differently and that there is no guarantee that your embedding of
provenance will survive a processing step.

—
Reply to this email directly or view it on GitHub
#1111 (comment).

[JasperVanDenBosch]

Also, I made a niprov integration piece for the python mne package, I
imagine I could do the same for nipype.
On Dec 7, 2015 7:11 PM, "Jasper van den Bosch" japsai@gmail.com wrote:

Which w3c standard did you have in mind Satra? That would be a great thing.

Niprov optionally stores provenance for the file in question in the nifti
file header, but I'm more focused on the central (mongodb) storage. One of
the reasons for this is what Souheil mentions.
On Dec 7, 2015 7:05 PM, "Souheil Inati" notifications@github.com wrote:

Be careful that every package that uses the nifti meta data does it
differently and that there is no guarantee that your embedding of
provenance will survive a processing step.

—
Reply to this email directly or view it on GitHub
#1111 (comment).

[satra]

w3c prov

the latest version that nipype implements is in this PR #1279 (see the provenance.py file, as well as pipeline/utils.py).

[JasperVanDenBosch]

Okay I have implemented some support for exporting to W3C PROV xml in niprov. It's a work in progress (see JasperVanDenBosch/niprov#71) but the basics for one image are there.

Also you can now configure niprov to attach the provenance to a file header with:

attach = xml

in the ~/niprov.cfg configuration file or in python code. Other formats include e.g. json and narrated. This works for nifti and fif (the mne format).

Other ways of exporting include to a text file or on the commandline:

provenance export -f xml

I ended up not using the python prov library, as it depends on the binary lxml package.

Let me know what you think!

[djarecka]

@satra @chrisfilo any new comments? is it still an issue?

[satra]

we have gone back and forth on whether to store provenance in files. given the work on pynidm, i would like to focus on that. in the longer term i think provenance will be outside a file with a relatively easy option to get provenance of files from a given registered file.