using datalad for provenance tracking in nipype - how to?

[pvavra]

I am interested in using nipype to set up my analysis pipeline and datalad, especially datalad run for provenance tracking (and version control). However, I’m not sure about how to go about this and I would appreciate your opinion on this.

I wasn't sure where to post this. Since this not a bug per se, but a rather large feature-request involving datalad too, I created a neurostars.org thread on this. Feel free to close this issue, and reply your thoughts on neurostars instead. To get the datalad devs to see the post too, I'm posting a similar issue over at the datalad repo.

[yarikoptic]

I think there are more than 1 way to skin a cat here, or in other words -- provenance could be useful across different level. I tend to think that using datalad run (or even better datalad containers-run) on top of nipype workflows might be the best way ATM.

But as individual interface calls within a pipeline typically work within a workdir I think it might be quite feasible (unless for huge pipelines/datasets) for those who would like to keep those workdir's somewhere in the archive (but does anyone? well, people could keep them empty without files and be able to datalad rerun specific step to get output for some interface) to

• have workdir an independent datalad dataset
• wrap each call with datalad run record to record effects
  and then have datalad run (or containers-run) could record the modified state of that workdir subdataset if it is linked to it.