prototype workflow RO-Crate from snakemake workflow

[douglowe]

Work coming from the BGE hackathon in Leiden. Reporting of products made should go in the report here: https://docs.google.com/document/d/1if6ukMKN3xHQHAwGEQPhhgvp7iQcFnauj4W1ZtIs8wk/edit

Aim is to write a python tool which will create a workflow RO-Crate from the outputs and reports created from a snakemake workflow.

Snakemake workflow used: https://github.com/o-william-white/skim2mt.git

[tbrown91]

Hi @douglowe

I am able to give this more thought this week, so am wondering what the best next steps would be. At the moment all of the information I have pulled from the html are just sitting in variables. Do you think it will be easy to turn this into provenance ro-crate?

[douglowe]

Hi @tbrown91 - I'm getting a bit of time to look at this too, and have conflicting ideas about how to go about this.

In the long-term I think we can add to the snakemake runner itself, creating an 'ro-crate' report option, as an alternative to the html report. See this issue I created in a local copy of the snakemake repo: eScienceLab/snakemake#1

This probably should start with creating some example RO-Crate files (first a workflow crate, then script the building of a provenance crate from that, using the metadata pulled from the html report), so that we can build a test to include in the snakemake testing suites. Let's have a go at creating that this week?

[tbrown91]

Baby steps befd0dd

There are many things I don't like about the snakemake report, but particularly that the input and output files are not really listed or names. There are a number of wildcards left in, but maybe this is not important for a workflow RO-crate. For the provenance RO-crate I think we will not be able to extract the information we are looking for

[fbartusch]

Hi, I was added by @douglowe as collaborator to this repository.

You're scraping the information directly from the html report, right? The html report itself is generated by a Snakemake report plugin and uses the data stored in .snakemake/metadata/ in the workflow's main directory.
Since Snakemake 8 there is a plugin system for some functionality, among other things the report function.

There's a poetry template provided by Snakemake project for new plugins.
I fiddled a bit around a few weeks ago with the template was able to some provenance information rather quickly.

I think that's a cleaner way to get the needed information. Also, this does not break if html report changes it's structure/layout/content. I can provide some code in the next days, when my work schedule allows it.

[fbartusch]

The skim2mt workflow ran through on our cluster. I added an usable Snakemake report plugin to the repo and documented how you can rebuild it in the README.
The plugin works on the skim2mt metadata on our cluster (e.g. it does not throw errors). Although the plugin does nothing useful in the moment this is a good sign :)