First attempt at writing a new command to generate an RO Crate

[ewels]

New command nf-core rocrate:

$ nf-core rocrate --help

                                          ,--./,-.
          ___     __   __   __   ___     /,-._.--~\
    |\ | |__  __ /  ` /  \ |__) |__         }  {
    | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                          `._,._,'
    nf-core/tools version 1.10.dev0


Usage: nf-core rocrate [OPTIONS] <pipeline directory>

  Make an Research Object Crate

Options:
  -j, --json TEXT  Path to save RO Crate metadata json
  -z, --zip TEXT   Path to save RO Crate zip
  --help           Show this message and exit.

Generates an RO Crate .zip archive of the whole workflow + metadata files, or saves just the JSON metadata file.

For example:

$ nf-core rocrate nf-core-testpipeline/ --json ro-crate-metadata.json

                                          ,--./,-.
          ___     __   __   __   ___     /,-._.--~\
    |\ | |__  __ /  ` /  \ |__) |__         }  {
    | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                          `._,._,'
    nf-core/tools version 1.10.dev0


  INFO     Adding 38 workflow files                                                                                                                            ro_crate.py:112
  INFO     Saving metadata file 'ro-crate-metadata.json'

ro-crate-metadata.json

{
    "@context": "https://w3id.org/ro/crate/1.0/context",
    "@graph": [
        {
            "@id": "ro-crate-metadata.jsonld",
            "@type": "CreativeWork",
            "about": {
                "@id": "./"
            },
            "conformsTo": {
                "@id": "https://w3id.org/ro/crate/1.0"
            }
        },
        {
            "@id": "ro-crate-preview.html",
            "@type": "CreativeWork",
            "about": {
                "@id": "./"
            }
        },
        {
            "@id": "./",
            "@type": "CreativeWork",
            "about": {
                "@id": "./"
            },
            "datePublished": "2020-07-30 11:16:47.918409",
            "hasPart": [
                {
                    "@id": "nextflow.config"
                },
                {
                    "@id": "CODE_OF_CONDUCT.md"
                },
                {
                    "@id": "LICENSE"
                },
                {
                    "@id": "environment.yml"
                },
                {
                    "@id": "CHANGELOG.md"
                },
                {
                    "@id": "Dockerfile"
                },
                {
                    "@id": "README.md"
                },
                {
                    "@id": "gitignore"
                },
                {
                    "@id": "gitattributes"
                },
                {
                    "@id": "nextflow_schema.json"
                },
                {
                    "@id": "main.nf"
                },
                {
                    "@id": "bin/markdown_to_html.py"
                },
                {
                    "@id": "bin/scrape_software_versions.py"
                },
                {
                    "@id": "docs/usage.md"
                },
                {
                    "@id": "docs/output.md"
                },
                {
                    "@id": "docs/README.md"
                },
                {
                    "@id": "docs/images/nf-core-testpipeline_logo.png"
                },
                {
                    "@id": "github/PULL_REQUEST_TEMPLATE.md"
                },
                {
                    "@id": "github/.dockstore.yml"
                },
                {
                    "@id": "github/CONTRIBUTING.md"
                },
                {
                    "@id": "github/markdownlint.yml"
                },
                {
                    "@id": "github/workflows/branch.yml"
                },
                {
                    "@id": "github/workflows/push_dockerhub.yml"
                },
                {
                    "@id": "github/workflows/linting.yml"
                },
                {
                    "@id": "github/workflows/awsfulltest.yml"
                },
                {
                    "@id": "github/workflows/awstest.yml"
                },
                {
                    "@id": "github/workflows/ci.yml"
                },
                {
                    "@id": "github/ISSUE_TEMPLATE/feature_request.md"
                },
                {
                    "@id": "github/ISSUE_TEMPLATE/bug_report.md"
                },
                {
                    "@id": "assets/nf-core-testpipeline_logo.png"
                },
                {
                    "@id": "assets/email_template.html"
                },
                {
                    "@id": "assets/multiqc_config.yaml"
                },
                {
                    "@id": "assets/email_template.txt"
                },
                {
                    "@id": "assets/sendmail_template.txt"
                },
                {
                    "@id": "conf/test_full.config"
                },
                {
                    "@id": "conf/base.config"
                },
                {
                    "@id": "conf/test.config"
                },
                {
                    "@id": "conf/igenomes.config"
                }
            ],
            "mainEntity": {
                "@id": "nextflow.config"
            }
        },
        {
            "@id": "nextflow.config",
            "@type": [
                "File",
                "Workflow",
                "SoftwareSourceCode"
            ],
            "programmingLanguage": {
                "@id": "https://www.nextflow.io"
            }
        },
        {
            "@id": "CODE_OF_CONDUCT.md",
            "@type": "File"
        },
        {
            "@id": "LICENSE",
            "@type": "File"
        },
        {
            "@id": "environment.yml",
            "@type": "File"
        },
        {
            "@id": "CHANGELOG.md",
            "@type": "File"
        },
        {
            "@id": "Dockerfile",
            "@type": "File"
        },
        {
            "@id": "README.md",
            "@type": "File"
        },
        {
            "@id": "gitignore",
            "@type": "File"
        },
        {
            "@id": "gitattributes",
            "@type": "File"
        },
        {
            "@id": "nextflow_schema.json",
            "@type": "File"
        },
        {
            "@id": "main.nf",
            "@type": "File"
        },
        {
            "@id": "bin/markdown_to_html.py",
            "@type": "File"
        },
        {
            "@id": "bin/scrape_software_versions.py",
            "@type": "File"
        },
        {
            "@id": "docs/usage.md",
            "@type": "File"
        },
        {
            "@id": "docs/output.md",
            "@type": "File"
        },
        {
            "@id": "docs/README.md",
            "@type": "File"
        },
        {
            "@id": "docs/images/nf-core-testpipeline_logo.png",
            "@type": "File"
        },
        {
            "@id": "github/PULL_REQUEST_TEMPLATE.md",
            "@type": "File"
        },
        {
            "@id": "github/.dockstore.yml",
            "@type": "File"
        },
        {
            "@id": "github/CONTRIBUTING.md",
            "@type": "File"
        },
        {
            "@id": "github/markdownlint.yml",
            "@type": "File"
        },
        {
            "@id": "github/workflows/branch.yml",
            "@type": "File"
        },
        {
            "@id": "github/workflows/push_dockerhub.yml",
            "@type": "File"
        },
        {
            "@id": "github/workflows/linting.yml",
            "@type": "File"
        },
        {
            "@id": "github/workflows/awsfulltest.yml",
            "@type": "File"
        },
        {
            "@id": "github/workflows/awstest.yml",
            "@type": "File"
        },
        {
            "@id": "github/workflows/ci.yml",
            "@type": "File"
        },
        {
            "@id": "github/ISSUE_TEMPLATE/feature_request.md",
            "@type": "File"
        },
        {
            "@id": "github/ISSUE_TEMPLATE/bug_report.md",
            "@type": "File"
        },
        {
            "@id": "assets/nf-core-testpipeline_logo.png",
            "@type": "File"
        },
        {
            "@id": "assets/email_template.html",
            "@type": "File"
        },
        {
            "@id": "assets/multiqc_config.yaml",
            "@type": "File"
        },
        {
            "@id": "assets/email_template.txt",
            "@type": "File"
        },
        {
            "@id": "assets/sendmail_template.txt",
            "@type": "File"
        },
        {
            "@id": "conf/test_full.config",
            "@type": "File"
        },
        {
            "@id": "conf/base.config",
            "@type": "File"
        },
        {
            "@id": "conf/test.config",
            "@type": "File"
        },
        {
            "@id": "conf/igenomes.config",
            "@type": "File"
        }
    ]
}

See #686

PR checklist

• This comment contains a description of changes (with reason)
• If you've fixed a bug or added code that should be tested, add tests!
• Documentation in docs is updated
• CHANGELOG.md is updated
• README.md is updated

[ewels]

Now that this proof-of-concept is working, I think we should prep a PR to the ro-crate library for some of this functionality. Then wait for a release there and then use it..

There's also a lot more metadata that we can add yet.

[ewels]

Could be useful: tutorial by @stain - https://biocompute-objects.github.io/bco-ro-crate/

Also a YouTube talk: https://youtu.be/3APqPwRIRkA

[ewels]

May now be redundant due to https://github.com/crs4/repo2crate

[mr-c]

May now be redundant due to https://github.com/crs4/repo2crate

From what I can see, repo2crate is snakemake only?

[ewels]

From what I can see, repo2crate is snakemake only?

I haven't tried using it (only spied its existence in the WorkflowHub notes), but there's a nextflow.py file here? It's also listed under LANG_MODULES for the -l command line flag.

[stain]

Hi, happy to join in this!

@simleo is trying in repo2crate to generalize what was before parsing just snakemake.

It picks up some nf-core metadata if the repository is well-formed following the nf-core guidelines (which would be the same for this tool, I guess).

It can help for registration in our https://workflowhub.eu/ and also in Simone's https://lifemonitor.eu/ of the Nf-core tests.

I am not sure if @simleo has added a pypi release that you could try use as a Python dependency and call its make_crate() function?

[simleo]

Hi everyone. I published a first 0.1.0 release of repo2rocrate (that's the name of the package and of the command line tool, I should rename the repository accordingly) to PyPI yesterday. The project did start as Snakemake only, then I added support for Galaxy (IWC) and Nextflow (nf-core). I've just finished updating the docs, hopefully they are enough to get started. For now there's nothing about the API, but it should be straightforward: users should call make_crate from the relevant language-specific module (this is shown in the CLI implementation).

Feedback is very welcome!

[ewels]

Closing in favour of #2680