Multiple nextflow scripts in a single repository

[ewels]

Note about finding a nice way to deal with having multiple nextflow scripts in a single GitHub repository, sharing a single config setup.

Reasons for doing this:

• Keeps everything together in one place
  • A single name, URL
  • Simple versioning (GitHub releases)
• Shared configuration profiles can be used across sub-pipelines
• Splitting up into multiple scripts keeps distinct functionalities separate
  • eg. using different sets of tools to process the same data type, as in NGI-MethylSeq
  • eg. massive pipelines made up of a handful of stages, as in Sarek (formerly CAW)

This generally works pretty well, simply defining the repo and the script name when running the pipeline. However, the shared configs need to define all processes, and Nextflow now generates a tonne of warning messages about processes that are configured but not part of the pipeline script.

There's also the question of how to manage and string such scripts together. Some approaches are already discussed by @skptic here. We're also thinking about writing a simple Python package tool that could manage things, read and write configs, build long nextflow launch commands and string multiple nextflow pipelines together.

Pinging @maxulysse and @boulund as we've discussed similar questions previously internally.

[pditommaso]

The warning has been introduced because it's quite easy to have typo in a process name in the config file which can badly effect a pipeline run without notice.

However I see your point, it could be possible to introduce an option to disable such warning. What do you think ?

[ewels]

I think that would be nice. Maybe with the option of providing a list a named processes to ignore? Then we'd still catch the typos. But still a lot less verbose than making a load of dummy processes.

[pditommaso]

This issue is solved by #623.