Integrating workflows into a single workflow

[mahesh-panchal]

It would be helpful if all stages of genome annotation could be run in a single workflow.

@Juke34 Provided this diagram of how the single workflow might look like.

With the introduction of Nextflow modules, each stage can be built as independent workflows and
then imported into a single workflow.

e.g. (note: modules must be locally available).

include foo from './some/module'

workflow {
    data = Channel.fromPath('/some/data/*.txt')
    foo(data)
}

Part of the suggested workflow proposes to include a part from another workflow:
https://github.com/ikmb-denbi/genome-annotation

A collaboration has been suggested, such that we just need to include the proposed part(s) into the single workflow. This requires the ikmb-denbi workflow to be converted to DSL2 and modularized. Together, we need to ensure interoperability between modules. The NBIS annotation workflows are almost complete to be used as modules.

Tasks:

• Organelle detection pipeline see [New pipeline] OrganelleDetection pipeline #5 (not important yet)
• Preprocessing pipeline (NBIS)
• De novo repeat library pipeline see [New pipeline] DeNovoRepeatLib #32 and [New pipeline] RepeatMaskMyGenome #33 (high priority)
• Transcript assembly pipeline (NBIS)
• Evidence alignment pipeline (?) (low priority) see [New pipeline] EvidenceAlignment #35
• PASA pipeline (IKMB) see [New pipeline] PASA #31 (low priority)
• Maker pipeline (NBIS) see [New pipeline] MAKER3 #34 (low priority)
• Abintio training pipeline (NBIS)
• Functional annotation pipeline (NBIS)
• EMBLmyGFF3 pipeline (NBIS) see [New pipeline] AnnotationToENA  #30 (high priority)

Should training genemark be included into the Abinitio training pipeline?

Which pipeline should the agat_sp_complement_annotations process go into?

Trinity will be added to the transcript assembly pipeline.

[mahesh-panchal]

The include statement can be included in config files rather than hard coded into the code. This potentially allows for profile specific include statements. This needs to be tested.

An alternative would be to use a workflow parameter to provide the module path.

[mahesh-panchal]

Marcs (IKMB) cleaned up code -> DSL2:
https://github.com/ikmb/esga

[mahesh-panchal]

Just to clarify what's in Marc's modules:

• assembly: assembly stats, and assembly filtering.
• augustus: takes input from Pasa, and trains Augustus with it.
• evm: runs evm
• pasa: includes two workflows for fast and standard. Runs Pasa and creates models
• proteins: Produces annotation hints from protein sequences with Diamondx and Exonerate.
• repeatmasker: Runs repeat masker.
• repeatmodeler: Runs repeat modeler.
• rnaseq: Maps and QC's reads to a reference with Hisat2 and fastp
• transcripts: Generate hints from Ests
• trinity: Genome guided assembly using trinity.
• fasta.nf: Some fasta utilities.
• util.nf: Some gff utilities.

If some one with a bit more knowledge could also annotate the new module suggestions with what could be reused from here, it would be appreciated.

[mahesh-panchal]

nf-core has a #denovohybrid channel where they are building a de novo genome assembly and annotation workflow.