Github hosting of DSL2 includes?

[photocyte]

Hi there,

I have a interproscan_parallel.nf nextflow script (using nextflow 23.04.1), that pulls processes from a variety of other places using the DSL2 syntax:

nextflow.enable.dsl=2

include { download_pfam_A ; prepare_pfam_A ; pfam_run ; hmmpress_generic } from '/home/tfallon/databases/pfam/pfam.nf'
include { generate_svg_colors ; svg_2_pdf ; svg_utils_merge ; split_gff_by_seqid ; DNA_features_viewer } from '/home/tfallon/source/nextflow/dna_features_viewer/dna_features_viewer.nf'
include { detab ; debar ; decolon ; dummy_publish_path } from '/home/tfallon/source/nextflow/utility/utility.nf'

Within it's nextflow.config, it also pulls the nextflow.config from various other places (these typically supply the per-process Docker image URL, so it can be executed in a Singularity/Apptainer container, if it isn't using Conda):

conda.enabled = true

includeConfig '/home/tfallon/source/nextflow/dna_features_viewer/nextflow.config'
includeConfig '/home/tfallon/Scripts/nextflow/singularity.nf.config'

manifest {
 mainScript = 'interproscan_parallel.nf'
 description = 'A wrapper pipeline that runs, simplifies, and plots an interproscan analysis of a polypetide fasta'
 version = '1.0.0'
}

dag.file = "results/${manifest.mainScript}__dag.html"
dag {
    enabled = true
    overwrite = true
}

report.file = "results/${manifest.mainScript}__report.html"
report {
    enabled = true
    overwrite = true
}

This all works well on machine with a fully populated /home/tfallon/source/nextflow/ with lots of subpipelines & their config files.

But I've been trying to make it externally reproducible. The first step is to put the interproscan_parallel.nf & its nextflow.config into a Github repo. But is there a way to also refer to the DSL2 includes from a github repo?

I didn't see anything about that in the current documentation, and this nf-core thread (nf-core/modules#8) suggests it might be much more complicated than I expected / not a core nextflow feature?

[mribeirodantas]

You can have relative paths for that, @photocyte. That's what nf-core pipelines do. Check the nf-core/rnaseq pipeline, for example. In the nextflow.config file, it includes multiple configuration files through relative paths that work in any machine that clones this GitHub repository. The same thing for modules (check here). That's what you're looking for, right?

[photocyte]

Hi there @mribeirodantas , not exactly. I'd like to have the depency sub-workflows live in separate github repositories. But I agree if all the dependencies are explicitly shipped in the same monolithic github repository (like shown here), one should use relative paths.

Put another way, I'd like to know if there is a current best practice way to specify & use Nextflow sub-workflow dependencies, via them living in separate github repositories.

Ideally, this dependency resolution would happen with a single command, i.e.:
nextflow run https://github.com/photocyte/interproscan_parallel/ -resume --fasta polypeptides_to_annotate.pep.fa

Perhaps you're suggesting there is a way to do it with relative paths using ../ & nextflow pull? Using the below example:
interproscan_parallel.nf:

nextflow.enable.dsl=2
include { download_pfam_A ; prepare_pfam_A ; pfam_run ; hmmpress_generic } from '../pfam/'
include { generate_svg_colors ; svg_2_pdf ; svg_utils_merge ; split_gff_by_seqid ; DNA_features_viewer } from '../dna_features_viewer/'
include { detab ; debar ; decolon ; dummy_publish_path } from '../nf_utility/'

nextflow pull https://github.com/photocyte/interproscan_parallel/
nextflow pull https://github.com/photocyte/dna_features_viewer/
nextflow pull https://github.com/photocyte/nf_utility/
nextflow run https://github.com/photocyte/interproscan_parallel/ -resume --fasta polypeptides_to_annotate.pep.fa

[mribeirodantas]

Perhaps you're...

Yeah, something like that

[bentsherman]

It is trivial if you put all of the modules you want to include in the same git repo as your main script. And config files can be included from anywhere, so that's no problem. But if you want to include modules from other sources, that is currently not supported.

You can use the nf-core modules command to install and update standard nf-core modules in your pipeline repo. See #4112 for the current discussion about adding this support into Nextflow itself.