Repository navigation
Nextflow Developer Guide
This page is for anyone who wants to change what a pipeline does rather than only how it is parameterised: adapting a shipped workflow to a biological question it was not written for, assembling an entirely new pipeline out of the wrappers already in the repository, or extending one with external tools or scripts of your own, run as ordinary Nextflow processes alongside the OmicsBox ones. Architecture describes how the repository is built; this page explains the rules that structure implies, and how to work inside them without breaking the pieces you are not touching.
These pipelines run no bioinformatics themselves. Every process wraps exactly one omicsbox
command, and the OmicsBox Engine performs the analysis in the cloud — including parallelisation,
resource allocation and input validation. The Nextflow layer only sequences the calls, wires each
step's output into the next step's input, and lays the results out on disk.
Because the computation lives elsewhere, a process definition has nearly no logic of its own, and a wrapper with no logic has nothing pipeline-specific to hard-code. That is what makes the wrappers reusable, and reuse is what the whole layout is optimised for: module files currently define processes, shared across different pipelines.
The repository separates what a tool is from what a pipeline does from how a run is configured, and each layer is allowed to know only about the layer below it. Most mistakes when extending these pipelines are a value written into the wrong layer.
| Layer | Owns | Must never contain |
|---|---|---|
modules/<category>/<tool>.nf |
The tool's CLI surface: its inputs, its outputs, the shape of its command line | A pipeline name, a step number, or a flag value the user is meant to choose |
workflows/<workflow>/<workflow>.nf |
Step order, the DAG, the input contract and its validation | A flag value, or an output filename |
workflows/<workflow>/<workflow>.config |
Per-step flags and result folders, plus the user-facing parameters | Control flow, or anything that decides which steps run and in what order |
A module is therefore generic by construction. It receives everything pipeline-specific through two
task.ext values that the configuration supplies per step:
-
task.ext.outdir— the numbered result folder this instance writes to. -
task.ext.args— the complete flag list for this instance, pre-joined into one string.
Neither is baked into the module, which is why the same file serves every pipeline that needs the tool. The corollary is the rule that matters most when editing: a module edit changes every pipeline that imports it.
The boundary is about who chooses the value, not about flags as such. A module does set a flag
whose value its own wiring already determines — the user has no choice left to make, and leaving it
to ext.args would let a configuration contradict the DAG. FASTQC switches --provide-adapters=true
on only in the branch where an adapter file was actually wired in; MEGAHIT and CONT_REM derive
--sequencing from the read layout; TRANSDECODER and FLAIR set their --use-…/--provide-…
switches from whether the corresponding optional input arrived. What a module must not do is fix a
value the user is entitled to pick — a sensitivity, a cut-off, a database.
For the same reason, a handful of modules read a step-scoped parameter directly, when that
parameter decides the structure of the command rather than one flag's value. RSEM reads
params.rsem.gene_level because it gates both --i-transcript-to-gene-file and whether the
gene-level output exists at all; REPEATMASKER reads params.repeatmasker.search_engine because
--database-type applies to one engine only. ext.args is a flat string and cannot express either
condition. Outside those cases, a step's values belong in ext.args.
Absolute paths to OmicsBox. Every path handed to omicsbox is prefixed with \$PWD/. A
relative --local-folder is not resolved against the Nextflow task directory, so the results land
outside it — and because the CLI still exits 0, nothing reports an error until Nextflow fails the
task on a missing output. Prefixing the task directory removes the failure mode for good:
"""
mkdir -p ${outdir}
omicsbox diamond \\
--i-local-project=\$PWD/${omicsbox_project} \\
--local-folder=\$PWD/${outdir} \\
${args}
"""\$PWD is escaped so that Groovy leaves it alone and the task's own shell expands it at run time.
An unescaped ${PWD} would be interpolated by Nextflow instead and resolve to the wrong directory.
Declared outputs are the module's contract. The output: block is what the next step consumes,
and each declaration carries an emit: name that the workflow wires by. Outputs that a tool
produces only under certain flags are marked optional: true, so the step does not fail when the
flag is off:
output:
path "${task.ext.outdir}/*.bam", emit: bam_sorted
path "${task.ext.outdir}/*_SJ.out.tab", emit: splice_junctions, optional: trueEvery option a workflow exposes is already declared in its .config, annotated inline with its
accepted values, a one-clause description, the default OmicsBox applies, and any dependency on
another flag. This removes the need for an external parameter reference and for any knowledge of
OmicsBox CLI syntax — the accepted values travel with the parameter, on the same line.
That is a contract for developers too. A flag added without its annotation still works, and a user
can always fall back to omicsbox describe <tool> to find out what values it accepts — but having to
leave the file to ask the CLI is exactly the friction the template exists to remove. The annotation
is what keeps it self-sufficient:
withName: 'LOAD_FASTA' {
ext.outdir = '01_load_fasta'
ext.args = [
"--handle-double-ids=replace", // [skip, replace, add_with_new_id] Decide what to do with duplicated sequence IDs. Default: replace
"--add-description=true" // [true, false] Add everything after first whitespace in header as description. Default: true
].join(' ')
}Writing the flags as a Groovy list and joining them is what makes each one independently editable:
// comments out exactly one element and leaves the rest of the step untouched.
Working out the layer before editing is most of the work. The wrong layer is what turns a local tweak into a repository-wide change.
| Goal | Edit | What it affects |
|---|---|---|
| Change a flag value, enable or disable an option | <workflow>.config |
That step, in that pipeline |
| Change an input path, output root or chart format |
<workflow>.config (params block) |
That pipeline |
| Rename a step's result folder |
<workflow>.config (ext.outdir) |
That step's published folder |
| Add, remove or re-order steps |
<workflow>.nf and <workflow>.config
|
That pipeline's DAG |
| Change how a tool is invoked, or which outputs it declares | modules/<category>/<tool>.nf |
Every pipeline importing that module |
Most biological adaptation needs no .nf edit at all. Dump the template, edit it, and pass it with
-c; the pipeline code stays untouched and the edited file is the artifact that records how the run was parameterised. See Anatomy of the template for the editing model and Where Parameter Values Come From
for how a -c file layers over the defaults.
Two failure modes account for most broken configurations:
Warning
ext.args resolves to a single string value, so assigning it replaces the step's entire flag
list rather than merging flag by flag. Always edit the flags inside the full dumped list —
hand-writing a minimal ext.args silently drops every other flag for that step.
Important
Flags are documented individually but they are not independent. A parent switch gates its
children, mode flags are mutually exclusive, and a file in params is often ignored unless a
matching flag switches its use on. Read a flag's comment for a dependency clause before changing
it — see Respecting the dependencies between flags.
Changing a step's flags changes that task's hash, so a subsequent -resume re-runs that step and
everything downstream of it while keeping earlier steps cached. Editing a late step is therefore
cheap; editing the first one is not.
Rewiring is a .nf edit, and the .nf is the only place a pipeline's order lives. Steps are
connected by passing one process's named output into the next:
LOAD_FASTA(ch_fasta)
DIAMOND_BLAST(LOAD_FASTA.out.fasta_project)
GO_MAPPING(DIAMOND_BLAST.out.blasted_project)
GO_ANNOTATION(GO_MAPPING.out.mapped_project)Two consumers of the same output simply both read it, which is how a pipeline branches:
EGGNOG_MAPPER(ch_fasta)
INTERPROSCAN(LOAD_FASTA.out.fasta_project)Arguments are positional. A process call has no argument names: the values must be passed in
exactly the order the module declares them in its input: block. Before calling a multi-input step,
read that block and match it position by position:
// modules/transcriptomics/star.nf declares, in this order:
// path reads path fasta path annotation
STAR(TRIMMOMATIC.out.trimmed_reads, ch_fasta, ch_gff)Passing the wrong number of arguments fails immediately and says so. Passing the right number in
the wrong order often does not, because the inputs are all path — nothing about a .box file
distinguishes it from another .box file, so the run proceeds and hands each one to the wrong flag.
COMBINE_PROJECTS is the clearest case: it takes two OmicsBox projects, project1 then project2,
and its --main-project flag decides which of the two wins wherever their annotations conflict.
Swap the two arguments and the pipeline still completes — with the merge resolved the wrong way
round, in a result that looks entirely plausible.
Adding a step to an existing pipeline takes three coordinated edits, and all three are required:
-
includethe module at the top of the.nf. - Call it in the workflow body, passing the upstream output it consumes.
- Add a
withName:block to the.configwith itsext.outdirandext.args.
Removing a step means re-pointing every consumer of its output at a suitable upstream output
before deleting the call, then deleting its withName: block. A step whose output nothing consumes
still executes.
Reusing one tool twice does not require a second module. Import it under an alias, and each alias then takes its own configuration entry:
include { FASTQC as FASTQC_RAW } from '../../modules/general_tools/fastqc.nf'
include { TRIMMOMATIC } from '../../modules/general_tools/trimmomatic.nf'
include { FASTQC as FASTQC_POST } from '../../modules/general_tools/fastqc.nf'Keep ext.outdir numbering aligned with the new topological order when steps move. The numbers are
what make the results tree readable; they are not read by any code, so nothing warns when they drift
out of sequence.
A new pipeline is a new directory and nothing else. No shared index, registry or root file has to be touched, which is what makes the modules plug-and-play: assembling existing wrappers in a new order is usually the entire job.
Check what already exists before writing anything. Wrappers are grouped by analysis domain, and the file name is the lower_snake_case form of the operation, while the process name is UPPERCASE:
| Category | Examples |
|---|---|
functional_analysis/ |
load_fasta.nf, diamond_blast.nf, go_mapping.nf, ips.nf, go_slim.nf
|
transcriptomics/ |
star.nf, htseq.nf, rsem.nf, edger.nf, trinity.nf, minimap2.nf
|
genome_analysis/ |
spades.nf, flye.nf, busco.nf, quast.nf, augustus.nf, pilon.nf
|
genetic_variation/ |
bcftools.nf, beagle.nf, gwas.nf, variant_filtering.nf
|
metagenomics/ |
megahit.nf, prodigal.nf, eggnog_mapper.nf, pfam_scan.nf
|
general_tools/ |
fastqc.nf, trimmomatic.nf, longqc.nf
|
export_annotations/ |
export_genesets.nf |
utilities/ |
combine_projects.nf |
Read the input: and output: blocks of each candidate — those two blocks are all you have to work
with. An emit: name is what a downstream step gets wired to, and the output glob decides what that
name actually carries; together they determine what can be connected to what.
nextflow/workflows/my_pipeline/
├── my_pipeline.nf # entry point
├── my_pipeline.config # user-facing template
└── nextflow.config # loader
Two lines, and the order matters: the root is included first so the workflow's own template overrides it wherever both declare the same key.
includeConfig '../../nextflow.config'
includeConfig './my_pipeline.config'Import the modules, then wire them. Every workflow follows the same four-part body.
Import the modules.
include { LOAD_FASTA } from '../../modules/functional_analysis/load_fasta.nf'
include { DIAMOND_BLAST } from '../../modules/functional_analysis/diamond_blast.nf'
include { GO_MAPPING } from '../../modules/functional_analysis/go_mapping.nf'Support --dump_config. Every pipeline implements it the same way, copying its own template to
the launch directory and exiting 0 before any process is submitted, so nothing reaches the cloud.
moduleDir resolves to the workflow's own directory, which is what makes the block work identically
for a local clone and for a run straight from GitHub:
if (params.dump_config) {
def sourceConfig = file("${moduleDir}/my_pipeline.config")
def targetConfig = file("./my_pipeline.config")
if (sourceConfig.exists()) {
sourceConfig.copyTo(targetConfig)
log.info " File generated at: ./my_pipeline.config"
} else {
log.error " [ERROR] Could not find the internal template at: ${sourceConfig}"
}
exit 0
}Validate the input contract. Guard every required parameter, and name the parameter in the message. This is the cheapest error in the whole system: it fails locally in seconds instead of inside a cloud job, and the shipped pipelines validate coupled parameters too, not just presence:
if (!params.input_fasta) {
exit 1, "ERROR: You must provide a FASTA input file via --input_fasta=/path/to/sequences.fasta"
}
if (params.input_paired_end && params.input_single_end) {
exit 1, "ERROR: Provide either --input_paired_end or --input_single_end, not both."
}Build the channels. Three conventions cover almost every case, and each exists for a specific reason:
// Required input. checkIfExists fails the run immediately on a bad path.
def ch_fasta = channel.fromPath(params.input_fasta, checkIfExists: true)
// Many read files, ONE OmicsBox task: OmicsBox parallelises over samples internally,
// so .collect() prevents Nextflow from fanning out one cloud job per file.
def ch_reads = channel.fromPath(params.input_paired_end, checkIfExists: true).collect()
// Optional file input. channel.value([]) is the "not provided" placeholder the
// module tests for before it emits the corresponding flag.
def ch_goslim_obo = params.goslim.obo_file
? channel.fromPath(params.goslim.obo_file, checkIfExists: true).first()
: channel.value([])A module receiving an optional input tests for the empty List and omits the flag entirely:
def obo_flag = (!(obo_file instanceof List) || !obo_file.isEmpty())
? "--i-go-slim-obo-file=\$PWD/${obo_file}"
: ""Wire the DAG. Pass each named output into the next step, keeping every call's arguments in the order the module declares its inputs, as in Changing the DAG above.
One params block, then one withName: entry per step in topological order. Number the ext.outdir
values in that same order, and annotate every flag inline with its accepted values, description and
default — that annotation is the pipeline's entire user documentation.
params {
local_folder = './results'
input_fasta = null // Path to the input multi-FASTA file
chart_format = 'box' // Output chart format for all chart processes: [box, png, pdf, csv]
}
process {
withName: 'LOAD_FASTA' {
ext.outdir = '01_load_fasta'
ext.args = [
"--handle-double-ids=replace" // [skip, replace, add_with_new_id] Decide what to do with duplicated sequence IDs. Default: replace
].join(' ')
}
}A step that needs no flags still takes an entry, written as ext.args = [].join(' '). Do not omit
the block: ext.outdir is what places the step in the results tree.
Two checks are safe to run as often as needed, because neither submits a process:
nextflow config nextflow/workflows/my_pipeline/nextflow.config
nextflow run nextflow/workflows/my_pipeline/my_pipeline.nf --dump_configThe first resolves the configuration layers and prints the merged result, which catches a missing
withName: block, a typo in a process name, or a broken includeConfig path. The second proves the
--dump_config block works and produces the template a user will actually edit.
A pipeline can also include steps of your own. Nothing in the layout requires a step to be an
omicsbox call: a process running an external tool or a script you wrote is wired into the DAG
exactly like any other, reading its input from an upstream emit: and handing its own output to the
next step.
Keep the same two task.ext values and the step stays configurable from the template like the rest of
the pipeline:
process TRANSLATE_ORFS {
input:
path transcripts // Nucleotide FASTA from an upstream step
output:
path "${task.ext.outdir}/proteins.faa", emit: proteins
script:
def outdir = task.ext.outdir ?: task.process.toLowerCase()
def args = task.ext.args ?: ''
"""
mkdir -p ${outdir}
translate_orfs.py --input ${transcripts} --output ${outdir}/proteins.faa ${args}
"""
}Everything else behaves identically. The root publishDir policy applies to every process, so a
custom step's declared outputs are published into the same results/<session-id>/<ext.outdir>/ tree,
and -resume caches it on the same terms as an OmicsBox step.
These two concerns never arise for an OmicsBox step, and they are the reason this kind of step deserves its own attention.
An omicsbox process does not compute anything locally — it submits a cloud analysis and waits. It
needs no CPU or memory worth reserving, and the repository declares none: there is not a single
cpus, memory, or time directive anywhere in nextflow/. A tool of your own is the
first step in the pipeline that actually consumes the machine it runs on, so it is also the first
one for which those directives mean anything.
Declare them in the step's own withName: block, next to ext.outdir and ext.args, so the step
stays configurable from the template like every other:
withName: 'TRANSLATE_ORFS' {
ext.outdir = '05_translate_orfs'
ext.args = [].join(' ')
cpus = 4
memory = '8 GB'
}The tool itself has to be available where the process runs. By default that means the host: the
command must be on PATH, exactly as omicsbox is. Nextflow can also provision the step itself
instead — from a Conda environment, a Spack package, an Environment Modules module or a container
image — by declaring the matching directive on the process. See Nextflow's
process directives reference for the options and what each one requires.
- Architecture — the structure these rules follow from: the root configuration, the reusable module wrappers, the three files of a workflow directory, and how a step is assembled from all three at run time
- Execution & Configuration — the same pipelines from the user's side: dumping and editing the template, which layer a final parameter value comes from, launching and resuming a run, and where the results are written
- Available Workflows — the catalogue of published pipelines, each with its workflow ID and what it does, linking to per-pipeline step graphs and output layouts
- Snakemake Developer Guide — the equivalent guide for the Snakemake port
-
Nextflow
-
Snakemake