Repository navigation
Nextflow Architecture
The Nextflow side of this repository is built on a deliberate separation between what a tool is, what a pipeline does, and how a run is configured. Each of the three lives in its own place and none of them reaches into the others. This page explains that structure, why it is shaped this way, and how the pieces compose at run time.
These pipelines do not perform bioinformatics analyses themselves. Every process wraps exactly one
omicsbox command, and the OmicsBox Engine runs the actual analysis in the cloud, handling
parallelisation, resource allocation and input validation internally. What the Nextflow layer
contributes is the part OmicsBox does not do: sequencing the steps, wiring each step's output into
the next step's input, and laying the results out on disk.
This single fact explains most of the architectural decisions below. Because the heavy lifting happens elsewhere, a process definition has almost no logic of its own — it is a thin, generic wrapper around one CLI call. And because the wrapper is generic, it can be shared by every pipeline that needs that tool.
Everything for this engine lives under nextflow/ in the published repository:
nextflow/
├── nextflow.config # Root configuration: shared params, publishDir policy
├── modules/ # Decoupled tool wrappers — one process per file
│ ├── functional_analysis/
│ ├── transcriptomics/
│ ├── genome_analysis/
│ ├── genetic_variation/
│ ├── metagenomics/
│ ├── general_tools/
│ ├── export_annotations/
│ └── utilities/
└── workflows/ # Encapsulated execution points — one directory per pipeline
└── <workflow>/
├── <workflow>.nf # Entry point: includes modules and wires the DAG
├── <workflow>.config # User-facing self-documenting template
└── nextflow.config # Loader: stacks the root config under the template
flowchart TD
RC["nextflow.config<br/>root defaults"]
TPL["workflow template<br/>.config"]
LOADER["workflow loader<br/>nextflow.config"]
MODS["modules/<br/>process definitions"]
WF["workflow entry<br/>.nf"]
CLI["omicsbox CLI"]
RC --> LOADER
TPL --> LOADER
LOADER --> WF
MODS --> WF
WF --> CLI
classDef config fill:#e8f4fd,stroke:#1f77b4,stroke-width:1px;
classDef code fill:#fff3e0,stroke:#e67e22,stroke-width:1px;
class RC,TPL,LOADER config;
class MODS,WF,CLI code;
The root configuration is the only place where settings that apply to every process of every pipeline are declared. It is short by design:
docker {
enabled = false
runOptions = '-u $(id -u):$(id -g)'
}
params {
dump_config = false // Flag to dump the configuration parameters to a file
omicsboxImage = null // No OmicsBox container image published yet; the host CLI is used
local_folder = './results' // Base output directory for all workflow results
}
process {
container = params.omicsboxImage
publishDir = [
path: { "${params.local_folder}/${workflow.sessionId}" },
mode: 'link',
overwrite: true
]
}Three cross-cutting concerns are settled here once and never repeated:
-
The execution environment —
docker.enabled = false, so the hostomicsboxinstallation is used.params.omicsboxImageexists as the hook for a containerised run but isnull, since no OmicsBox image is published yet. - The publishing policy — where results appear and how they get there, applied uniformly to every process. See Result Handling for what the policy means in practice.
-
The parameters every pipeline shares —
dump_configandlocal_folder.
No individual module or workflow re-declares any of this. A change to the publishing policy is a one-line change that propagates to all 10 pipelines.
A module is one file containing one process that wraps one omicsbox command. Modules are
grouped into category directories by analysis domain, and the file name is the lower_snake_case form
of the operation it wraps, while the process name is UPPERCASE.
The decisive property is that a module contains no pipeline-specific information. It does not
know which pipeline is calling it, which step number it occupies, or which flags it should pass. All
of that arrives from outside, through two task.ext values:
process DIAMOND_BLAST {
input:
path omicsbox_project // OmicsBox Sequence Project (.box)
output:
path "${task.ext.outdir}/*output_project*.box", emit: blasted_project // BLAST-annotated project
script:
def outdir = task.ext.outdir ?: task.process.toLowerCase()
def args = task.ext.args ?: ''
"""
mkdir -p ${outdir}
omicsbox diamond \\
--i-local-project=\$PWD/${omicsbox_project} \\
--local-folder=\$PWD/${outdir} \\
${args}
"""
}-
task.ext.outdir— the numbered result folder this instance writes to, supplied per pipeline. The?:fallback keeps the module runnable even if a configuration omits it. -
task.ext.args— the complete flag list for this instance, supplied per pipeline as a single pre-joined string.
Because neither is baked in, the same file serves every pipeline that needs the tool.
Each pipeline is a self-contained directory holding exactly three files. Nothing else in the repository needs to change to add one, and nothing outside the directory describes it.
| File | Role |
|---|---|
<workflow>.nf |
The entry point. Imports the modules it needs and wires the DAG. |
<workflow>.config |
The user-facing template: the params block and one withName: entry per step. |
nextflow.config |
The loader. Stacks the root configuration underneath the template. |
The .nf file is the only place where the order of a pipeline lives. It imports processes and
connects them by passing one step's named output into the next:
include { LOAD_FASTA } from '../../modules/functional_analysis/load_fasta.nf'
include { DIAMOND_BLAST } from '../../modules/functional_analysis/diamond_blast.nf'Where a pipeline needs the same tool twice, it imports the module twice under different aliases, and each alias then receives its own configuration entry. Some workflows do exactly this with FastQC, to run quality control before and after trimming:
include { FASTQC as FASTQC_RAW } from '../../modules/general_tools/fastqc.nf'
include { TRIMMOMATIC } from '../../modules/general_tools/trimmomatic.nf'
include { FASTQC as FASTQC_POST } from '../../modules/general_tools/fastqc.nf'The entry point also owns the pipeline's input contract. It validates required parameters before any
process is submitted and aborts with a message naming the parameter, so a misconfigured run fails
locally in seconds instead of inside a cloud job. It is also where --dump_config is
implemented, copying the template to the launch directory and exiting.
<workflow>.config is the pipeline's public surface: a params block for the workflow-level
settings and one withName: entry per step supplying that step's ext.outdir and ext.args. This
is where all pipeline-specific knowledge about a tool is concentrated. Its structure and editing model
are covered in Anatomy of the template.
The third file exists only to stack the configuration layers in the right order:
includeConfig '../../nextflow.config'
includeConfig './functional_annotation.config'The root is included first, so the workflow's own template overrides it wherever both declare the same key. Anything the user passes on the command line then layers on top of that — see The configuration layers.
The three layers meet only at the moment a task is created. For a single step:
- The loader resolves the configuration, stacking the template over the root defaults.
- The entry point calls the process with an input channel, under a name that may be an alias.
- Nextflow matches that name against a
withName:entry in the template and attaches itsext.outdirandext.argsto the task. - The module reads those two values, builds the
omicsboxcommand, and runs it in the task's ownwork/directory. - The root
publishDirpolicy links the declared outputs into the results tree.
What passes between the layers is deliberately small. Into a module go its ext.outdir and
ext.args; out of it comes a named output channel. Modules also read params directly, but only for
values that belong to the run as a whole rather than to one step's flag list — params.chart_format,
which every chart step formats its output with, or the read-layout parameters a step needs in order
to build its input flags correctly.
- Execution & Configuration — the user-facing side of this structure: dumping and editing a workflow's template, which layer a final parameter value comes from, the commands that launch and resume a run, and where the results are written
- Available Workflows — the catalogue of published pipelines, each with its workflow ID and what it does, linking to per-pipeline step graphs and output layouts
- Developer Guide — working inside this structure: which layer a change belongs to, how to assemble a new pipeline from existing modules, and how to add steps of your own
- Snakemake Architecture — the equivalent structure for the Snakemake port
-
Nextflow
-
Snakemake