Skip to content

Nextflow Architecture

rlopez edited this page Jul 29, 2026 · 2 revisions

The Nextflow side of this repository is built on a deliberate separation between what a tool is, what a pipeline does, and how a run is configured. Each of the three lives in its own place and none of them reaches into the others. This page explains that structure, why it is shaped this way, and how the pieces compose at run time.


The orchestration model

These pipelines do not perform bioinformatics analyses themselves. Every process wraps exactly one omicsbox command, and the OmicsBox Engine runs the actual analysis in the cloud, handling parallelisation, resource allocation and input validation internally. What the Nextflow layer contributes is the part OmicsBox does not do: sequencing the steps, wiring each step's output into the next step's input, and laying the results out on disk.

This single fact explains most of the architectural decisions below. Because the heavy lifting happens elsewhere, a process definition has almost no logic of its own — it is a thin, generic wrapper around one CLI call. And because the wrapper is generic, it can be shared by every pipeline that needs that tool.


Repository layout

Everything for this engine lives under nextflow/ in the published repository:

nextflow/
├── nextflow.config               # Root configuration: shared params, publishDir policy
├── modules/                      # Decoupled tool wrappers — one process per file
│   ├── functional_analysis/
│   ├── transcriptomics/
│   ├── genome_analysis/
│   ├── genetic_variation/
│   ├── metagenomics/
│   ├── general_tools/
│   ├── export_annotations/
│   └── utilities/
└── workflows/                    # Encapsulated execution points — one directory per pipeline
    └── <workflow>/
        ├── <workflow>.nf         # Entry point: includes modules and wires the DAG
        ├── <workflow>.config     # User-facing self-documenting template
        └── nextflow.config       # Loader: stacks the root config under the template
flowchart TD
    RC["nextflow.config<br/>root defaults"]
    TPL["workflow template<br/>.config"]
    LOADER["workflow loader<br/>nextflow.config"]
    MODS["modules/<br/>process definitions"]
    WF["workflow entry<br/>.nf"]
    CLI["omicsbox CLI"]

    RC --> LOADER
    TPL --> LOADER
    LOADER --> WF
    MODS --> WF
    WF --> CLI

    classDef config fill:#e8f4fd,stroke:#1f77b4,stroke-width:1px;
    classDef code fill:#fff3e0,stroke:#e67e22,stroke-width:1px;
    class RC,TPL,LOADER config;
    class MODS,WF,CLI code;
Loading

nextflow.config at the root

The root configuration is the only place where settings that apply to every process of every pipeline are declared. It is short by design:

docker {
    enabled    = false
    runOptions = '-u $(id -u):$(id -g)'
}

params {
    dump_config = false             // Flag to dump the configuration parameters to a file
    omicsboxImage = null            // No OmicsBox container image published yet; the host CLI is used
    local_folder  = './results'     // Base output directory for all workflow results
}

process {
    container = params.omicsboxImage
    publishDir = [
        path:      { "${params.local_folder}/${workflow.sessionId}" },
        mode:      'link',
        overwrite: true
    ]
}

Three cross-cutting concerns are settled here once and never repeated:

  • The execution environment — docker.enabled = false, so the host omicsbox installation is used. params.omicsboxImage exists as the hook for a containerised run but is null, since no OmicsBox image is published yet.
  • The publishing policy — where results appear and how they get there, applied uniformly to every process. See Result Handling for what the policy means in practice.
  • The parameters every pipeline shares — dump_config and local_folder.

No individual module or workflow re-declares any of this. A change to the publishing policy is a one-line change that propagates to all 10 pipelines.


modules/ — decoupled tool wrappers

A module is one file containing one process that wraps one omicsbox command. Modules are grouped into category directories by analysis domain, and the file name is the lower_snake_case form of the operation it wraps, while the process name is UPPERCASE.

The decisive property is that a module contains no pipeline-specific information. It does not know which pipeline is calling it, which step number it occupies, or which flags it should pass. All of that arrives from outside, through two task.ext values:

process DIAMOND_BLAST {

    input:
    path omicsbox_project   // OmicsBox Sequence Project (.box)

    output:
    path "${task.ext.outdir}/*output_project*.box", emit: blasted_project   // BLAST-annotated project

    script:
    def outdir = task.ext.outdir ?: task.process.toLowerCase()
    def args = task.ext.args ?: ''


    """
    mkdir -p ${outdir}
    omicsbox diamond \\
        --i-local-project=\$PWD/${omicsbox_project} \\
        --local-folder=\$PWD/${outdir} \\
        ${args}
    """
}
  • task.ext.outdir — the numbered result folder this instance writes to, supplied per pipeline. The ?: fallback keeps the module runnable even if a configuration omits it.
  • task.ext.args — the complete flag list for this instance, supplied per pipeline as a single pre-joined string.

Because neither is baked in, the same file serves every pipeline that needs the tool.


workflows/ — encapsulated execution points

Each pipeline is a self-contained directory holding exactly three files. Nothing else in the repository needs to change to add one, and nothing outside the directory describes it.

File Role
<workflow>.nf The entry point. Imports the modules it needs and wires the DAG.
<workflow>.config The user-facing template: the params block and one withName: entry per step.
nextflow.config The loader. Stacks the root configuration underneath the template.

The entry point

The .nf file is the only place where the order of a pipeline lives. It imports processes and connects them by passing one step's named output into the next:

include { LOAD_FASTA                                       } from '../../modules/functional_analysis/load_fasta.nf'
include { DIAMOND_BLAST                                    } from '../../modules/functional_analysis/diamond_blast.nf'

Where a pipeline needs the same tool twice, it imports the module twice under different aliases, and each alias then receives its own configuration entry. Some workflows do exactly this with FastQC, to run quality control before and after trimming:

include { FASTQC as FASTQC_RAW   } from '../../modules/general_tools/fastqc.nf'
include { TRIMMOMATIC            } from '../../modules/general_tools/trimmomatic.nf'
include { FASTQC as FASTQC_POST  } from '../../modules/general_tools/fastqc.nf'

The entry point also owns the pipeline's input contract. It validates required parameters before any process is submitted and aborts with a message naming the parameter, so a misconfigured run fails locally in seconds instead of inside a cloud job. It is also where --dump_config is implemented, copying the template to the launch directory and exiting.

The template

<workflow>.config is the pipeline's public surface: a params block for the workflow-level settings and one withName: entry per step supplying that step's ext.outdir and ext.args. This is where all pipeline-specific knowledge about a tool is concentrated. Its structure and editing model are covered in Anatomy of the template.

The loader

The third file exists only to stack the configuration layers in the right order:

includeConfig '../../nextflow.config'
includeConfig './functional_annotation.config'

The root is included first, so the workflow's own template overrides it wherever both declare the same key. Anything the user passes on the command line then layers on top of that — see The configuration layers.


How a step comes together at run time

The three layers meet only at the moment a task is created. For a single step:

  1. The loader resolves the configuration, stacking the template over the root defaults.
  2. The entry point calls the process with an input channel, under a name that may be an alias.
  3. Nextflow matches that name against a withName: entry in the template and attaches its ext.outdir and ext.args to the task.
  4. The module reads those two values, builds the omicsbox command, and runs it in the task's own work/ directory.
  5. The root publishDir policy links the declared outputs into the results tree.

What passes between the layers is deliberately small. Into a module go its ext.outdir and ext.args; out of it comes a named output channel. Modules also read params directly, but only for values that belong to the run as a whole rather than to one step's flag list — params.chart_format, which every chart step formats its output with, or the read-layout parameters a step needs in order to build its input flags correctly.


Related pages

  • Execution & Configuration — the user-facing side of this structure: dumping and editing a workflow's template, which layer a final parameter value comes from, the commands that launch and resume a run, and where the results are written
  • Available Workflows — the catalogue of published pipelines, each with its workflow ID and what it does, linking to per-pipeline step graphs and output layouts
  • Developer Guide — working inside this structure: which layer a change belongs to, how to assemble a new pipeline from existing modules, and how to add steps of your own
  • Snakemake Architecture — the equivalent structure for the Snakemake port

Clone this wiki locally