Skip to content

Nextflow Execution & Configuration

rlopez edited this page Jul 31, 2026 · 4 revisions

This page covers everything shared by the Nextflow pipelines in this repository: how they are configured, how a value set in one place overrides a value set in another, how to launch a run, how to resume one, and where the results end up. The individual pipeline pages document what each workflow does; this page documents how to drive any of them.


How a Workflow Is Configured

The self-documenting template

Every pipeline ships a single user-facing configuration file, nextflow/workflows/<workflow>/<workflow>.config. It is the only file a user is expected to edit, and it is designed so that no external parameter reference is needed: every option a workflow exposes is already declared in it, grouped, and annotated inline with its accepted values and its default.

This is a deliberate strategy. There is no parameter table to consult and no OmicsBox CLI syntax to memorise — the accepted values travel with the parameter, in the same file, on the same line.

Generating your own copy

Every workflow supports --dump_config. It copies the packaged template into the current directory and exits with status 0 before any process is submitted, so nothing reaches the cloud:

nextflow run nextflow/workflows/<workflow>/<workflow>.nf --dump_config

The dumped file is a complete, valid, standalone configuration — not a stub. Edit that copy and keep it; it is the artifact that records how a run was parameterised.

Anatomy of the template

A template has two blocks: a params block and a process block.

The params block

Its first section holds the workflow-level settings — input paths, the output location, and formatting options:

params {

    // ---------------------------------------------------
    // GLOBAL PIPELINE PARAMETERS
    // ---------------------------------------------------
    input_single_end = null         // Glob for single-end reads (e.g. 'data/*.fastq.gz')
    input_paired_end = null         // Glob for paired-end reads (e.g. 'data/*_{1,2}.fastq.gz')
    input_fasta      = null         // REQUIRED: reference genome FASTA (SHARED: used by BWA, BCFtools and Variant Annotation). Ensembl recommended.
    local_folder     = './results'  // Base output directory for all workflow results
    chart_format     = 'box'        // Output chart format: [box, png, pdf, csv]

Most workflows then declare a second section of step-scoped parameters, grouped into nested blocks named after the step that consumes them — typically the optional input files a step accepts and the mode choices it needs:

    bcftools {
        group_experiment = null  // Optional: tab-delimited sample->group file. To use it you MUST also set --use-groups=true below.
    }

    gwas {
        input_pheno      = null  // REQUIRED: tab-separated traits table (1st column = sample names matching the VCF; traits in following columns; header required)
        kinship          = null  // Optional: precomputed kinship matrix file. To use it you MUST also set --use-kinship=true below.
        covariate_matrix = null  // Optional: covariate-matrix metadata file (requires --use-covariate-matrix=true).
    }

Several of these are coupled to a flag in the step's own list: a file supplied here is ignored unless the matching flag also switches its use on. See Respecting the dependencies between flags below.

Every path you set in this params block — inputs as well as local_folder — may be given as an absolute path or as a path relative to the directory you launch the workflow from; both are handled identically.

The process block

This block carries one withName: entry per pipeline step. Each entry sets an ext.outdir — the numbered result folder that step publishes to — and an ext.args, the list of flags handed to the underlying omicsbox command:

    withName: 'GO_MAPPING' {
        ext.outdir = '03_go_mapping'
        ext.args = [
            "--latest-database-version=true", // [true, false] Use the latest GOA database version. Default: true
            //"--goa-version=\"\"",            // [string] Specify a particular GOA version if latest is set to false.
        ].join(' ')
    }

ext.args is a list of flags joined into one string, which gives the "checkbox" editing model: commenting out a line disables exactly that one flag, and uncommenting one enables it. Nothing else in the step changes. The values shown above are the shipped defaults — they are meant to be edited.

The comment beside each flag is its documentation, and carries up to four pieces of information: the accepted values in square brackets, a one-clause description of what the flag controls, the default OmicsBox applies when the flag is absent — so a commented-out line leaves the flag at its default rather than switching it off — and, where one exists, the dependency that decides whether the flag is read at all.

Warning

Because ext.args resolves to a single string value, assigning it replaces the step's entire flag list — it does not merge flag by flag. Always edit the flags inside the full dumped list. Hand-writing a minimal ext.args with one entry silently drops every other flag for that step.

Respecting the dependencies between flags

Important

Flags are documented individually but they are not independent. Commenting, uncommenting or flipping a true/false in isolation is the most common way to break an otherwise valid configuration. Before changing a flag, read its comment for a dependency clause and check the flags it names.

Four patterns of dependency appear in the templates:

  • Parent switch and its children. A group of flags is only read when the flag that enables the feature is true — every (only if remove-adapters=true) in a trimming step, for example.
  • Mutually exclusive modes. Choosing one value of a mode flag disables the flags belonging to the other mode, so switching mode means editing more than one line.
  • Coupling between params and ext.args. A file supplied in the params block is frequently ignored unless a matching flag switches its use on — group_experiment needs --use-groups=true, kinship needs --use-kinship=true, covariate_matrix needs --use-covariate-matrix=true.
  • Constraints from the input layout or from a later step. Some flags are valid only for one read layout and cause OmicsBox to error otherwise; others are safe in themselves but change an output in a way a downstream step depends on, such as a filename prefix that renames samples and must then match the experimental design file.

The templates state these couplings where they exist. The GO-Slim step shows all of the first three at once, and the comment spells out every line the change touches:

    withName: 'GO_SLIM' {
        ext.outdir = '13_go_slim'
        ext.args = [
            "--option=web",              // [web, custom] Source for the GO-Slim OBO file. Default: web.
                                         // If set to 'custom': COMMENT OUT --go-slim-web-file below (custom disables it) and
                                         // provide your OBO file via params.goslim.obo_file.
            "--go-slim-web-file=generic" // [generic, plant, candida, yeast, protein, pombe, aspergillus, metagenomics, agr, chembl, drosophila, mouse, prokaryote, synapse, virus, ...] GO-Slim subset. Default: generic. (Only when --option=web.)
        ].join(' ')
    }

Switching that one value from web to custom therefore requires three coordinated edits: change the mode, comment out the flag the new mode disables, and set the file in the params block.


Where Parameter Values Come From

The configuration layers

Each workflow directory contains a small loader, nextflow/workflows/<workflow>/nextflow.config, whose whole job is to stack the shared configuration underneath the workflow-specific one:

includeConfig '../../nextflow.config'
includeConfig './functional_annotation.config'

Order matters: the root configuration is included first, so the workflow's own template overrides it wherever the two declare the same key. On top of that sit the two command-line mechanisms. From lowest to highest precedence:

# Layer What it provides How it is applied
1 nextflow/nextflow.config dump_config, local_folder, and the global publishDir policy includeConfig from the loader
2 nextflow/workflows/<workflow>/<workflow>.config the workflow's own params and every step's ext.outdir / ext.args includeConfig from the loader
3 -c my.config your edited copy of the template command line
4 --<param>=<value> one-off overrides for a single run command line
flowchart TD
    L1["nextflow.config<br/>(root defaults)"]
    L2["workflow.config<br/>(packaged template)"]
    L3["-c your.config"]
    L4["--param=value"]
    R["Value used<br/>by the run"]

    L1 --> L2 --> L3 --> L4 --> R
Loading

What a -c file overrides

-c layers a complete configuration over the packaged one, so it can override any scope: the params block, the process block, each individual withName: entry with its ext.outdir and ext.args. The dumped template is delivered with -c because it is exactly that — a full configuration.

Note

-params-file is not a substitute for -c. Nextflow also accepts a JSON or YAML file of parameters via -params-file, and it sits above -c in the precedence order — but it carries params and nothing else. The dumped template is not only a params block: it also holds the process block with every step's ext.outdir and ext.args, which a parameters file cannot express. Passing a template's parameters through -params-file while omitting -c therefore leaves every step with an empty flag list.

Overriding on the command line

These workflows take no positional arguments. Every input is a named parameter, and any of them can be supplied — or overridden — directly on the command line, which is the quickest way to point a configured pipeline at a different dataset without editing a file:

nextflow run nextflow/workflows/gene_level_analysis/gene_level_analysis.nf \
    -c gene_level_analysis.config \
    --input_paired_end='data/*_{1,2}.fastq.gz' \
    --input_fasta=/data/reference/genome.fa \
    --input_gff=/data/reference/genome.gtf \
    --experimental_design=/data/design.tsv \
    --local_folder=/scratch/run_042/results

Points to keep straight:

  • Two dashes for pipeline parameters, one dash for Nextflow options. --input_fasta sets params.input_fasta; -c, -resume, -r and -main-script are options for Nextflow itself. A single dash on a pipeline parameter is silently interpreted as a Nextflow option and will not reach the workflow.
  • Quote your globs. Write --input_paired_end='data/*_{1,2}.fastq.gz' with the quotes. Without them the shell rewrites {1,2} into two separate arguments before Nextflow ever sees the command, and only the first is read as the parameter. Quoting passes the pattern through untouched, which is what the workflow needs — it matches the files itself.
  • Nested parameters use dotted names. A step-scoped parameter such as params.trimmomatic.adapters is set with --trimmomatic.adapters=/path/to/adapters.fa.
  • CLI overrides win over everything. A value passed with -- beats the same key in your -c file, which is what makes it useful for sweeping one setting across otherwise identical runs.

Running a Workflow

Two ways to run any of these workflows are supported. Option A is recommended when you want to inspect or modify the pipeline code; Option B runs the published version straight from GitHub with no manual clone. Replace <workflow> with the directory name of the pipeline you are running — see Available Workflows, where each page documents these same steps for its own pipeline.

Option A — Local Clone

git clone https://github.com/BioBam/omicsbox-engine-workflows.git
cd omicsbox-engine-workflows

Step 1 — Dump the configuration file

nextflow run nextflow/workflows/<workflow>/<workflow>.nf --dump_config

This copies the self-documenting template to ./<workflow>.config in your current directory and exits. No OmicsBox job is launched.

Step 2 — Customise it

Open the generated <workflow>.config and edit it as described in Anatomy of the template: set your input paths and output folder, then fill in or comment/uncomment the step-specific flags you want to change. Save the file.

Step 3 — Run the workflow

nextflow run nextflow/workflows/<workflow>/<workflow>.nf \
    -c <workflow>.config \
    -resume

Option B — Directly from GitHub

No clone required — the engine fetches the workflow for you.

Step 1 — Dump the configuration file

nextflow run BioBam/omicsbox-engine-workflows \
    -r main \
    -main-script nextflow/workflows/<workflow>/<workflow>.nf \
    --dump_config

The template is written to your current directory exactly as in Option A. -r main selects the published branch and -main-script selects the pipeline within the repository.

Step 2 — Customise it

Edit the generated <workflow>.config exactly as in Option A. Since the pipeline code lives remotely, this file is the only thing you own and edit.

Step 3 — Run the workflow

nextflow run BioBam/omicsbox-engine-workflows \
    -r main \
    -main-script nextflow/workflows/<workflow>/<workflow>.nf \
    -c <workflow>.config \
    -resume

Resuming a Run with -resume

-resume restarts a pipeline from where it stopped: steps whose inputs, parameters and code are unchanged are taken from the cache, and execution picks up at the first step that actually needs to run. For these pipelines it is not a convenience but a cost control — each cached step is a cloud analysis that does not have to be paid for or waited on twice.

Use it whenever you relaunch: after a transient failure, after fixing a bad path, or after changing a flag on a late step.

nextflow run nextflow/workflows/<workflow>/<workflow>.nf \
    -c <workflow>.config \
    -resume

What resume reuses, and where it looks

The cache is bound to the launch directory — the directory you were in when you typed nextflow run, not the directory the pipeline lives in. Nextflow keeps the run history and the task cache under .nextflow/ there, and the task outputs under work/. Two consequences follow:

  • Resuming only works if you relaunch from the same directory as the original run.
  • Deleting work/ or .nextflow/ discards the cache, and the next -resume re-runs everything.

One further behaviour matters, because it decides where the results go. A run launched without -resume generates a new session ID; a run launched with -resume instead reuses the session ID of the run it is resuming. Each attempt still gets its own run name, but the session ID is carried over — which is exactly what makes a resumed attempt continue filling the same results/<session-id>/ folder its first attempt started, rather than opening a second one. Listing the runs in a launch directory shows this directly: consecutive -resume attempts appear under different run names against a single shared session ID.

Run one workflow per directory

Important

Strongly recommended: give every workflow — and ideally every dataset — its own launch directory. A bare -resume resumes the last run recorded in the current directory, whichever pipeline that was — Nextflow does not check that it was the same workflow.

Launching workflow B with -resume from a directory where workflow A ran last therefore achieves nothing: the cache it opens is A's, none of B's tasks match anything in it, and every step of B re-executes from the beginning.

It is also actively unsafe. Since -resume carries the previous session ID over, B publishes into results/<A's session-id>/ instead of a folder of its own, so two unrelated pipelines write into one results directory — and where their step folders and filenames coincide, B's output replaces A's.

A layout that avoids the problem entirely:

~/analyses/
├── project_A_functional_annotation/
│   ├── functional_annotation.config
│   ├── .nextflow/          # history and cache for this workflow only
│   ├── work/
│   └── results/
└── project_A_gene_level/
    ├── gene_level_analysis.config
    ├── .nextflow/
    ├── work/
    └── results/

Each directory holds one workflow's configuration, its own cache and its own results. -resume is then unambiguous, and the two runs can proceed in parallel without interfering.

Resuming a specific run

To resume something other than the most recent run, list the runs recorded in the current directory and resume one by session ID:

nextflow log
nextflow run nextflow/workflows/<workflow>/<workflow>.nf \
    -c <workflow>.config \
    -resume <session-id>

The session IDs listed by nextflow log are the same ones that name the result subfolders, so a result directory can always be traced back to the run that produced it, and that run can be resumed.


Result Handling

The work directory

Nextflow does not run anything inside results/. Every task executes in its own isolated directory under work/, created in the launch directory and named after the task's hash:

work/
└── 49/
    └── 8e1a1bf6d35724f76038203bf2238e/
        ├── .command.sh    # the exact omicsbox command that was run
        ├── .command.log   # its console output
        ├── .exitcode      # its exit status
        └── …              # the staged inputs and everything the task wrote

That directory is where the input files are staged, where the omicsbox command runs, and where all of its output is written — including the numbered folder the step creates for its own results. results/ is not a second location the pipeline writes to: it is a published view of selected outputs, assembled after each task completes by the policy described below.

The results directory

local_folder sets the base output directory. It defaults to ./results, declared in every workflow template, and is resolved relative to the launch directory. Set it to an absolute path when you want results somewhere other than where you launched from:

--local_folder=/scratch/run_042/results

Session-scoped subfolders

The publishing policy is defined once, in the root configuration, and applies to every process of every workflow:

process {
    publishDir = [
        path:      { "${params.local_folder}/${workflow.sessionId}" },
        mode:      'link',
        overwrite: true
    ]
}

Results therefore land in <local_folder>/<session-id>/, and each step writes to its own numbered subdirectory — its ext.outdir:

results/
└── <session-id>/
    ├── 01_<step_name>/
    ├── 02_<step_name>/
    └── NN_<final_step>/

Hard links, not copies

mode: 'link' publishes each output as a hard link rather than a copy. The file under results/ and the file under work/ are two directory entries pointing at the same data on disk. This is chosen deliberately, and it buys both of the things a copy and a symlink each give up:

  • No duplication. Genomic outputs are large and there is no second copy — the data occupies its space once, however many links point at it. Publishing is also instantaneous regardless of file size, since nothing is transferred.
  • Safe to delete work/. Removing the work directory drops one link; the data itself survives as long as the entry under results/ remains. This is the crucial difference from the symlink publishing mode, where clearing work/ would leave a results tree of dangling pointers.

Two practical consequences:

  • Keep local_folder on the same filesystem as work/. Hard links cannot cross filesystem boundaries, so pointing local_folder at a different mount or network share defeats the mechanism.
  • Treat published files as read-only. Editing a file under results/ in place edits the very same data the task cache accounts for in work/, which will invalidate a later -resume. Copy it elsewhere first.

Cleaning up

Once a run has finished and the results have been checked, the work directory is disposable:

rm -rf work/

The published results survive, as described above. What does not survive is the resume cache — after this, a relaunch re-runs every step from the start, so clear work/ only when you are done with the run. nextflow clean offers finer-grained removal of individual runs when several share a directory.


Related pages

  • Available Workflows — the catalogue of published pipelines, each with its workflow ID and what it does, linking to per-pipeline step graphs and output layouts
  • Architecture — what the configuration on this page actually drives: the root configuration, the reusable module wrappers, the three files of a workflow directory, and how they combine into a task at run time
  • Developer Guide — going beyond configuration: which layer a change belongs to, how to rewire or assemble a pipeline, and how to add steps of your own
  • Snakemake Execution & Configuration — the same material for the Snakemake port

Clone this wiki locally