Repository navigation
Nextflow Execution & Configuration
This page covers everything shared by the Nextflow pipelines in this repository: how they are configured, how a value set in one place overrides a value set in another, how to launch a run, how to resume one, and where the results end up. The individual pipeline pages document what each workflow does; this page documents how to drive any of them.
Every pipeline ships a single user-facing configuration file,
nextflow/workflows/<workflow>/<workflow>.config. It is the only file a user is expected to
edit, and it is designed so that no external parameter reference is needed: every option a workflow
exposes is already declared in it, grouped, and annotated inline with its accepted values and its
default.
This is a deliberate strategy. There is no parameter table to consult and no OmicsBox CLI syntax to memorise — the accepted values travel with the parameter, in the same file, on the same line.
Every workflow supports --dump_config. It copies the packaged template into the current directory
and exits with status 0 before any process is submitted, so nothing reaches the cloud:
nextflow run nextflow/workflows/<workflow>/<workflow>.nf --dump_configThe dumped file is a complete, valid, standalone configuration — not a stub. Edit that copy and keep it; it is the artifact that records how a run was parameterised.
A template has two blocks: a params block and a process block.
Its first section holds the workflow-level settings — input paths, the output location, and formatting options:
params {
// ---------------------------------------------------
// GLOBAL PIPELINE PARAMETERS
// ---------------------------------------------------
input_single_end = null // Glob for single-end reads (e.g. 'data/*.fastq.gz')
input_paired_end = null // Glob for paired-end reads (e.g. 'data/*_{1,2}.fastq.gz')
input_fasta = null // REQUIRED: reference genome FASTA (SHARED: used by BWA, BCFtools and Variant Annotation). Ensembl recommended.
local_folder = './results' // Base output directory for all workflow results
chart_format = 'box' // Output chart format: [box, png, pdf, csv]Most workflows then declare a second section of step-scoped parameters, grouped into nested blocks named after the step that consumes them — typically the optional input files a step accepts and the mode choices it needs:
bcftools {
group_experiment = null // Optional: tab-delimited sample->group file. To use it you MUST also set --use-groups=true below.
}
gwas {
input_pheno = null // REQUIRED: tab-separated traits table (1st column = sample names matching the VCF; traits in following columns; header required)
kinship = null // Optional: precomputed kinship matrix file. To use it you MUST also set --use-kinship=true below.
covariate_matrix = null // Optional: covariate-matrix metadata file (requires --use-covariate-matrix=true).
}Several of these are coupled to a flag in the step's own list: a file supplied here is ignored unless the matching flag also switches its use on. See Respecting the dependencies between flags below.
This block carries one withName: entry per pipeline step. Each entry sets an ext.outdir — the
numbered result folder that step publishes to — and an ext.args, the list of flags handed to the
underlying omicsbox command:
withName: 'GO_MAPPING' {
ext.outdir = '03_go_mapping'
ext.args = [
"--latest-database-version=true", // [true, false] Use the latest GOA database version. Default: true
//"--goa-version=\"\"", // [string] Specify a particular GOA version if latest is set to false.
].join(' ')
}ext.args is a list of flags joined into one string, which gives the "checkbox" editing model:
commenting out a line disables exactly that one flag, and uncommenting one enables it. Nothing else
in the step changes. The values shown above are the shipped defaults — they are meant to be edited.
The comment beside each flag is its documentation, and carries up to four pieces of information: the accepted values in square brackets, a one-clause description of what the flag controls, the default OmicsBox applies when the flag is absent — so a commented-out line leaves the flag at its default rather than switching it off — and, where one exists, the dependency that decides whether the flag is read at all.
Warning
Because ext.args resolves to a single string value, assigning it replaces the step's entire
flag list — it does not merge flag by flag. Always edit the flags inside the full dumped list.
Hand-writing a minimal ext.args with one entry silently drops every other flag for that step.
Important
Flags are documented individually but they are not independent. Commenting, uncommenting or
flipping a true/false in isolation is the most common way to break an otherwise valid
configuration.
Before changing a flag, read its comment for a dependency clause and check the flags it names.
Four patterns of dependency appear in the templates:
-
Parent switch and its children. A group of flags is only read when the flag that enables the
feature is
true— every(only if remove-adapters=true)in a trimming step, for example. - Mutually exclusive modes. Choosing one value of a mode flag disables the flags belonging to the other mode, so switching mode means editing more than one line.
-
Coupling between
paramsandext.args. A file supplied in theparamsblock is frequently ignored unless a matching flag switches its use on —group_experimentneeds--use-groups=true,kinshipneeds--use-kinship=true,covariate_matrixneeds--use-covariate-matrix=true. - Constraints from the input layout or from a later step. Some flags are valid only for one read layout and cause OmicsBox to error otherwise; others are safe in themselves but change an output in a way a downstream step depends on, such as a filename prefix that renames samples and must then match the experimental design file.
The templates state these couplings where they exist. The GO-Slim step shows all of the first three at once, and the comment spells out every line the change touches:
withName: 'GO_SLIM' {
ext.outdir = '13_go_slim'
ext.args = [
"--option=web", // [web, custom] Source for the GO-Slim OBO file. Default: web.
// If set to 'custom': COMMENT OUT --go-slim-web-file below (custom disables it) and
// provide your OBO file via params.goslim.obo_file.
"--go-slim-web-file=generic" // [generic, plant, candida, yeast, protein, pombe, aspergillus, metagenomics, agr, chembl, drosophila, mouse, prokaryote, synapse, virus, ...] GO-Slim subset. Default: generic. (Only when --option=web.)
].join(' ')
}Switching that one value from web to custom therefore requires three coordinated edits: change
the mode, comment out the flag the new mode disables, and set the file in the params block.
Each workflow directory contains a small loader, nextflow/workflows/<workflow>/nextflow.config,
whose whole job is to stack the shared configuration underneath the workflow-specific one:
includeConfig '../../nextflow.config'
includeConfig './functional_annotation.config'Order matters: the root configuration is included first, so the workflow's own template overrides it wherever the two declare the same key. On top of that sit the two command-line mechanisms. From lowest to highest precedence:
| # | Layer | What it provides | How it is applied |
|---|---|---|---|
| 1 | nextflow/nextflow.config |
dump_config, local_folder, and the global publishDir policy |
includeConfig from the loader |
| 2 | nextflow/workflows/<workflow>/<workflow>.config |
the workflow's own params and every step's ext.outdir / ext.args
|
includeConfig from the loader |
| 3 | -c my.config |
your edited copy of the template | command line |
| 4 | --<param>=<value> |
one-off overrides for a single run | command line |
flowchart TD
L1["nextflow.config<br/>(root defaults)"]
L2["workflow.config<br/>(packaged template)"]
L3["-c your.config"]
L4["--param=value"]
R["Value used<br/>by the run"]
L1 --> L2 --> L3 --> L4 --> R
-c layers a complete configuration over the packaged one, so it can override any scope: the
params block, the process block, each individual withName: entry with its ext.outdir and
ext.args. The dumped template is delivered with -c because it is exactly that — a full configuration.
Note
-params-file is not a substitute for -c. Nextflow also accepts a JSON or YAML file of
parameters via -params-file, and it sits above -c in the precedence order — but it carries
params and nothing else. The dumped template is not only a params block: it also holds the
process block with every step's ext.outdir and ext.args, which a parameters file cannot
express. Passing a template's parameters through -params-file while omitting -c therefore
leaves every step with an empty flag list.
These workflows take no positional arguments. Every input is a named parameter, and any of them can be supplied — or overridden — directly on the command line, which is the quickest way to point a configured pipeline at a different dataset without editing a file:
nextflow run nextflow/workflows/gene_level_analysis/gene_level_analysis.nf \
-c gene_level_analysis.config \
--input_paired_end='data/*_{1,2}.fastq.gz' \
--input_fasta=/data/reference/genome.fa \
--input_gff=/data/reference/genome.gtf \
--experimental_design=/data/design.tsv \
--local_folder=/scratch/run_042/resultsPoints to keep straight:
-
Two dashes for pipeline parameters, one dash for Nextflow options.
--input_fastasetsparams.input_fasta;-c,-resume,-rand-main-scriptare options for Nextflow itself. A single dash on a pipeline parameter is silently interpreted as a Nextflow option and will not reach the workflow. -
Quote your globs. Write
--input_paired_end='data/*_{1,2}.fastq.gz'with the quotes. Without them the shell rewrites{1,2}into two separate arguments before Nextflow ever sees the command, and only the first is read as the parameter. Quoting passes the pattern through untouched, which is what the workflow needs — it matches the files itself. -
Nested parameters use dotted names. A step-scoped parameter such as
params.trimmomatic.adaptersis set with--trimmomatic.adapters=/path/to/adapters.fa. -
CLI overrides win over everything. A value passed with
--beats the same key in your-cfile, which is what makes it useful for sweeping one setting across otherwise identical runs.
Two ways to run any of these workflows are supported. Option A is recommended when you want to
inspect or modify the pipeline code; Option B runs the published version straight from GitHub
with no manual clone. Replace <workflow> with the directory name of the pipeline you are running —
see Available Workflows, where each page documents these same steps
for its own pipeline.
git clone https://github.com/BioBam/omicsbox-engine-workflows.git
cd omicsbox-engine-workflowsStep 1 — Dump the configuration file
nextflow run nextflow/workflows/<workflow>/<workflow>.nf --dump_configThis copies the self-documenting template to ./<workflow>.config in your current directory and
exits. No OmicsBox job is launched.
Step 2 — Customise it
Open the generated <workflow>.config and edit it as described in
Anatomy of the template: set your input paths and output folder, then
fill in or comment/uncomment the step-specific flags you want to change. Save the file.
Step 3 — Run the workflow
nextflow run nextflow/workflows/<workflow>/<workflow>.nf \
-c <workflow>.config \
-resumeNo clone required — the engine fetches the workflow for you.
Step 1 — Dump the configuration file
nextflow run BioBam/omicsbox-engine-workflows \
-r develop \
-main-script nextflow/workflows/<workflow>/<workflow>.nf \
--dump_configThe template is written to your current directory exactly as in Option A. -r develop selects the
published branch and -main-script selects the pipeline within the repository.
Step 2 — Customise it
Edit the generated <workflow>.config exactly as in Option A. Since the pipeline code lives
remotely, this file is the only thing you own and edit.
Step 3 — Run the workflow
nextflow run BioBam/omicsbox-engine-workflows \
-r develop \
-main-script nextflow/workflows/<workflow>/<workflow>.nf \
-c <workflow>.config \
-resume-resume restarts a pipeline from where it stopped: steps whose inputs, parameters and code are
unchanged are taken from the cache, and execution picks up at the first step that actually needs to
run. For these pipelines it is not a convenience but a cost control — each cached step is a cloud
analysis that does not have to be paid for or waited on twice.
Use it whenever you relaunch: after a transient failure, after fixing a bad path, or after changing a flag on a late step.
nextflow run nextflow/workflows/<workflow>/<workflow>.nf \
-c <workflow>.config \
-resumeThe cache is bound to the launch directory — the directory you were in when you typed
nextflow run, not the directory the pipeline lives in. Nextflow keeps the run history and the task
cache under .nextflow/ there, and the task outputs under work/. Two consequences follow:
- Resuming only works if you relaunch from the same directory as the original run.
- Deleting
work/or.nextflow/discards the cache, and the next-resumere-runs everything.
One further behaviour matters, because it decides where the results go. A run launched without
-resume generates a new session ID; a run launched with -resume instead reuses the session
ID of the run it is resuming. Each attempt still gets its own run name, but the session ID is
carried over — which is exactly what makes a resumed attempt continue filling the same
results/<session-id>/ folder its first attempt started, rather than opening a second one. Listing
the runs in a launch directory shows this directly: consecutive -resume attempts appear under
different run names against a single shared session ID.
Important
Strongly recommended: give every workflow — and ideally every dataset — its own launch
directory. A bare -resume resumes the last run recorded in the current directory, whichever
pipeline that was — Nextflow does not check that it was the same workflow.
Launching workflow B with -resume from a directory where workflow A ran last therefore achieves
nothing: the cache it opens is A's, none of B's tasks match anything in it, and every step of B
re-executes from the beginning.
A layout that avoids the problem entirely:
~/analyses/
├── project_A_functional_annotation/
│ ├── functional_annotation.config
│ ├── .nextflow/ # history and cache for this workflow only
│ ├── work/
│ └── results/
└── project_A_gene_level/
├── gene_level_analysis.config
├── .nextflow/
├── work/
└── results/
Each directory holds one workflow's configuration, its own cache and its own results. -resume is
then unambiguous, and the two runs can proceed in parallel without interfering.
To resume something other than the most recent run, list the runs recorded in the current directory and resume one by session ID:
nextflow log
nextflow run nextflow/workflows/<workflow>/<workflow>.nf \
-c <workflow>.config \
-resume <session-id>The session IDs listed by nextflow log are the same ones that name the result subfolders, so a
result directory can always be traced back to the run that produced it, and that run can be resumed.
Nextflow does not run anything inside results/. Every task executes in its own isolated directory
under work/, created in the launch directory and named after the task's hash:
work/
└── 49/
└── 8e1a1bf6d35724f76038203bf2238e/
├── .command.sh # the exact omicsbox command that was run
├── .command.log # its console output
├── .exitcode # its exit status
└── … # the staged inputs and everything the task wrote
That directory is where the input files are staged, where the omicsbox command runs, and where
all of its output is written — including the numbered folder the step creates for its own
results. results/ is not a second location the pipeline writes to: it is a published view of
selected outputs, assembled after each task completes by the policy described below.
local_folder sets the base output directory. It defaults to ./results, declared in every workflow template,
and is resolved relative to the launch directory. Set it to an absolute path when you want results somewhere
other than where you launched from:
--local_folder=/scratch/run_042/resultsThe publishing policy is defined once, in the root configuration, and applies to every process of every workflow:
process {
publishDir = [
path: { "${params.local_folder}/${workflow.sessionId}" },
mode: 'link',
overwrite: true
]
}Results therefore land in <local_folder>/<session-id>/, and each step writes to its own numbered
subdirectory — its ext.outdir:
results/
└── <session-id>/
├── 01_<step_name>/
├── 02_<step_name>/
└── NN_<final_step>/
mode: 'link' publishes each output as a hard link rather than a copy. The file under
results/ and the file under work/ are two directory entries pointing at the same data on disk.
This is chosen deliberately, and it buys both of the things a copy and a symlink each give up:
- No duplication. Genomic outputs are large and there is no second copy — the data occupies its space once, however many links point at it. Publishing is also instantaneous regardless of file size, since nothing is transferred.
-
Safe to delete
work/. Removing the work directory drops one link; the data itself survives as long as the entry underresults/remains. This is the crucial difference from the symlink publishing mode, where clearingwork/would leave a results tree of dangling pointers.
Two practical consequences:
-
Keep
local_folderon the same filesystem aswork/. Hard links cannot cross filesystem boundaries, so pointinglocal_folderat a different mount or network share defeats the mechanism. -
Treat published files as read-only. Editing a file under
results/in place edits the very same data the task cache accounts for inwork/, which will invalidate a later-resume. Copy it elsewhere first.
Once a run has finished and the results have been checked, the work directory is disposable:
rm -rf work/The published results survive, as described above. What does not survive is the resume cache — after
this, a relaunch re-runs every step from the start, so clear work/ only when you are done with the
run. nextflow clean offers finer-grained removal of individual runs when several share a directory.
- Available Workflows — per-pipeline inputs, steps and outputs
- Architecture — how modules, workflows and configuration fit together
- Developer Guide — adding or modifying a module or workflow
- Snakemake Execution & Configuration — the same material for the Snakemake port
-
Nextflow
-
Snakemake