From 44344c3b7eccdce70b9c66efa709b83c7f38e274 Mon Sep 17 00:00:00 2001 From: Christopher Hakkaart Date: Fri, 17 Apr 2026 21:53:18 +1200 Subject: [PATCH 01/10] What is a module --- .../developing/components/what-is-a-module.md | 70 +++++++++++++++++++ 1 file changed, 70 insertions(+) create mode 100644 sites/docs/src/content/docs/developing/components/what-is-a-module.md diff --git a/sites/docs/src/content/docs/developing/components/what-is-a-module.md b/sites/docs/src/content/docs/developing/components/what-is-a-module.md new file mode 100644 index 0000000000..c66854910b --- /dev/null +++ b/sites/docs/src/content/docs/developing/components/what-is-a-module.md @@ -0,0 +1,70 @@ +--- +title: "What is an nf-core module?" +subtitle: "Learn what defines an nf-core module" +shortTitle: "What is a module?" +--- + +## Introduction + +An nf-core module is an opinionated, open-source Nextflow wrapper around a single command-line tool command or script. +A [central community repository](https://github.com/nf-core/modules) holds more than [1000 modules](https://nf-co.re/modules) that you can reuse in your own pipelines. + +This page describes what an nf-core module is and what distinguishes it from a standard Nextflow process. + +## Contents of an nf-core module + +An nf-core module usually wraps one tool command, or one tool with a single subcommand. + +:::info{collapse title="Comparison with Nextflow modules"} +A Nextflow process or module has no restrictions on the number of tools or subcommands it can execute. + +An nf-core module aims for a single analysis command per module. +This atomic design supports lego-block construction of pipelines and makes modules easier to understand, share, and test, although it is not always the most resource-efficient approach. +::: + +Each nf-core module consists of a set of files whose structure and contents follow strict [specifications](../../specifications/component/overview). +The nf-core community agrees these specifications through consensus, and they cover areas such as: + +- [Naming](../../specifications/components/modules/naming-conventions) and [formatting](../../specifications/components/modules/formatting) conventions +- [Software environments](../../specifications/components/modules/software-requirements) +- [Standardised tags](../../specifications/components/modules/testing#tags) +- [Input and output channels](../../specifications/components/modules/input-output-options) +- [Tool parameters](../../specifications/components/modules/module-parameters) that pipeline developers can configure +- Use of ['meta maps'](../../specifications/components/modules/general#use-of-meta-maps) +- Use of [stubs](../../specifications/components/modules/general#stubs) + +The specifications also expand the minimum file set from a single Nextflow `.nf` script to five required files. +This reflects the nf-core focus on standardisation, reproducibility, and high-quality documentation. + +For example, a complete nf-core module directory can contain up to six files: + +```tree +├── environment.yml +├── main.nf +├── meta.yml +└── tests/ + ├── main.nf.test + ├── main.nf.test.snap + └── nextflow.config ## optional! +``` + +These files fall into three categories: + +- Module execution +- Module documentation +- Module testing + +The following diagram shows how these files relate to each other: + +Diagram showing how the execution, documentation, and testing files in an nf-core module relate to each other. + +_Schematic diagram showing the relationship between the three main categories of files in an nf-core module._ + +- `main.nf` is the main Nextflow script and defines the process that Nextflow executes. +- `environment.yml` is a Conda environment file loaded by `main.nf`. It specifies the software dependencies for the module when a pipeline runs with `-profile conda`. +- `meta.yml` documents the contents of `main.nf`, including tool metadata and input and output specifications. +- `main.nf.test` describes an nf-test unit test for the `main.nf` process. +- `main.nf.test.snap` is a snapshot file generated by nf-test. It compares the output of one test run with another to confirm that the process is reproducible. +- `nextflow.config` is an optional Nextflow configuration file used by the test that runs when `main.nf.test` executes. + +Writing an nf-core module involves creating these files and populating them with content that follows the nf-core specifications. From 8217164b65fa4b137864fa19e054c480b918f97c Mon Sep 17 00:00:00 2001 From: Christopher Hakkaart Date: Mon, 20 Apr 2026 10:03:18 +1200 Subject: [PATCH 02/10] Add tutorial --- .../developing/components/what-is-a-module.md | 70 -- .../writing-nf-core-modules/1-introduction.md | 57 ++ .../2-getting-started.md | 41 + .../3-what-is-a-module.md | 64 ++ .../writing-nf-core-modules/4-boilerplate.md | 84 ++ .../5-writing-modules.md | 790 ++++++++++++++++++ .../writing-nf-core-modules/6-testing.md | 604 +++++++++++++ .../writing-nf-core-modules/7-development.md | 89 ++ .../writing-nf-core-modules/8-using.md | 157 ++++ sites/main-site/public/_redirects | 11 + 10 files changed, 1897 insertions(+), 70 deletions(-) delete mode 100644 sites/docs/src/content/docs/developing/components/what-is-a-module.md create mode 100644 sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/1-introduction.md create mode 100644 sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/2-getting-started.md create mode 100644 sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/3-what-is-a-module.md create mode 100644 sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/4-boilerplate.md create mode 100644 sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/5-writing-modules.md create mode 100644 sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/6-testing.md create mode 100644 sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/7-development.md create mode 100644 sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/8-using.md diff --git a/sites/docs/src/content/docs/developing/components/what-is-a-module.md b/sites/docs/src/content/docs/developing/components/what-is-a-module.md deleted file mode 100644 index c66854910b..0000000000 --- a/sites/docs/src/content/docs/developing/components/what-is-a-module.md +++ /dev/null @@ -1,70 +0,0 @@ ---- -title: "What is an nf-core module?" -subtitle: "Learn what defines an nf-core module" -shortTitle: "What is a module?" ---- - -## Introduction - -An nf-core module is an opinionated, open-source Nextflow wrapper around a single command-line tool command or script. -A [central community repository](https://github.com/nf-core/modules) holds more than [1000 modules](https://nf-co.re/modules) that you can reuse in your own pipelines. - -This page describes what an nf-core module is and what distinguishes it from a standard Nextflow process. - -## Contents of an nf-core module - -An nf-core module usually wraps one tool command, or one tool with a single subcommand. - -:::info{collapse title="Comparison with Nextflow modules"} -A Nextflow process or module has no restrictions on the number of tools or subcommands it can execute. - -An nf-core module aims for a single analysis command per module. -This atomic design supports lego-block construction of pipelines and makes modules easier to understand, share, and test, although it is not always the most resource-efficient approach. -::: - -Each nf-core module consists of a set of files whose structure and contents follow strict [specifications](../../specifications/component/overview). -The nf-core community agrees these specifications through consensus, and they cover areas such as: - -- [Naming](../../specifications/components/modules/naming-conventions) and [formatting](../../specifications/components/modules/formatting) conventions -- [Software environments](../../specifications/components/modules/software-requirements) -- [Standardised tags](../../specifications/components/modules/testing#tags) -- [Input and output channels](../../specifications/components/modules/input-output-options) -- [Tool parameters](../../specifications/components/modules/module-parameters) that pipeline developers can configure -- Use of ['meta maps'](../../specifications/components/modules/general#use-of-meta-maps) -- Use of [stubs](../../specifications/components/modules/general#stubs) - -The specifications also expand the minimum file set from a single Nextflow `.nf` script to five required files. -This reflects the nf-core focus on standardisation, reproducibility, and high-quality documentation. - -For example, a complete nf-core module directory can contain up to six files: - -```tree -├── environment.yml -├── main.nf -├── meta.yml -└── tests/ - ├── main.nf.test - ├── main.nf.test.snap - └── nextflow.config ## optional! -``` - -These files fall into three categories: - -- Module execution -- Module documentation -- Module testing - -The following diagram shows how these files relate to each other: - -Diagram showing how the execution, documentation, and testing files in an nf-core module relate to each other. - -_Schematic diagram showing the relationship between the three main categories of files in an nf-core module._ - -- `main.nf` is the main Nextflow script and defines the process that Nextflow executes. -- `environment.yml` is a Conda environment file loaded by `main.nf`. It specifies the software dependencies for the module when a pipeline runs with `-profile conda`. -- `meta.yml` documents the contents of `main.nf`, including tool metadata and input and output specifications. -- `main.nf.test` describes an nf-test unit test for the `main.nf` process. -- `main.nf.test.snap` is a snapshot file generated by nf-test. It compares the output of one test run with another to confirm that the process is reproducible. -- `nextflow.config` is an optional Nextflow configuration file used by the test that runs when `main.nf.test` executes. - -Writing an nf-core module involves creating these files and populating them with content that follows the nf-core specifications. diff --git a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/1-introduction.md b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/1-introduction.md new file mode 100644 index 0000000000..e414d75c41 --- /dev/null +++ b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/1-introduction.md @@ -0,0 +1,57 @@ +--- +title: "Chapter 1: Introduction" +subtitle: "Learn to write nf-core modules" +shortTitle: "Chapter 1: Introduction" +--- + +:::tip{title="Last updated"} +Material originally written for the **A practical introduction to nf-core components for Nextflow developers** training at the nf-core Hackathon March 2025, Berlin local site _(see [event](https://nf-co.re/events/2025/hackathon-march-2025/robert-koch-institute))_. + +Duration: **4hr** + +Original author: James A. Fellows Yates (@jfy133), with corrections and improvements from Niklas Schandry (@nschan) and Alexandru Mizeranschi (@amizeranschi). +::: + +An nf-core module is an atomic, standardised, reproducible, and tested [Nextflow DSL2 module](https://www.nextflow.io/docs/latest/module.html). You can install an nf-core module into any Nextflow pipeline with a single command, then integrate it into your workflow thanks to its consistent internal structure. + +## Why use nf-core modules + +- **Efficiency**: reuse community modules instead of writing your own. +- **Consistency**: every module follows the same structure, so modules connect together predictably. +- **Documentation**: each module ships with metadata and descriptions. +- **Future-proofing**: modules cover all inputs and outputs from the start, so you avoid rewrites when requirements change. +- **Automation**: standardisation enables future automated workflow development. + +## Scope + +This training covers: + +- Writing nf-core modules. +- Using nf-core modules in official nf-core pipelines and custom Nextflow pipelines. + +## Learning objectives + +By the end of this training, you will be able to: + +- Identify the file components of an nf-core module. +- Apply the nf-core module specifications. +- Locate documentation for developing an nf-core module. +- Write an nf-core module. +- Write nf-test unit tests for an nf-core module. +- Follow the workflow for contributing modules to the community repository. +- Use an nf-core module in any Nextflow pipeline. + +## Chapters + +- Chapter 1: [Introduction](/docs/developing/tutorials/writing-nf-core-modules/1-introduction) +- Chapter 2: [Getting started](/docs/developing/tutorials/writing-nf-core-modules/2-getting-started) +- Chapter 3: [What is an nf-core module?](/docs/developing/tutorials/writing-nf-core-modules/3-what-is-a-module) +- Chapter 4: [Generating boilerplate files](/docs/developing/tutorials/writing-nf-core-modules/4-boilerplate) +- Chapter 5: [Writing an nf-core module](/docs/developing/tutorials/writing-nf-core-modules/5-writing-modules) +- Chapter 6: [Testing an nf-core module](/docs/developing/tutorials/writing-nf-core-modules/6-testing) +- Chapter 7: [Development workflow](/docs/developing/tutorials/writing-nf-core-modules/7-development) +- Chapter 8: [Using nf-core modules in pipelines](/docs/developing/tutorials/writing-nf-core-modules/8-using) + +:::tip +Read chapters 2–7 through once before starting development. When you reach chapter 7, return to chapter 2 to begin writing the module itself. +::: diff --git a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/2-getting-started.md b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/2-getting-started.md new file mode 100644 index 0000000000..d1f22badcc --- /dev/null +++ b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/2-getting-started.md @@ -0,0 +1,41 @@ +--- +title: "Chapter 2: Getting started" +subtitle: "What you need before starting" +shortTitle: "Chapter 2: Getting started" +--- + +This chapter lists what you need before starting the training. + +## Prerequisite knowledge + +You should already be familiar with: + +- Writing Nextflow pipelines (see the [Nextflow training](https://training.nextflow.io/)). +- How Nextflow modules and processes work (see the [Nextflow modules documentation](https://nextflow.io/docs/latest/module.html)). +- Git and GitHub basics (see the [git tutorial](https://git-scm.com/docs/gittutorial) and [GitHub Skills](https://skills.github.com/)). + +## Required software and accounts + +You need the following installed or set up: + +- [Nextflow](https://www.nextflow.io/docs/latest/install.html). +- [nf-core tools](https://nf-co.re/docs/nf-core-tools/installation). +- A Nextflow-supported software management system, one of: + - [Conda](https://conda-forge.org/download/) + - [Docker](https://docs.docker.com/manuals/) + - [Apptainer](https://apptainer.org/docs/user/latest/quick_start.html#installation) +- A [GitHub account](https://github.com). +- A fork and local clone of the [nf-core/modules repository](https://github.com/nf-core/modules). + +## Tool assumptions + +This training assumes you have: + +- A command-line tool in mind to wrap as an nf-core module. +- A [Bioconda](https://bioconda.github.io/conda-package_index.html) recipe for that tool. + +:::note +This training was written and tested with nf-core/tools 3.2.0 and Nextflow 24.10.4. +::: + +The next chapter introduces what defines an nf-core module. diff --git a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/3-what-is-a-module.md b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/3-what-is-a-module.md new file mode 100644 index 0000000000..4cf222fe29 --- /dev/null +++ b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/3-what-is-a-module.md @@ -0,0 +1,64 @@ +--- +title: "Chapter 3: What is an nf-core module?" +subtitle: "What defines an nf-core module" +shortTitle: "Chapter 3: What is a module?" +--- + +An nf-core module is an opinionated, open-source Nextflow wrapper around a single command-line tool or script. The [central community repository](https://github.com/nf-core/modules) holds more than [1000 modules](https://nf-co.re/modules) that you can reuse in your own pipelines. + +This page describes what an nf-core module is and what distinguishes it from a standard Nextflow process. + +## Contents of an nf-core module + +An nf-core module usually wraps one tool command, or one tool with a single subcommand. + +:::info{collapse title="Comparison with Nextflow modules"} +A Nextflow process or module has no restrictions on the number of tools or subcommands it can execute. + +An nf-core module aims for a single analysis command per module. This atomic design supports lego-block construction of pipelines and makes modules easier to understand, share, and test, although it is not always the most resource-efficient approach. +::: + +Each nf-core module follows strict [specifications](../../../specifications/components/overview) agreed by community consensus. The specifications cover: + +- [Naming](../../../specifications/components/modules/naming-conventions) and [formatting](../../../specifications/components/modules/formatting) conventions. +- [Software environments](../../../specifications/components/modules/software-requirements). +- [Standardised tags](../../../specifications/components/modules/testing#tags). +- [Input and output channels](../../../specifications/components/modules/input-output-options). +- [Tool parameters](../../../specifications/components/modules/module-parameters) configurable by pipeline developers. +- Use of [meta maps](../../../specifications/components/modules/general#use-of-meta-maps). +- Use of [stubs](../../../specifications/components/modules/general#stubs). + +The specifications also expand the minimum file set from a single Nextflow `.nf` script to five required files. This reflects the nf-core focus on standardisation, reproducibility, and high-quality documentation. + +A complete nf-core module directory can contain up to six files: + +```tree +├── environment.yml +├── main.nf +├── meta.yml +└── tests/ + ├── main.nf.test + ├── main.nf.test.snap + └── nextflow.config ## optional! +``` + +These files fall into three categories: + +- Module execution +- Module documentation +- Module testing + +The following diagram shows how these files relate to each other: + +Diagram showing how the execution, documentation, and testing files in an nf-core module relate to each other. + +_Schematic diagram showing the relationship between the three main categories of files in an nf-core module._ + +- `main.nf` defines the Nextflow process that the module executes. +- `environment.yml` is a Conda environment file loaded by `main.nf`. It specifies the software dependencies used when a pipeline runs with `-profile conda`. +- `meta.yml` documents the contents of `main.nf`, including tool metadata and input and output specifications. +- `main.nf.test` describes an nf-test unit test for the `main.nf` process. +- `main.nf.test.snap` is a snapshot file generated by nf-test. It compares the output of one test run with another to confirm reproducibility. +- `nextflow.config` is an optional Nextflow configuration file used when `main.nf.test` executes. + +Writing an nf-core module means creating these files and populating them with content that follows the nf-core specifications. diff --git a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/4-boilerplate.md b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/4-boilerplate.md new file mode 100644 index 0000000000..22d3dc5bd0 --- /dev/null +++ b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/4-boilerplate.md @@ -0,0 +1,84 @@ +--- +title: "Chapter 4: Generating boilerplate files" +subtitle: "How to generate the module file skeleton" +shortTitle: "Chapter 4: Boilerplate files" +--- + +This chapter explains how to generate an nf-core module template and what each file contains. + +## Preparation + +Fork and clone the [nf-core/modules GitHub repository](https://github.com/nf-core/modules) to your working environment, then create a new branch for your module: + +```bash +git switch master ## ensure you have the latest state of the repository +git switch -c +``` + +## Generate the boilerplate files + +From the root of the repository, run: + +```bash +nf-core modules create / +``` + +### Naming conventions + +- All parts of the module name must be lowercase, alphanumeric, with no punctuation or special characters. +- Single-command tools use the tool name only. For a tool executed with `fastp -i -o `, run `nf-core modules create fastp`. +- Tools with subcommands use `/`, even if you only plan to wrap one subcommand. For `samtools view`, run `nf-core modules create samtools/view`. +- For a third level of subcommand, append it to the subcommand name. For `samtools view flagstats`, run `nf-core modules create samtools/viewflagstat`. + +For example, to create a module for the tool `drep` with the subcommand `compare`: + +```bash +nf-core modules create drep/compare +``` + +### Prompts from `nf-core modules create` + +The command tries to pre-fill the boilerplate. It searches [Bioconda](https://bioconda.github.io/) and [biocontainers](https://biocontainers.pro/) for the latest version of your tool and adds the container definitions automatically. + +You will then be prompted for: + +- **Your GitHub username.** +- **A process resource label.** These standardised tags map to default memory, CPU, and wall time. Choose the label that best matches your tool's typical requirements. The defaults for each label are defined in the pipeline template [base.config](https://github.com/nf-core/tools/blob/52e810986e382972ffad0aab28e94f828ffd509b/nf_core/pipeline-template/conf/base.config#L29-L54). Pipelines can override these values. +- **Whether the module should use a [meta map](https://nf-co.re/docs/developing/components/meta-map).** Answer yes in most cases. Meta maps carry sample metadata alongside files, which pipelines use to drive downstream processing decisions. + +:::tip +You can pass any of these values as command-line flags to skip the prompts. +::: + +## Output + +After the command completes, you will see the following files and directories: + +```tree {8-13} +modules/nf-core/drep/compare/ +├── environment.yml +├── main.nf +├── meta.yml +└── tests + └── main.nf.test +``` + +If you later create a second subcommand (`dereplicate`), the directory structure becomes: + +```tree {8-13} +modules/nf-core/drep/ +├── compare +│ ├── environment.yml +│ ├── main.nf +│ ├── meta.yml +│ └── tests +│ └── main.nf.test +└── dereplicate + ├── environment.yml + ├── main.nf + ├── meta.yml + └── tests + └── main.nf.test +``` + +The next chapter walks through each generated file and explains what to change. \ No newline at end of file diff --git a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/5-writing-modules.md b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/5-writing-modules.md new file mode 100644 index 0000000000..440e8e4c0f --- /dev/null +++ b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/5-writing-modules.md @@ -0,0 +1,790 @@ +--- +title: "Chapter 5: Writing your module" +subtitle: "How to fill in the module files" +shortTitle: "Chapter 5: Writing modules" +--- + +Once you have generated the boilerplate template files, you can update them to make your module function. The boilerplate contains many `TODO` comments and example content to guide you. + +This chapter walks through each file, explaining what each section does and why, as defined by the nf-core specifications. + +## The `environment.yml` file + +The `environment.yml` file is a Conda environment specification. + +If `nf-core modules create` found your tool on Bioconda, the file already contains the channel, tool name, and version: + +```yaml +--- +# yaml-language-server: $schema=https://raw.githubusercontent.com/nf-core/modules/master/modules/environment-schema.json +channels: + - conda-forge + - bioconda +dependencies: + - "bioconda::drep=3.5.0" +``` + +You rarely need to modify this file. The tool and version specified here will usually have a matching biocontainer in the `main.nf` file. + +Only modify `environment.yml` when your tool is not on Bioconda and no recipe exists for it. nf-core guidelines strongly recommend Conda support, as Conda is the most accessible software management system. If your tool is not on Bioconda or [conda-forge](https://conda-forge.org/), add it to the appropriate repository. + +:::info{title="Behind the scenes" collapse} +nf-core uses a separate Conda file rather than the `main.nf` conda directive for two reasons: + +- It enables [automated container building with Seqera's wave infrastructure](https://nf-co.re/blog/2024/seqera-containers-part-1). +- It makes multi-tool environments easier to read and manage than multiple conda declarations on a single line. +::: + +## The `main.nf` file + +The `main.nf` file contains the Nextflow module code. When generated, parts of the module are pre-filled and `TODO` comments mark where you need to make changes. + +An nf-core module has 9 main Nextflow process blocks: + +- `tag:` +- `label:` +- `conda:` +- `container:` +- `input:` +- `output:` +- `when:` +- `script:` +- `stub:` + +:::info{title="Click here to see full 'raw' file example" collapse} + +The boilerplate `TODO` comments have been removed for readability. + +```nextflow +process DREP_COMPARE { + tag "$meta.id" + label 'process_single' + + conda "${moduleDir}/environment.yml" + container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ? + 'https://depot.galaxyproject.org/singularity/drep:3.5.0--pyhdfd78af_0': + 'biocontainers/drep:3.5.0--pyhdfd78af_0' }" + + input: + tuple val(meta), path(bam) + + output: + tuple val(meta), path("*.bam"), emit: bam + path "versions.yml" , emit: versions + + when: + task.ext.when == null || task.ext.when + + script: + def args = task.ext.args ?: '' + def prefix = task.ext.prefix ?: "${meta.id}" + """ + samtools \\ + sort \\ + $args \\ + -@ $task.cpus \\ + -o ${prefix}.bam \\ + -T $prefix \\ + $bam + + cat <<-END_VERSIONS > versions.yml + "${task.process}": + drep: \$(samtools --version |& sed '1!d ; s/samtools //') + END_VERSIONS + """ + + stub: + def args = task.ext.args ?: '' + def prefix = task.ext.prefix ?: "${meta.id}" + """ + touch ${prefix}.bam + + cat <<-END_VERSIONS > versions.yml + "${task.process}": + drep: \$(samtools --version |& sed '1!d ; s/samtools //') + END_VERSIONS + """ +} +``` + +::: + +Each block is described below. + +### The `tag` block + +The `tag` block propagates standard metadata from each input channel's [meta map](https://nf-co.re/docs/developing/components/meta-map). + +```nextflow +tag "$meta.id" +``` + +You generally do not need to modify this. All nf-core pipelines assume an `id` element in each meta map, and pipeline developers can customise it. + +### The `label` block + +The `label` block is pre-filled with the label you selected during boilerplate generation. + +```nextflow +label 'process_single' +``` + +Key points: + +- nf-core defines a preset set of [standard labels with default resources](https://github.com/nf-core/tools/blob/52e810986e382972ffad0aab28e94f828ffd509b/nf_core/pipeline-template/conf/base.config#L22-L54). +- Select the label that best matches your tool's typical resource needs. +- Do not use custom labels. Pipeline developers can tune default resources in the pipeline's `conf/base.config`, and users can override per-module resources using `withName:` in a custom config. +- Standard labels keep modules and pipelines consistent in how resources are defined. + +### The `conda` block + +The `conda` block tells Nextflow to use the `environment.yml` file when a pipeline runs with Conda. + +```nextflow +conda "${moduleDir}/environment.yml" +``` + +Do not modify this. The only exception is a tool that does not support Conda (typically proprietary tools), where you may remove the line. + +### The `container` block + +The `container` block tells Nextflow where to pull the tool's Docker and Singularity containers from. + +```nextflow +container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ? + 'https://depot.galaxyproject.org/singularity/drep:3.5.0--pyhdfd78af_0': + 'biocontainers/drep:3.5.0--pyhdfd78af_0' }" +``` + +The `nf-core modules create` command auto-detects the container name, version, and build from Bioconda and biocontainers. + +Do not modify this unless the tool has no Bioconda recipe or biocontainer (typically proprietary tools). In that case, point the lines to the relevant container. + +### The `input` block + +The `input` block defines what the module receives. + +```nextflow +input: +tuple val(meta), path(bam) +``` + +The boilerplate includes an example BAM input and a [meta map](https://nf-co.re/docs/developing/components/meta-map). Edit this section to declare every file your module needs. + +Guidelines: + +- Declare **all possible files** with a `path()` entry, both mandatory and optional. Check the tool's documentation to capture them all. +- Declare mandatory non-file arguments as `val` input channels, usually without a meta map. +- Name channels after the file format or the tool's command-line argument. +- Use one channel per input file. +- Each extra input channel gets its own meta map, named `meta2`, `meta3`, and so on. +- Every input channel should have a meta, unless you can guarantee only one version of that file exists across all samples (for example, a shared configuration file). +- Closely related files can share a tuple channel when neither can be used alone, such as a `.bai` index with a `.bam` file. + +```nextflow +input: +tuple val(meta), path(bam) +val mode +``` + +```nextflow +input: +tuple val(meta), path(bam), path(bai) +``` + +:::info{title="Behind the scenes" collapse} +Nextflow processes must receive files through input channels to stage them correctly in the job's working directory. Declaring every possible input keeps the module reusable across pipelines. + +Nextflow has no native optional input syntax. Pipeline developers pass an empty list `[]` to indicate an input is not provided. + +Mandatory non-file arguments belong in the input block so the module can run without extra configuration. Optional parameters are handled through `ext.args`, described below. + +The one-channel-one-file rule keeps channel structures consistent. Pipeline developers can use Nextflow's [`.multiMap()`](https://www.nextflow.io/docs/latest/reference/operator.html#multimap) operator to split multi-element channels into sub-channels while keeping elements associated. +::: + +:::tip{title="Examples" collapse} + +Single input file with a mandatory argument: + +```nextflow +input: +tuple val(meta), path(fasta) +val(output_format) +``` + +Multiple input files: + +```nextflow +input: +tuple val(meta), path(reads) +tuple val(meta2), path(fasta) +tuple val(meta3), path(fasta_index) +``` + +Input with a closely associated index file: + +```nextflow +input: +tuple val(meta), path(input), path(intervals) +path fasta +``` + +::: + +### The `output` block + +The `output:` block declares every file the tool can produce. + +```nextflow +output: +path "versions.yml" , emit: versions +``` + +The boilerplate always includes a mandatory `versions.yml` emission, because nf-core pipelines must report every tool version. + +Guidelines: + +- Declare as many outputs as possible during first creation, whether default or optional, to maximise reuse and minimise future channel changes. +- If the module uses a meta map, emit all files with it except `versions.yml`. +- Give each file type its own `emit` entry, usually one per format. +- Combine two mutually exclusive formats that serve the same purpose (such as `.bai` and `.csi` indexes for `.bam`) in one channel. +- Compress output files where the tool supports compressed input, and reflect this in the path pattern. +- Name `emit:` channels after the file format or suffix. +- Mark variable outputs with Nextflow's `optional: true`. + +```nextflow +output: +tuple val(meta), path("${prefix}.bam"), emit: bam, optional: true +tuple val(meta), path("${prefix}.cram"), emit: cram, optional: true +tuple val(meta), path("${prefix}.sam"), emit: sam, optional: true +path "versions.yml", +``` + +```nextflow +output: +tuple val(meta), path("*.{vcf,vcf.gz}"), emit: vcf +``` + +:::info{title="Behind the scenes" collapse} +Comprehensive inputs and outputs during initial creation keep modules reusable and reduce breaking changes. Pipeline developers rely on a stable channel structure. +::: + +:::tip{title="Examples" collapse} +A module producing multiple files, some optional: + +```bash +output: +tuple val(meta), path('*.fastp.fastq.gz') , optional:true, emit: reads +tuple val(meta), path('*.json') , emit: json +tuple val(meta), path('*.html') , emit: html +tuple val(meta), path('*.log') , emit: log +tuple val(meta), path('*.fail.fastq.gz') , optional:true, emit: reads_fail +tuple val(meta), path('*.merged.fastq.gz'), optional:true, emit: reads_merged +path "versions.yml" , emit: versions +``` + +A module producing three mutually exclusive primary outputs: + +```nextflow +output: +tuple val(meta), path("*.{vcf,vcf.gz,bcf,bcf.gz}"), emit: vcf +tuple val(meta), path("*.tbi") , emit: tbi, optional: true +tuple val(meta), path("*.csi") , emit: csi, optional: true +path "versions.yml" , emit: versions +``` + +A module producing multiple files that all support compressed variants: + +```nextflow +output: +tuple val(meta), path('*.{blast,blast.gz}'), optional: true, emit: blast +tuple val(meta), path('*.{xml,xml.gz}') , optional: true, emit: xml +tuple val(meta), path('*.{txt,txt.gz}') , optional: true, emit: txt +tuple val(meta), path('*.{daa,daa.gz}') , optional: true, emit: daa +tuple val(meta), path('*.{sam,sam.gz}') , optional: true, emit: sam +tuple val(meta), path('*.{tsv,tsv.gz}') , optional: true, emit: tsv +tuple val(meta), path('*.{paf,paf.gz}') , optional: true, emit: paf +path "versions.yml" , emit: versions +``` + +::: + +### The `when` block + +The `when` block lets pipeline developers dynamically activate or deactivate the module. + +```nextflow +when: +task.ext.when == null || task.ext.when +``` + +Do not modify or remove this block. + +### The `script` block + +The `script` block defines the command the module runs. You will edit this section most. + +```nextflow +script: +def args = task.ext.args ?: '' +def prefix = task.ext.prefix ?: "${meta.id}" +""" +samtools \\ + sort \\ + $args \\ + -@ $task.cpus \\ + -o ${prefix}.bam \\ + -T $prefix \\ + $bam + +cat <<-END_VERSIONS > versions.yml +"${task.process}": + drep: \$(samtools --version |& sed '1!d ; s/samtools //') +END_VERSIONS +""" +``` + +The boilerplate includes an example samtools command and a `versions.yml` HEREDOC. Replace both with commands for your tool. + +Two standard variables appear at the top of the block. Do not remove them: + +- `args` — how pipeline developers inject optional parameters into the command. The value comes from `ext.args` in the process scope of a Nextflow configuration file (defined in `modules.config` for nf-core pipelines). See the "Using in pipelines" chapter for details. +- `prefix` — the default output file basename. It defaults to the `id` value of the primary input channel's meta map. + +Replace the example command with your own command, split across multiple lines with escaped backslashes for readability. Reference all required input variables. + +Every command must use the `$args` variable, and the CPU count where supported. + +You can add extra variables after `args` and `prefix` to inject input files dynamically. For example, to handle single- or paired-end sequencing data: + +```nextflow +script: +def args = task.ext.args ?: '' +def prefix = task.ext.prefix ?: "${meta.id}" +def single_end = meta.single_end ? "--single" : "" +""" +elprep merge \\ + input/ \\ + output/${prefix}.bam \\ + $args \\ + ${single_end} \\ +... +""" +``` + +Keep the `versions.yml` HEREDOC, but replace the command to emit a clean version string (for example `1.2.3`, not `v1.2.3`). Use standard UNIX tools like `cut` or `sed` for the clean-up. If the tool does not report its version on the command line, use a dedicated variable after `args` and `prefix`: + +```nextflow +script: +def args = task.ext.args ?: '' +def prefix = task.ext.prefix ?: "${meta.id}" +def VERSION='2.1.3' // WARN: Version information not provided by tool on CLI. Please update this string when bumping +""" +... + +cat <<-END_VERSIONS > versions.yml +"${task.process}": + scimap: $VERSION +END_VERSIONS +``` + +If you must chain tools (for example piping or compression), define additional `args2`, `args3`, and so on, one per command in the pipe. + +Do not use custom `meta` elements in modules. + +:::info{title="Behind the scenes" collapse} +Custom meta fields are not standardised across tools and create extra work for pipeline developers. All optional parameters for dynamic command construction should come from `ext.args`. +::: + +:::warning +This chapter does not include explicit `script` block examples because tool commands vary widely. Browse the [nf-core/modules GitHub repository](https://github.com/nf-core/modules) for reference modules. +::: + +### The `stub` block + +The `stub` block simulates the module's output during a `-dry-run` of a pipeline. + +```nextflow + stub: + def args = task.ext.args ?: '' + def prefix = task.ext.prefix ?: "${meta.id}" + """ + touch ${prefix}.bam + + cat <<-END_VERSIONS > versions.yml + "${task.process}": + drep: \$(samtools --version |& sed '1!d ; s/samtools //') + END_VERSIONS + """ +``` + +The stub block should mirror the `script` block, except instead of running the tool, it uses `touch` to create empty files matching every output name. For gzipped outputs, pipe an empty echo into `gzip`: + +```nextflow +stub: +def args = task.ext.args ?: '' +def prefix = task.ext.prefix ?: "${meta.id}" +""" +echo "" | gzip > ${prefix}.txt.gz + +cat <<-END_VERSIONS > versions.yml +"${task.process}": + tool: \$(samtools --version |& sed '1!d ; s/samtools //') +END_VERSIONS +""" +``` + +Create files that every `output` channel can pick up. + +If you edit in VS Code and see [Nextflow language server](https://www.nextflow.io/docs/latest/vscode.html) errors, either copy the command from the `script` block and wrap it in `echo`, or remove the `args` definition: + +```nextflow +stub: +def args = task.ext.args ?: '' +def prefix = task.ext.prefix ?: "${meta.id}" +""" +echo "$args" +echo "" | gzip > ${prefix}.txt.gz + +cat <<-END_VERSIONS > versions.yml +"${task.process}": +tool: \$(samtools --version |& sed '1!d ; s/samtools //') +END_VERSIONS +""" +``` + +Keep the `versions.yml` HEREDOC command. It runs during the dry run. + +## The `meta.yml` file + +The `meta.yml` file documents the module with descriptions, keywords, and links to the tool's resources. This information powers the searchable [modules page](https://nf-co.re/modules), improves discoverability on external databases, and supports future automated linkage between modules. + +The file has four main sections: + +- Tool and module description. +- Input channel descriptions. +- Output channel descriptions. +- Contributors. + +:::info{title="Click here to see full 'raw' file example" collapse} + +The boilerplate `TODO` comments have been removed for readability. + +```yaml +--- +# yaml-language-server: $schema=https://raw.githubusercontent.com/nf-core/modules/master/modules/meta-schema.json +name: "drep_compare" +description: write your description here +keywords: + - sort + - example + - genomics +tools: + - "drep": + description: "De-replication of microbial genomes assembled from multiple samples" + homepage: "https://drep.readthedocs.io/en/latest/" + documentation: "https://drep.readthedocs.io/en/latest/" + tool_dev_url: "https://github.com/MrOlm/drep" + doi: "" + licence: ["MIT"] + identifier: biotools:drep + +input: + - - meta: + type: map + description: | + Groovy Map containing sample information + e.g. `[ id:'sample1', single_end:false ]` + - bam: + type: file + description: Sorted BAM/CRAM/SAM file + pattern: "*.{bam,cram,sam}" + ontologies: + - edam: "http://edamontology.org/format_25722" + - edam: "http://edamontology.org/format_2573" + - edam: "http://edamontology.org/format_3462" + +output: + - bam: + - meta: + type: map + description: | + Groovy Map containing sample information + e.g. `[ id:'sample1', single_end:false ]` + - "*.bam": + type: file + description: Sorted BAM/CRAM/SAM file + pattern: "*.{bam,cram,sam}" + ontologies: + - edam: "http://edamontology.org/format_25722" + - edam: "http://edamontology.org/format_2573" + - edam: "http://edamontology.org/format_3462" + + - versions: + - "versions.yml": + type: file + description: File containing software versions + pattern: "versions.yml" + +authors: + - "@jfy133" +maintainers: + - "@jfy133" +``` + +::: + +### Tool and module description + +This section holds metadata for finding the module on the nf-core website. + +```yaml +name: "drep_compare" +description: write your description here +keywords: + - sort + - example + - genomics +tools: + - "drep": + description: "De-replication of microbial genomes assembled from multiple samples" + homepage: "https://drep.readthedocs.io/en/latest/" + documentation: "https://drep.readthedocs.io/en/latest/" + tool_dev_url: "https://github.com/MrOlm/drep" + doi: "" + licence: ["MIT"] + identifier: biotools:drep +``` + +Much of this section is pre-filled from the module name and the Bioconda recipe referenced in `environment.yml`. + +What you need to update: + +- **`description`** — a short sentence describing the module's purpose. +- **`keywords`** — a minimum of three, all lowercase. Include the tool name, subcommand, the action the module performs (such as `sort` or `filter`), the research field (such as `genomics` or `metagenomics`), and the file formats involved. More keywords improve discoverability. +- **URLs** — fill in source code and documentation links. Leave empty or duplicate where the tool has no dedicated documentation page. +- **Multiple tools** — if your module uses more than one tool (for example the tool and `gzip`), add a description block for each. + +### Input channel descriptions + +This section describes every input channel. + +```yaml +input: + # Only when we have meta + - - meta: + type: map + description: | + Groovy Map containing sample information + e.g. `[ id:'sample1', single_end:false ]` + + - bam: + type: file + description: Sorted BAM/CRAM/SAM file + pattern: "*.{bam,cram,sam}" + ontologies: + - edam: "http://edamontology.org/format_25722" + - edam: "http://edamontology.org/format_2573" + - edam: "http://edamontology.org/format_3462" +``` + +The boilerplate template shows a single input channel with a meta map and a path element. + +For each channel, update: + +- **Name** — match the input channel variable. +- **`type`** — one of the [fixed categories](https://nf-co.re/docs/specifications/components/modules/input-output-options) (`file`, `integer`, `boolean`, and so on). +- **`description`** — describe the contents or preparation, for example "A TSV file containing 5 columns generated by XYZ", not just "TSV". +- **`pattern`** — match the input the tool expects. +- **`ontologies`** — link to matching entries in a controlled vocabulary, typically the [EDAM ontology](https://www.ebi.ac.uk/ols4/ontologies/edam). Choose the level of specificity that suits your input. + +Add one entry per input channel. When a channel has its own meta map, rename it to `meta2`, `meta3`, and so on. + +:::tip +Run `nf-core modules lint` to auto-populate placeholders for every input and output channel. +::: + +:::tip{title="Examples" collapse} +A complete `meta.yml` with ontologies and multiple meta entries (see below for `outputs:`): + +```yml +name: bwamem2_mem +description: Performs fastq alignment to a fasta reference using BWA +keywords: + - mem + - bwa + - alignment + - map + - fastq + - bam + - sam +tools: + - bwa: + description: | + BWA-mem2 is a software package for mapping DNA sequences against + a large reference genome, such as the human genome. + homepage: https://github.com/bwa-mem2/bwa-mem2 + documentation: http://www.htslib.org/doc/samtools.html + arxiv: arXiv:1303.3997 + licence: ["MIT"] + identifier: "biotools:bwa-mem2" +input: + - - meta: + type: map + description: | + Groovy Map containing sample information + e.g. [ id:'test', single_end:false ] + - reads: + type: file + description: | + List of input FastQ files of size 1 and 2 for single-end and paired-end data, + respectively. + ontologies: + - edam: "http://edamontology.org/data_2044" # Sequence + - edam: "http://edamontology.org/format_1930" # FASTQ + - - meta2: + type: map + description: | + Groovy Map containing reference/index information + e.g. [ id:'test' ] + - index: + type: file + description: BWA genome index files + pattern: "Directory containing BWA index *.{0132,amb,ann,bwt.2bit.64,pac}" + ontologies: + - edam: "http://edamontology.org/data_3210" # Genome index + - - meta3: + type: map + description: | + Groovy Map containing reference information + e.g. [ id:'genome' ] + - fasta: + type: file + description: Reference genome in FASTA format + pattern: "*.{fa,fasta,fna}" + ontologies: + - edam: "http://edamontology.org/data_2044" # Sequence + - edam: "http://edamontology.org/format_1929" # FASTA + - - sort_bam: + type: boolean + description: use samtools sort (true) or samtools view (false) + pattern: "true or false" +output: + - sam: + - meta: + type: map + description: | + Groovy Map containing sample information + e.g. [ id:'test', single_end:false ] + - "*.sam": + type: file + description: Output SAM file containing read alignments + pattern: "*.{sam}" + ontologies: + - edam: "http://edamontology.org/format_2573" # SAM + - bam: + - meta: + type: map + description: | + Groovy Map containing sample information + e.g. [ id:'test', single_end:false ] + - "*.bam": + type: file + description: Output BAM file containing read alignments + pattern: "*.{bam}" + ontologies: + - edam: "http://edamontology.org/format_2572" # BAM + - cram: + - meta: + type: map + description: | + Groovy Map containing sample information + e.g. [ id:'test', single_end:false ] + - "*.cram": + type: file + description: Output CRAM file containing read alignments + pattern: "*.{cram}" + ontologies: + - edam: "http://edamontology.org/format_3462" # CRAM + - crai: + - meta: + type: map + description: | + Groovy Map containing sample information + e.g. [ id:'test', single_end:false ] + - "*.crai": + type: file + description: Index file for CRAM file + pattern: "*.{crai}" + - csi: + - meta: + type: map + description: | + Groovy Map containing sample information + e.g. [ id:'test', single_end:false ] + - "*.csi": + type: file + description: Index file for BAM file + pattern: "*.{csi}" + - versions: + - versions.yml: + type: file + description: File containing software versions + pattern: "versions.yml" +authors: + - "@maxulysse" + - "@matthdsm" +maintainers: + - "@maxulysse" + - "@matthdsm" +``` + +::: + +### Output channel descriptions + +This section follows the same pattern as inputs. + +```yaml +output: + - bam: + #Only when we have meta + - meta: + type: map + description: | + Groovy Map containing sample information + e.g. `[ id:'sample1', single_end:false ]` + - "*.bam": + type: file + description: Sorted BAM/CRAM/SAM file + pattern: "*.{bam,cram,sam}" + ontologies: + - edam: "http://edamontology.org/format_25722" + - edam: "http://edamontology.org/format_2573" + - edam: "http://edamontology.org/format_3462" +``` + +The only difference: the output file key (not the channel name) must match the pattern used in the output channel in `main.nf`. + +Update the `type`, `description`, `pattern`, and `ontologies` as for inputs. + +### Contributors + +This section lists the original author and any subsequent contributors. + +```yaml +authors: + - "@jfy133" +maintainers: + - "@jfy133" +``` + +The boilerplate populates this automatically. You do not need to modify it. + +If you later update a module originally written by someone else, add yourself to `maintainers:`. + +## Further reading + +This chapter cannot cover every specification detail or edge case. Always refer to the [nf-core module specifications](https://nf-co.re/docs/specifications/components/modules/general) when writing a module. + +The next chapter covers unit testing the module with nf-test. \ No newline at end of file diff --git a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/6-testing.md b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/6-testing.md new file mode 100644 index 0000000000..a348ed649b --- /dev/null +++ b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/6-testing.md @@ -0,0 +1,604 @@ +--- +title: "Chapter 6: Testing your module" +subtitle: "How to test an nf-core module" +shortTitle: "Chapter 6: Testing modules" +--- + +This chapter describes how to unit test an nf-core module with the [nf-test](https://www.nf-test.com/) framework. + +:::warning +Setting up and debugging tests is a topic of its own. This chapter covers the basics to get you started. Tests are required for all nf-core modules, so do not be discouraged if you need multiple attempts to get them right. +::: + +## The `main.nf.test` file + +The `main.nf.test` file has three main sections: + +- File paths and tags. +- Test block, made up of a `when` block (inputs) and a `then` block (assertions). +- Setup block (optional). + +:::info{title="Click here to see full 'raw' file example" collapse} + +The boilerplate `TODO` comments have been removed for readability. + +```nextflow +nextflow_process { + + name "Test Process DREP_COMPARE" + script "../main.nf" + process "DREP_COMPARE" + + tag "modules" + tag "modules_nfcore" + tag "drep" + tag "drep/compare" + + test("sarscov2 - bam") { + + when { + process { + """ + input[0] = [ + [ id:'test', single_end:false ], // meta map + file(params.modules_testdata_base_path + 'genomics/sarscov2/illumina/bam/test.paired_end.sorted.bam', checkIfExists: true), + ] + """ + } + } + + then { + assertAll( + { assert process.success }, + { assert snapshot(process.out).match() } + ) + } + + } + + test("sarscov2 - bam - stub") { + + options "-stub" + + when { + process { + """ + input[0] = [ + [ id:'test', single_end:false ], // meta map + file(params.modules_testdata_base_path + 'genomics/sarscov2/illumina/bam/test.paired_end.sorted.bam', checkIfExists: true), + ] + """ + } + } + + then { + assertAll( + { assert process.success }, + { assert snapshot(process.out).match() } + ) + } + + } + +} + +``` + +::: + +### File paths and tags + +This section is pre-filled with the module's metadata. + +```nextflow +nextflow_process { + + name "Test Process DREP_COMPARE" + script "../main.nf" + process "DREP_COMPARE" + + tag "modules" + tag "modules_nfcore" + tag "drep" + tag "drep/compare" +``` + +It contains: + +- An umbrella test name derived from the module name. +- The script location and process name. +- Auto-generated tags that nf-test uses to group tests. + +You rarely need to edit this section. You may need to extend it to: + +- Add a path to an optional `nextflow.config` (see below). +- Add tags for modules used in a `setup` block, so that upstream module changes do not silently break your test. + +### The `test` block + +The test block declares inputs and assertions. You will spend most of your time here. + +```nextflow +test("sarscov2 - bam") { + when { + process { + """ + input[0] = [ + [ id:'test', single_end:false ], // meta map + file(params.modules_testdata_base_path + 'genomics/sarscov2/illumina/bam/test.paired_end.sorted.bam', checkIfExists: true), + ] + """ + } + } + + then { + assertAll( + { assert process.success }, + { assert snapshot(process.out).match() } + ) + } +} +``` + +A test block has three parts: + +- The test title. +- The `when` block (inputs). +- The `then` block (assertions). + +The boilerplate provides one test with a single input file plus a stub-run test. All nf-core modules need a stub-run test. You only need to update its title and inputs to match the first test. + +#### `when` block + +Update the test title to make it distinct. At a minimum include the organism of the test data and the file format of the primary input, for example: + +```nextflow +test("sarscov2 - fastq - pairedend") +``` + +or + +```nextflow +test("sarscov2 - fastq - bakta annotation input") +``` + +In the `when` block, declare each input channel using `input[index]` notation (`input[0]` for the first, `input[1]` for the second, and so on). The block accepts standard Nextflow code — channel factories, operators, and so on — as you would use in a pipeline. Make sure each `input[]` matches the channel structure of your module, including any meta map: + +```nextflow +when { + process { + """ + ch_samplesheet = Channel.of([ + [ id:'test' ], + file(params.modules_testdata_base_path + 'genomics/homo_sapiens/array_expression/GSE38751.csv', checkIfExists: true) + ] + ) + input[0] = ch_samplesheet.join(UNTAR.out.untar) + input[1] = [[],[]] + """ + } +} +``` + +Key rules: + +- Reference all input files via the [nf-core/test-datasets repository](https://github.com/nf-core/test-datasets), using `params.modules_testdata_base_path` with `checkIfExists`. +- For optional input channels, pass `[]`, or `[[],[]]` if a meta map is required. +- To reference output from a `setup` block, use `PROCESS_NAME.out.foo`. + +#### `then` block + +The `then` block holds assertions — what nf-test compares between runs. The boilerplate checks the contents of every output channel by default. + +Some tools produce variable outputs (logs with timestamps, for example). You can expand `snapshot()` to reference specific channels and assert them in different ways: + +```nextflow +then { + assertAll( + { assert process.success }, + { assert snapshot( + process.out.metrics, + process.out.versions, + file(process.out.qc_report[0][1]).name, + ).match() + } + ) +} +``` + +Here, `metrics` and `versions` have stable md5 sums and use the default snapshot check. The `qc_report` varies, so we only assert that the filename is unchanged. + +For a full list of nf-test assertions, see the [nf-core documentation on assertions](https://nf-co.re/docs/developing/testing/assertions). + +Test outputs using these methods, in order of preference: + +1. md5 sums. +2. String contents within a file. +3. File name or existence. + +If you vary assertion types per channel, always include `process.out.versions`. + +Once you have completed one test, copy its structure for each additional test case. Write as many tests as needed to cover every configuration, and at a minimum test every mandatory and optional input and output channel once. + +### The `setup` block (optional) + +The `setup` block runs one or more modules before your new module. The outputs of these upstream modules can then feed into your test. The setup block is not included by default. + +```nextflow + setup { + run ("UNTAR") { + script "../../../untar/main.nf" + process { + """ + input[0] = [[],file(params.modules_testdata_base_path + 'genomics/sarscov2/genome/db/kraken2_bracken.tar.gz', checkIfExists: true)] + """ + } + } + } +``` + +Placement options: + +- Before all tests, to reuse the upstream output across every test. +- Inside a test block, to run the setup only for that test. + +Outputs produced in a setup block are not asserted against. + +Fill in the setup block like a test block, but include the script path of the upstream module. Use the same `input[0]`, `input[1]` syntax. + +We discourage setup blocks because they increase runtime. They are useful when the module needs inputs too large for the nf-core/test-datasets repository, or when the upstream module runs quickly. + +:::tip{title="Examples" collapse} +A global setup block, reused in every test: + +```nextflow +nextflow_process { + + name "Test Process ADAPTERREMOVALFIXPREFIX" + script "../main.nf" + process "ADAPTERREMOVALFIXPREFIX" + + tag "modules" + tag "modules_nfcore" + tag "adapterremoval" + tag "adapterremovalfixprefix" + + setup { + run("ADAPTERREMOVAL") { + script "../../adapterremoval/main.nf" + config "./nextflow.config" + process { + """ + input[0] = [ + [ id:'test', single_end:false ], // meta map + [ + file(params.modules_testdata_base_path + 'genomics/sarscov2/illumina/fastq/test_1.fastq.gz', checkIfExists: true), + file(params.modules_testdata_base_path + 'genomics/sarscov2/illumina/fastq/test_2.fastq.gz', checkIfExists: true) + ] + ] + input[1] = [] + """ + } + } + } + + test("paired-end - sarscov2 - [fastq]") { + + when { + + process { + """ + input[0] = ADAPTERREMOVAL.out.collapsed + """ + } + } + + then { + assertAll( + { assert process.success }, + { assert snapshot(process.out).match() } + ) + } + + } +... +``` + +A setup block scoped to a specific test: + +```nextflow +nextflow_process { + + name "Test Process BCFTOOLS_PLUGINIMPUTEINFO" + script "../main.nf" + process "BCFTOOLS_PLUGINIMPUTEINFO" + + tag "modules" + tag "modules_nfcore" + tag "bcftools" + tag "bcftools/pluginimputeinfo" + tag "bcftools/plugintag2tag" + + test("sarscov2 - [vcf, tbi], [], []") { + + config "./nextflow.config" + + setup { + run("BCFTOOLS_PLUGINTAG2TAG") { + script "../../plugintag2tag/main.nf" + process { + """ + input[0] = [ + [ id:'out', single_end:false ], // meta map + file(params.modules_testdata_base_path + 'genomics/sarscov2/illumina/vcf/test.vcf.gz', checkIfExists: true), + file(params.modules_testdata_base_path + 'genomics/sarscov2/illumina/vcf/test.vcf.gz.tbi', checkIfExists: true) + ] + input[1] = [] + input[2] = [] + """ + } + } + } + + when { + process { + """ + input[0] = BCFTOOLS_PLUGINTAG2TAG.out.vcf.join(BCFTOOLS_PLUGINTAG2TAG.out.tbi) + input[1] = [] + input[2] = [] + """ + } + } + + then { + assertAll( + { assert process.success }, + { assert process.success }, + { assert snapshot( + process.out.vcf, + process.out.versions + ).match() } + ) + } + + } +``` + +::: + +## The `nextflow.config` file (optional) + +The optional `nextflow.config` file is not generated by the boilerplate. It sits alongside `main.nf.test` and `main.nf.test.snap` in the `tests/` directory. + +Use it to provide extra settings, for example `ext.args` for test configurations, optional parameters that trigger extra outputs, or per-import settings when a module is imported more than once in setup blocks. + +Create the file yourself when you need it: + +```tree {7} +├── environment.yml +├── main.nf +├── meta.yml +└── tests/ + ├── main.nf.test + ├── main.nf.test.snap + └── nextflow.config +``` + +Reference it in the test file: + +```nextflow +config './nextflow.config' +``` + +Place the line next to the `script` and `tags` declarations, or inside a test block before its `when` scope. + +:::tip{title="Examples" collapse} +A global `nextflow.config` for all tests: + +```nextflow +process { + ext.args = '--tbi' +} +``` + +Loaded at the top of the test file: + +```nextflow +nextflow_process { + + name "Test Process BCFTOOLS_INDEX" + script "../main.nf" + config "./nextflow.config" + process "BCFTOOLS_INDEX" + + tag "modules" + tag "modules_nfcore" + tag "bcftools" + tag "bcftools/index" + +... + +``` + +Or loaded inside a single test: + +```nextflow + test("sarscov2 - paired-end - fastq") { + + config "./nextflow.config" + + setup { +... +``` + +::: + +## The `main.nf.test.snap` file + +The `main.nf.test.snap` file is not part of the boilerplate. nf-test generates and updates it automatically the first time you run the test. + +```tree {6} +├── environment.yml +├── main.nf +├── meta.yml +└── tests/ + ├── main.nf.test + ├── main.nf.test.snap + └── nextflow.config +``` + +Generate it with: + +```bash +nf-core modules test / +``` + +Set the prompts, and the tests run twice. nf-test reports whether the outputs match and creates the snapshot on the first run. + +The snapshot records what every later test run is compared against, such as md5 sums or string matches. + +You never need to edit this file manually. If you need to regenerate it: + +```bash +nf-core modules test / --update +``` + +Add `--verbose` to see the full nf-test output. + +Inspect the snapshot after generation. Even if tests pass, look for unexpected values such as empty file md5 sums or missing assertions. + +To inspect a module's working directory, note the hash shown next to each test name: + +```bash +│ Test [528b411a] 'candidatus_portiera_aleyrodidarum proteome [fasta]' │ +│ PASSED (7.762s) +``` + +Change into `.nf-test/tests//work` (use TAB to autocomplete) to find the Nextflow working directories for that test. Each subdirectory contains the outputs of one process run, so you can verify the contents of every output file. + +Empty entries in the snapshot may be legitimate when an optional channel does not emit. Always verify this is expected for the test. + +Assertions outside the `snapshot()` function are not recorded in the snapshot. These are usually boolean checks such as file existence, and nf-test prints failures to the console instead. + +:::tip{title="Examples" collapse} + +```nextflow +{ + "sarscov2 nanopore [fastq_gz]": { + "content": [ + [ + [ + { + "id": "test" + }, + [ + "test.fastq_read-assignment-distributions.tsv:md5,c5c52bef375dee47407b3711b147b61d", + "test.fastq_rel-abundance.tsv:md5,6fb86cb4103ae5064bb2b7b43e9c9c24" + ] + ] + ], + [ + [ + { + "id": "test" + }, + "test.fastq_read-assignment-distributions.tsv:md5,c5c52bef375dee47407b3711b147b61d" + ] + ], + [ + [ + { + "id": "test" + }, + "test.fastq_emu_alignments.sam:md5,09f832656b47f44ff22f65ff7ac87824" + ] + ], + [ + "versions.yml:md5,5227d63df8748de6a8b0c3f212a79512" + ] + ], + "meta": { + "nf-test": "0.9.2", + "nextflow": "24.10.4" + }, + "timestamp": "2025-02-17T15:48:16.876032268" + }, + "sarscov2 nanopore [fastq_gz] - stub": { + "content": [ + { + "0": [ + [ + { + "id": "test" + }, + "test.tsv:md5,d41d8cd98f00b204e9800998ecf8427e" + ] + ], + "1": [ + + ], + "2": [ + [ + { + "id": "test" + }, + "test.sam:md5,d41d8cd98f00b204e9800998ecf8427e" + ] + ], + "3": [ + [ + { + "id": "test" + }, + "test.fasta:md5,d41d8cd98f00b204e9800998ecf8427e" + ] + ], + "4": [ + "versions.yml:md5,5227d63df8748de6a8b0c3f212a79512" + ], + "assignment_report": [ + + ], + "report": [ + [ + { + "id": "test" + }, + "test.tsv:md5,d41d8cd98f00b204e9800998ecf8427e" + ] + ], + "samfile": [ + [ + { + "id": "test" + }, + "test.sam:md5,d41d8cd98f00b204e9800998ecf8427e" + ] + ], + "unclassified_fa": [ + [ + { + "id": "test" + }, + "test.fasta:md5,d41d8cd98f00b204e9800998ecf8427e" + ] + ], + "versions": [ + "versions.yml:md5,5227d63df8748de6a8b0c3f212a79512" + ] + } + ], + "meta": { + "nf-test": "0.9.2", + "nextflow": "24.10.4" + }, + "timestamp": "2025-02-17T15:48:24.167181483" + } +} +``` + +::: + +Once your snapshot is stable, your module is ready for use. + +The next chapter summarises the development workflow and the steps for contributing to the [nf-core/modules](https://github.com/nf-core/modules) repository. \ No newline at end of file diff --git a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/7-development.md b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/7-development.md new file mode 100644 index 0000000000..831c87f12c --- /dev/null +++ b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/7-development.md @@ -0,0 +1,89 @@ +--- +title: "Chapter 7: Development workflow" +subtitle: "How to contribute to the community" +shortTitle: "Chapter 7: Development workflow" +--- + +This chapter outlines the workflow for contributing to the [nf-core/modules repository](https://github.com/nf-core/modules/). + +This training used a fork of that repository, but you can apply everything you have learnt to your own private module repository or to local modules inside a pipeline. The nf-core community encourages you to contribute modules upstream so others can benefit. + +## Check + +Before writing a module, check that someone has not already built it or started on it: + +1. Search the [modules page](https://nf-co.re/modules) on the nf-core website. +2. Search [open pull requests](https://github.com/nf-core/modules/pulls) on the nf-core/modules repository. +3. Search [open issues](https://github.com/nf-core/modules/issues) on the nf-core/modules repository. + +Based on what you find: + +- **Module exists**: skip the writing and install it into your pipeline. +- **Open PR**: offer to help out, or take over if development has stalled. +- **Open issue**: assign yourself, or ask the current assignee if they want help. +- **Nothing yet**: [open an issue](https://github.com/nf-core/modules/issues) to signal you are working on it. + +Once you claim a module, fork and clone the repository, create a new branch, and follow the previous chapters to write and test the module. + +## Write + +Generate the boilerplate template files: + +```bash +nf-core modules create / +``` + +Fill in each file by following the `TODO` comments, the [nf-core module specifications](https://nf-co.re/docs/specifications/components/modules/general), and this training material. + +## Test + +Run the tests and generate the snapshot: + +```bash +nf-core modules test / +``` + +Check the snapshot file looks correct before moving on. + +## Lint + +Standardisation is central to nf-core. The linting tool flags issues before submission: + +```bash +nf-core modules lint / +``` + +The console output lists any issues to fix. Some issues can be auto-fixed with `--fix`, in which case the linter prints the command to run. + +Check each fix carefully. The linter may add empty sections that you still need to populate. + +:::note +Linting is optional for private custom modules, but still recommended to catch small issues. +::: + +:::warning +The linter cannot check every aspect of the [nf-core module specification](https://nf-co.re/docs/specifications/components/modules/general). Review the specification yourself before submission. +::: + +## Submit + +Commit and push your module to your fork: + +```bash +git commit -am "Add new module /" +git push +``` + +Open a [pull request](https://github.com/nf-core/modules/pulls) from your fork and branch to the nf-core/modules repository. + +To attract a reviewer, ask in the nf-core Slack [#request-review](https://nfcore.slack.com/archives/CQY2U5QU9) channel. See the [join instructions](https://nf-co.re/join) if you are not already on Slack. + +If a reviewer asks for changes, do not close the PR. Push updates to the same branch, then request another review. + +Once the PR is approved, merge it and the module is available to everyone. + +If you get stuck, ask on the [#modules](https://nfcore.slack.com/archives/CJRH30T6V) channel or the [#nostupidquestions](https://nfcore.slack.com/archives/C043FMKUNLB) channel. + +Thank you for contributing to the community. + +The final chapter covers how to use modules in your own pipelines. \ No newline at end of file diff --git a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/8-using.md b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/8-using.md new file mode 100644 index 0000000000..fec47796fe --- /dev/null +++ b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/8-using.md @@ -0,0 +1,157 @@ +--- +title: "Chapter 8: Using nf-core modules in pipelines" +subtitle: "How to use nf-core modules in a pipeline" +shortTitle: "Chapter 8: Using modules" +--- + +This chapter explains how to install an nf-core module in a pipeline and what else to consider when using it. + +It assumes you have already contributed your module to the nf-core/modules repository or a personal or organisation-specific module repository, and that the module has been merged in. + +## Installing nf-core modules + +### Pipelines using the nf-core template + +If you generated your pipeline with [nf-core tools](https://nf-co.re/docs/nf-core-tools/pipelines/create) ([whether the pipeline is official or not](https://nf-co.re/docs/developing/pipelines/external-use)), you can install a module directly. Run the following from the root of the repository, or pass `--dir`: + +```bash +nf-core modules install / +``` + +This downloads the module files into the existing `modules/nf-core` directory. The console output prints an `include` line you can paste into your pipeline script. + +### Custom pipelines + +You can use the same command in a custom pipeline. On the first run you will be prompted to: + +1. Select `pipeline` when asked whether this is a `pipeline` or `modules` repository. +2. Accept creation of an `nf-core.yaml` file. This supports adding more modules later. +3. Accept creation of a `modules.json` file. This tracks installed module versions for future updates. + +The command creates `modules/nf-core//`, a `modules.json`, and an `.nf-core.yml` configuration file. The console output prints an `include` line for your pipeline script. + +:::warning +If installing a second module returns `ERROR 'manifest.name'`, add a `nextflow.config` with a [`manifest` scope](https://www.nextflow.io/docs/latest/reference/config.html#manifest) that includes `name`, `description`, and `version`. +::: + +## Using nf-core modules + +You can use an installed nf-core module as you would any Nextflow module. + +Paste the `include` line printed by the install command into the `.nf` file where you want to use the module: + +```nextflow +include { SAMTOOLS_FASTA } from '../modules/nf-core/samtools/fasta/main' +``` + +Adjust the path if you are using the module from a subworkflow in a nested directory. + +Invoke it as you would any Nextflow module: + +```nextflow +SAMTOOLS_FASTA (ch_input_for_samtoolsfasta, val_interleave) +``` + +Most pipelines need a few additional adjustments, covered below. + +### Module channel structures + +Reconfigure your input channels to match nf-core conventions. + +The most important convention is the use of [meta maps](https://nf-co.re/docs/developing/components/meta-map). If your pipeline does not use meta maps, add one before passing data to the module: + +```nextflow +def val_interleave = false +ch_input_for_samtoolsfasta = ch_input + .map {fasta -> [[id: fasta.simpleName] ,fasta]} + +SAMTOOLS_FASTA (ch_input_for_samtoolsfasta, val_interleave) +``` + +Here, the original pipeline channel contained only FASTA files. The module needs a meta map, so `.map {}` creates one with a single `id` attribute assigned from the file's [`simpleName`](https://www.nextflow.io/docs/latest/reference/stdlib.html#stdlib-types-path). + +### Tags and labels + +#### Tags + +Most nf-core modules set a `tag` directive to produce a human-readable process description in the run log. + +By default, all nf-core modules use the `id` attribute from the meta map as the tag value. Set `id` on every meta map passed to an nf-core module. + +#### Labels + +nf-core modules use standardised `label` directives that map to default memory, CPU, and wall time. + +- **Pipelines from the nf-core template**: defaults are defined in [`conf/base.config`](https://github.com/nf-core/tools/blob/52e810986e382972ffad0aab28e94f828ffd509b/nf_core/pipeline-template/conf/base.config#L22-L54). You do not need to change anything unless you want to override defaults for your pipeline. +- **Custom pipelines**: add a Nextflow config file that defines default resources for the labels used by your installed modules. + +For example, if an installed module uses `process_low`: + +```nextflow +process { + withLabel:process_low { + cpus = { 2 * task.attempt } + memory = { 12.GB * task.attempt } + time = { 4.h * task.attempt } + } +} +``` + +:::tip +You can override default resources for a specific module using `withName:` in the same process scope. +::: + +### The `modules.conf` file + +Pass optional tool arguments to modules through a [`modules.config` file](https://github.com/nf-core/tools/blob/52e810986e382972ffad0aab28e94f828ffd509b/nf_core/pipeline-template/conf/modules.config), which the nf-core pipeline template uses to inject pipeline-level parameters. + +For each installed module, specify a `withName:` entry with at least `ext.args` and `publishDir`. + +For example: + +```nextflow {3} +process{ + withName: FASTQC { + ext.args = '--quiet' + publishDir = [ + path: { "${params.outdir}/fastqc" }, + saveAs: { filename -> filename.equals('versions.yml') ? null : filename } + ] + } +} +``` + +This hardcodes FastQC's `--quiet` flag for every invocation and sets the default output directory under `--outdir`. It also excludes the required `versions.yml` file from publication, because nf-core pipelines render versions through [MultiQC](https://multiqc.info/). + +Pass arguments dynamically through a closure to expose pipeline parameters to users: + +```nextflow {3} +process{ + withName: FASTQC { + ext.args = { params.fastqc_quiet_mode ? '--quiet' : '' } + publishDir = [ + path: { "${params.outdir}/fastqc" }, + saveAs: { filename -> filename.equals('versions.yml') ? null : filename } + ] + } +} +``` + +Combine multiple values into the `args` string by joining a list with spaces: + +```nextflow +process{ + withName: FASTQC { + ext.args = { [ + params.fastqc_quiet_mode ? '--quiet' : '', + "--kmers ${params.fastqc_kmers}" + ].join(' ').trim() } + publishDir = [ + path: { "${params.outdir}/fastqc" }, + saveAs: { filename -> filename.equals('versions.yml') ? null : filename } + ] + } +} +``` + +With these steps, you can now use nf-core modules in your pipeline and benefit from the standardised, reproducible, and tested modules the community maintains. \ No newline at end of file diff --git a/sites/main-site/public/_redirects b/sites/main-site/public/_redirects index ace43531fa..cfecbd320c 100644 --- a/sites/main-site/public/_redirects +++ b/sites/main-site/public/_redirects @@ -200,6 +200,17 @@ /docs/tutorials/nf-core_components/writing_automated_methods_descriptions https://nf-core-docs.netlify.app/docs/developing/components/automated-methods 200 # NOTE: nf-core_components/components.mdx has no mapping yet - skip +# Tutorials - nf-core training (writing nf-core modules) +/docs/tutorials/nf-core_training/writing-nf-core-modules https://nf-core-docs.netlify.app/docs/developing/tutorials/writing-nf-core-modules/1-introduction 200 +/docs/tutorials/nf-core_training/writing-nf-core-modules/chapter-1-introduction https://nf-core-docs.netlify.app/docs/developing/tutorials/writing-nf-core-modules/1-introduction 200 +/docs/tutorials/nf-core_training/writing-nf-core-modules/chapter-2-getting-started https://nf-core-docs.netlify.app/docs/developing/tutorials/writing-nf-core-modules/2-getting-started 200 +/docs/tutorials/nf-core_training/writing-nf-core-modules/chapter-3-what-is-a-nf-core-module https://nf-core-docs.netlify.app/docs/developing/tutorials/writing-nf-core-modules/3-what-is-a-module 200 +/docs/tutorials/nf-core_training/writing-nf-core-modules/chapter-4-generating-boilerplate-files https://nf-core-docs.netlify.app/docs/developing/tutorials/writing-nf-core-modules/4-boilerplate 200 +/docs/tutorials/nf-core_training/writing-nf-core-modules/chapter-5-writing-an-nf-core-module https://nf-core-docs.netlify.app/docs/developing/tutorials/writing-nf-core-modules/5-writing-modules 200 +/docs/tutorials/nf-core_training/writing-nf-core-modules/chapter-6-testing-an-nf-core-module https://nf-core-docs.netlify.app/docs/developing/tutorials/writing-nf-core-modules/6-testing 200 +/docs/tutorials/nf-core_training/writing-nf-core-modules/chapter-7-development-workflow https://nf-core-docs.netlify.app/docs/developing/tutorials/writing-nf-core-modules/7-development 200 +/docs/tutorials/nf-core_training/writing-nf-core-modules/chapter-8-using-in-pipelines https://nf-core-docs.netlify.app/docs/developing/tutorials/writing-nf-core-modules/8-using 200 + # Tutorials - Pipelines /docs/tutorials/pipelines/switching_master_to_main https://nf-core-docs.netlify.app/docs/developing/pipelines/master_to_main 200 From 6d89ee375c6a49549ead8fd4dc95bd9f66fe432e Mon Sep 17 00:00:00 2001 From: nf-core-bot Date: Sun, 19 Apr 2026 22:14:11 +0000 Subject: [PATCH 03/10] [automated] Fix code linting --- .../tutorials/writing-nf-core-modules/4-boilerplate.md | 2 +- .../tutorials/writing-nf-core-modules/5-writing-modules.md | 4 ++-- .../developing/tutorials/writing-nf-core-modules/6-testing.md | 2 +- .../tutorials/writing-nf-core-modules/7-development.md | 2 +- .../developing/tutorials/writing-nf-core-modules/8-using.md | 2 +- 5 files changed, 6 insertions(+), 6 deletions(-) diff --git a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/4-boilerplate.md b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/4-boilerplate.md index 22d3dc5bd0..45e9f72cff 100644 --- a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/4-boilerplate.md +++ b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/4-boilerplate.md @@ -81,4 +81,4 @@ modules/nf-core/drep/ └── main.nf.test ``` -The next chapter walks through each generated file and explains what to change. \ No newline at end of file +The next chapter walks through each generated file and explains what to change. diff --git a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/5-writing-modules.md b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/5-writing-modules.md index 440e8e4c0f..c89918b26d 100644 --- a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/5-writing-modules.md +++ b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/5-writing-modules.md @@ -33,7 +33,7 @@ nf-core uses a separate Conda file rather than the `main.nf` conda directive for - It enables [automated container building with Seqera's wave infrastructure](https://nf-co.re/blog/2024/seqera-containers-part-1). - It makes multi-tool environments easier to read and manage than multiple conda declarations on a single line. -::: + ::: ## The `main.nf` file @@ -787,4 +787,4 @@ If you later update a module originally written by someone else, add yourself to This chapter cannot cover every specification detail or edge case. Always refer to the [nf-core module specifications](https://nf-co.re/docs/specifications/components/modules/general) when writing a module. -The next chapter covers unit testing the module with nf-test. \ No newline at end of file +The next chapter covers unit testing the module with nf-test. diff --git a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/6-testing.md b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/6-testing.md index a348ed649b..54e1a1b0da 100644 --- a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/6-testing.md +++ b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/6-testing.md @@ -601,4 +601,4 @@ Assertions outside the `snapshot()` function are not recorded in the snapshot. T Once your snapshot is stable, your module is ready for use. -The next chapter summarises the development workflow and the steps for contributing to the [nf-core/modules](https://github.com/nf-core/modules) repository. \ No newline at end of file +The next chapter summarises the development workflow and the steps for contributing to the [nf-core/modules](https://github.com/nf-core/modules) repository. diff --git a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/7-development.md b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/7-development.md index 831c87f12c..6b7b431435 100644 --- a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/7-development.md +++ b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/7-development.md @@ -86,4 +86,4 @@ If you get stuck, ask on the [#modules](https://nfcore.slack.com/archives/CJRH30 Thank you for contributing to the community. -The final chapter covers how to use modules in your own pipelines. \ No newline at end of file +The final chapter covers how to use modules in your own pipelines. diff --git a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/8-using.md b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/8-using.md index fec47796fe..2ff02f0038 100644 --- a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/8-using.md +++ b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/8-using.md @@ -154,4 +154,4 @@ process{ } ``` -With these steps, you can now use nf-core modules in your pipeline and benefit from the standardised, reproducible, and tested modules the community maintains. \ No newline at end of file +With these steps, you can now use nf-core modules in your pipeline and benefit from the standardised, reproducible, and tested modules the community maintains. From fb4e42d0e419cbb38eaa3297b0bffa67f5f0812e Mon Sep 17 00:00:00 2001 From: Chris Hakkaart Date: Wed, 22 Apr 2026 15:10:38 +1200 Subject: [PATCH 04/10] Apply suggestions from code review Co-authored-by: James A. Fellows Yates --- .../writing-nf-core-modules/3-what-is-a-module.md | 8 +++++--- .../writing-nf-core-modules/4-boilerplate.md | 13 +++++++++---- 2 files changed, 14 insertions(+), 7 deletions(-) diff --git a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/3-what-is-a-module.md b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/3-what-is-a-module.md index 4cf222fe29..e23e8fd81f 100644 --- a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/3-what-is-a-module.md +++ b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/3-what-is-a-module.md @@ -4,7 +4,7 @@ subtitle: "What defines an nf-core module" shortTitle: "Chapter 3: What is a module?" --- -An nf-core module is an opinionated, open-source Nextflow wrapper around a single command-line tool or script. The [central community repository](https://github.com/nf-core/modules) holds more than [1000 modules](https://nf-co.re/modules) that you can reuse in your own pipelines. +An nf-core module is an opinionated, open-source Nextflow wrapper around a single command-line tool or script. The [central community repository](https://github.com/nf-core/modules) holds more than [1800 modules](https://nf-co.re/modules) that you can reuse in your own pipelines. This page describes what an nf-core module is and what distinguishes it from a standard Nextflow process. @@ -15,7 +15,8 @@ An nf-core module usually wraps one tool command, or one tool with a single subc :::info{collapse title="Comparison with Nextflow modules"} A Nextflow process or module has no restrictions on the number of tools or subcommands it can execute. -An nf-core module aims for a single analysis command per module. This atomic design supports lego-block construction of pipelines and makes modules easier to understand, share, and test, although it is not always the most resource-efficient approach. +An nf-core module aims for a single analysis command per module. +This atomic design supports lego-block construction of pipelines and makes modules easier to understand, share, and test, although it is not always the most resource-efficient approach. ::: Each nf-core module follows strict [specifications](../../../specifications/components/overview) agreed by community consensus. The specifications cover: @@ -28,7 +29,8 @@ Each nf-core module follows strict [specifications](../../../specifications/comp - Use of [meta maps](../../../specifications/components/modules/general#use-of-meta-maps). - Use of [stubs](../../../specifications/components/modules/general#stubs). -The specifications also expand the minimum file set from a single Nextflow `.nf` script to five required files. This reflects the nf-core focus on standardisation, reproducibility, and high-quality documentation. +The specifications also expand the minimum required single file from a single Nextflow `.nf` script to five required files. +This reflects the nf-core focus on standardisation, reproducibility, and high-quality documentation. A complete nf-core module directory can contain up to six files: diff --git a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/4-boilerplate.md b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/4-boilerplate.md index 45e9f72cff..474b6598a6 100644 --- a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/4-boilerplate.md +++ b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/4-boilerplate.md @@ -22,12 +22,16 @@ From the root of the repository, run: ```bash nf-core modules create / ``` - + +:::note +If your module does not have subcommands, use `` +::: + ### Naming conventions - All parts of the module name must be lowercase, alphanumeric, with no punctuation or special characters. -- Single-command tools use the tool name only. For a tool executed with `fastp -i -o `, run `nf-core modules create fastp`. -- Tools with subcommands use `/`, even if you only plan to wrap one subcommand. For `samtools view`, run `nf-core modules create samtools/view`. +- Single-command tools use the tool name only. For example, for a tool executed with `fastp -i -o `, run `nf-core modules create fastp`. +- Tools with subcommands use `/`, even if you only plan to wrap one subcommand. For example, `samtools view`, run `nf-core modules create samtools/view`. - For a third level of subcommand, append it to the subcommand name. For `samtools view flagstats`, run `nf-core modules create samtools/viewflagstat`. For example, to create a module for the tool `drep` with the subcommand `compare`: @@ -38,7 +42,8 @@ nf-core modules create drep/compare ### Prompts from `nf-core modules create` -The command tries to pre-fill the boilerplate. It searches [Bioconda](https://bioconda.github.io/) and [biocontainers](https://biocontainers.pro/) for the latest version of your tool and adds the container definitions automatically. +The command tries to pre-fill the boilerplate. +It searches [Bioconda](https://bioconda.github.io/) and [biocontainers](https://biocontainers.pro/) for the latest version of your tool and adds the container definitions automatically. You will then be prompted for: From ff1aa911310abbdb3d548c7b732e64529851a17d Mon Sep 17 00:00:00 2001 From: "James A. Fellows Yates" Date: Wed, 22 Apr 2026 09:57:35 +0200 Subject: [PATCH 05/10] Apply suggestions from code review part 1 Co-authored-by: James A. Fellows Yates --- .../writing-nf-core-modules/1-introduction.md | 6 +- .../2-getting-started.md | 2 +- .../3-what-is-a-module.md | 2 + .../writing-nf-core-modules/4-boilerplate.md | 168 ++++++++-------- .../5-writing-modules.md | 183 +++++++++++------- .../writing-nf-core-modules/6-testing.md | 26 ++- 6 files changed, 218 insertions(+), 169 deletions(-) diff --git a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/1-introduction.md b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/1-introduction.md index e414d75c41..e7489ec267 100644 --- a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/1-introduction.md +++ b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/1-introduction.md @@ -10,9 +10,11 @@ Material originally written for the **A practical introduction to nf-core compon Duration: **4hr** Original author: James A. Fellows Yates (@jfy133), with corrections and improvements from Niklas Schandry (@nschan) and Alexandru Mizeranschi (@amizeranschi). +Refreshed style by: Chris Hakkaart (@christopher-hakkaart ) ::: -An nf-core module is an atomic, standardised, reproducible, and tested [Nextflow DSL2 module](https://www.nextflow.io/docs/latest/module.html). You can install an nf-core module into any Nextflow pipeline with a single command, then integrate it into your workflow thanks to its consistent internal structure. +An nf-core module is an atomic, standardised, reproducible, and tested [Nextflow DSL2 module](https://www.nextflow.io/docs/latest/module.html). +You can install an nf-core module into any Nextflow pipeline with a single command, then efficiently integrate it into your workflow thanks to its consistent internal structure. ## Why use nf-core modules @@ -53,5 +55,5 @@ By the end of this training, you will be able to: - Chapter 8: [Using nf-core modules in pipelines](/docs/developing/tutorials/writing-nf-core-modules/8-using) :::tip -Read chapters 2–7 through once before starting development. When you reach chapter 7, return to chapter 2 to begin writing the module itself. +Read chapters [2](./2-getting-started)–[7](./7-development) through once before starting development. When you reach [chapter 7](./7-development), return to [chapter 2](./2-getting-started) to begin writing the module itself. ::: diff --git a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/2-getting-started.md b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/2-getting-started.md index d1f22badcc..225613e71f 100644 --- a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/2-getting-started.md +++ b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/2-getting-started.md @@ -38,4 +38,4 @@ This training assumes you have: This training was written and tested with nf-core/tools 3.2.0 and Nextflow 24.10.4. ::: -The next chapter introduces what defines an nf-core module. +The [next chapter](./3-what-is-a-module) introduces what defines an nf-core module. diff --git a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/3-what-is-a-module.md b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/3-what-is-a-module.md index e23e8fd81f..e62fc7b153 100644 --- a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/3-what-is-a-module.md +++ b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/3-what-is-a-module.md @@ -64,3 +64,5 @@ _Schematic diagram showing the relationship between the three main categories of - `nextflow.config` is an optional Nextflow configuration file used when `main.nf.test` executes. Writing an nf-core module means creating these files and populating them with content that follows the nf-core specifications. + +The [next chapter](./4-boilerplate) goes into more detail about the contents of each file. diff --git a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/4-boilerplate.md b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/4-boilerplate.md index 474b6598a6..04ca33d095 100644 --- a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/4-boilerplate.md +++ b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/4-boilerplate.md @@ -1,89 +1,89 @@ ---- -title: "Chapter 4: Generating boilerplate files" -subtitle: "How to generate the module file skeleton" -shortTitle: "Chapter 4: Boilerplate files" ---- - -This chapter explains how to generate an nf-core module template and what each file contains. - -## Preparation - -Fork and clone the [nf-core/modules GitHub repository](https://github.com/nf-core/modules) to your working environment, then create a new branch for your module: - -```bash -git switch master ## ensure you have the latest state of the repository -git switch -c -``` - -## Generate the boilerplate files - -From the root of the repository, run: - -```bash -nf-core modules create / -``` +--- +title: "Chapter 4: Generating boilerplate files" +subtitle: "How to generate the module file skeleton" +shortTitle: "Chapter 4: Boilerplate files" +--- + +This chapter explains how to generate an nf-core module template and what each file contains. + +## Preparation + +Fork and clone the [nf-core/modules GitHub repository](https://github.com/nf-core/modules) to your working environment, then create a new branch for your module: + +```bash +git switch master ## ensure you have the latest state of the repository +git switch -c +``` + +## Generate the boilerplate files + +From the root of the repository, run: + +```bash +nf-core modules create / +``` :::note If your module does not have subcommands, use `` ::: -### Naming conventions - -- All parts of the module name must be lowercase, alphanumeric, with no punctuation or special characters. -- Single-command tools use the tool name only. For example, for a tool executed with `fastp -i -o `, run `nf-core modules create fastp`. -- Tools with subcommands use `/`, even if you only plan to wrap one subcommand. For example, `samtools view`, run `nf-core modules create samtools/view`. -- For a third level of subcommand, append it to the subcommand name. For `samtools view flagstats`, run `nf-core modules create samtools/viewflagstat`. - -For example, to create a module for the tool `drep` with the subcommand `compare`: - -```bash -nf-core modules create drep/compare -``` - -### Prompts from `nf-core modules create` - -The command tries to pre-fill the boilerplate. -It searches [Bioconda](https://bioconda.github.io/) and [biocontainers](https://biocontainers.pro/) for the latest version of your tool and adds the container definitions automatically. - -You will then be prompted for: - -- **Your GitHub username.** -- **A process resource label.** These standardised tags map to default memory, CPU, and wall time. Choose the label that best matches your tool's typical requirements. The defaults for each label are defined in the pipeline template [base.config](https://github.com/nf-core/tools/blob/52e810986e382972ffad0aab28e94f828ffd509b/nf_core/pipeline-template/conf/base.config#L29-L54). Pipelines can override these values. -- **Whether the module should use a [meta map](https://nf-co.re/docs/developing/components/meta-map).** Answer yes in most cases. Meta maps carry sample metadata alongside files, which pipelines use to drive downstream processing decisions. - -:::tip -You can pass any of these values as command-line flags to skip the prompts. -::: - -## Output - -After the command completes, you will see the following files and directories: - -```tree {8-13} -modules/nf-core/drep/compare/ -├── environment.yml -├── main.nf -├── meta.yml -└── tests - └── main.nf.test -``` - -If you later create a second subcommand (`dereplicate`), the directory structure becomes: - -```tree {8-13} -modules/nf-core/drep/ -├── compare -│ ├── environment.yml -│ ├── main.nf -│ ├── meta.yml -│ └── tests -│ └── main.nf.test -└── dereplicate - ├── environment.yml - ├── main.nf - ├── meta.yml - └── tests - └── main.nf.test -``` - -The next chapter walks through each generated file and explains what to change. +### Naming conventions + +- All parts of the module name must be lowercase, alphanumeric, with no punctuation or special characters. +- Single-command tools use the tool name only. For example, for a tool executed with `fastp -i -o `, run `nf-core modules create fastp`. +- Tools with subcommands use `/`, even if you only plan to wrap one subcommand. For example, `samtools view`, run `nf-core modules create samtools/view`. +- For a third level of subcommand, append it to the subcommand name. For `samtools view flagstats`, run `nf-core modules create samtools/viewflagstat`. + +For example, to create a module for the tool `drep` with the subcommand `compare`: + +```bash +nf-core modules create drep/compare +``` + +### Prompts from `nf-core modules create` + +The command tries to pre-fill the boilerplate. +It searches [Bioconda](https://bioconda.github.io/) and [biocontainers](https://biocontainers.pro/) for the latest version of your tool and adds the container definitions automatically. + +You will then be prompted for: + +- **Your GitHub username.** +- **A process resource label.** These standardised tags map to default memory, CPU, and wall time. Choose the label that best matches your tool's typical requirements. The defaults for each label are defined in the pipeline template [base.config](https://github.com/nf-core/tools/blob/52e810986e382972ffad0aab28e94f828ffd509b/nf_core/pipeline-template/conf/base.config#L29-L54). Pipelines can override these values. +- **Whether the module should use a [meta map](https://nf-co.re/docs/developing/components/meta-map).** Answer yes in most cases. Meta maps carry sample metadata alongside files, which pipelines use to drive downstream processing decisions. + +:::tip +You can pass any of these values as command-line flags to skip the prompts. +::: + +## Output + +After the command completes, you will see the following files and directories: + +```tree {8-13} +modules/nf-core/drep/compare/ +├── environment.yml +├── main.nf +├── meta.yml +└── tests + └── main.nf.test +``` + +If you later create a second subcommand (`dereplicate`), the directory structure becomes: + +```tree {8-13} +modules/nf-core/drep/ +├── compare +│ ├── environment.yml +│ ├── main.nf +│ ├── meta.yml +│ └── tests +│ └── main.nf.test +└── dereplicate + ├── environment.yml + ├── main.nf + ├── meta.yml + └── tests + └── main.nf.test +``` + +The [next chapter](./5-writing-modules) walks through each generated file and explains what to change. diff --git a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/5-writing-modules.md b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/5-writing-modules.md index c89918b26d..4448fc1536 100644 --- a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/5-writing-modules.md +++ b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/5-writing-modules.md @@ -4,7 +4,8 @@ subtitle: "How to fill in the module files" shortTitle: "Chapter 5: Writing modules" --- -Once you have generated the boilerplate template files, you can update them to make your module function. The boilerplate contains many `TODO` comments and example content to guide you. +Once you have generated the boilerplate template files, you can update them to make your module function. +The boilerplate contains many `TODO` comments and example content to guide you. This chapter walks through each file, explaining what each section does and why, as defined by the nf-core specifications. @@ -26,7 +27,10 @@ dependencies: You rarely need to modify this file. The tool and version specified here will usually have a matching biocontainer in the `main.nf` file. -Only modify `environment.yml` when your tool is not on Bioconda and no recipe exists for it. nf-core guidelines strongly recommend Conda support, as Conda is the most accessible software management system. If your tool is not on Bioconda or [conda-forge](https://conda-forge.org/), add it to the appropriate repository. +Only modify `environment.yml` when your tool is not on Bioconda and no recipe exists for it. +nf-core guidelines strongly recommend Conda support, as Conda is the most accessible software management system. +If your tool is not on Bioconda or [conda-forge](https://conda-forge.org/), add it to the appropriate repository. +Bioconda provides [tutorials](https://bioconda.github.io/tutorials/2024-adding-bioinformatic-software-to-bioconda.html) on their website. :::info{title="Behind the scenes" collapse} nf-core uses a separate Conda file rather than the `main.nf` conda directive for two reasons: @@ -37,7 +41,8 @@ nf-core uses a separate Conda file rather than the `main.nf` conda directive for ## The `main.nf` file -The `main.nf` file contains the Nextflow module code. When generated, parts of the module are pre-filled and `TODO` comments mark where you need to make changes. +The `main.nf` file contains the Nextflow module code. +When generated, parts of the module are pre-filled and `TODO` comments mark where you need to make changes. An nf-core module has 9 main Nextflow process blocks: @@ -119,7 +124,8 @@ The `tag` block propagates standard metadata from each input channel's [meta map tag "$meta.id" ``` -You generally do not need to modify this. All nf-core pipelines assume an `id` element in each meta map, and pipeline developers can customise it. +You generally do not need to modify this. +All nf-core pipelines assume an `id` element in each meta map, and pipeline developers can customise it. ### The `label` block @@ -134,7 +140,7 @@ Key points: - nf-core defines a preset set of [standard labels with default resources](https://github.com/nf-core/tools/blob/52e810986e382972ffad0aab28e94f828ffd509b/nf_core/pipeline-template/conf/base.config#L22-L54). - Select the label that best matches your tool's typical resource needs. - Do not use custom labels. Pipeline developers can tune default resources in the pipeline's `conf/base.config`, and users can override per-module resources using `withName:` in a custom config. -- Standard labels keep modules and pipelines consistent in how resources are defined. +- Standard labels provide modules and pipelines a consistent set of default resources. ### The `conda` block @@ -144,7 +150,8 @@ The `conda` block tells Nextflow to use the `environment.yml` file when a pipeli conda "${moduleDir}/environment.yml" ``` -Do not modify this. The only exception is a tool that does not support Conda (typically proprietary tools), where you may remove the line. +Do not modify this. +You may only remove this line when a tool that does not support Conda (typically proprietary tools). ### The `container` block @@ -158,7 +165,8 @@ container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity The `nf-core modules create` command auto-detects the container name, version, and build from Bioconda and biocontainers. -Do not modify this unless the tool has no Bioconda recipe or biocontainer (typically proprietary tools). In that case, point the lines to the relevant container. +Do not modify this unless the tool has no Bioconda recipe or biocontainer (typically proprietary tools). +In that case, update the links to the relevant container. ### The `input` block @@ -169,7 +177,8 @@ input: tuple val(meta), path(bam) ``` -The boilerplate includes an example BAM input and a [meta map](https://nf-co.re/docs/developing/components/meta-map). Edit this section to declare every file your module needs. +The boilerplate includes an example BAM input and a [meta map](https://nf-co.re/docs/developing/components/meta-map). +Edit this section to declare every file your module needs. Guidelines: @@ -193,13 +202,17 @@ tuple val(meta), path(bam), path(bai) ``` :::info{title="Behind the scenes" collapse} -Nextflow processes must receive files through input channels to stage them correctly in the job's working directory. Declaring every possible input keeps the module reusable across pipelines. +Nextflow processes must receive files through input channels to stage them correctly in the job's working directory. +Declaring every possible input keeps the module reusable across pipelines, and minimises breakages for other developers. Nextflow has no native optional input syntax. Pipeline developers pass an empty list `[]` to indicate an input is not provided. -Mandatory non-file arguments belong in the input block so the module can run without extra configuration. Optional parameters are handled through `ext.args`, described below. +Mandatory non-file arguments belong in the input block so the module can run without extra configuration. +Optional parameters are handled through `ext.args`, described below. -The one-channel-one-file rule keeps channel structures consistent. Pipeline developers can use Nextflow's [`.multiMap()`](https://www.nextflow.io/docs/latest/reference/operator.html#multimap) operator to split multi-element channels into sub-channels while keeping elements associated. +The one-channel-one-file rule keeps channel structures consistent. +See the nf-core [module specifications](../../../specifications/components/overview) for exceptions. +Pipeline developers can use Nextflow's [`.multiMap()`](https://www.nextflow.io/docs/latest/reference/operator.html#multimap) operator to split multi-element channels into sub-channels while keeping elements associated. ::: :::tip{title="Examples" collapse} @@ -237,15 +250,16 @@ The `output:` block declares every file the tool can produce. ```nextflow output: -path "versions.yml" , emit: versions +tuple val("${task.process}"), val(''), eval('tool1 --version'), emit: versions_tool1, topic: versions ``` -The boilerplate always includes a mandatory `versions.yml` emission, because nf-core pipelines must report every tool version. +The boilerplate always includes a mandatory 'topics' version emission channel because nf-core pipelines must report every tool version. Guidelines: - Declare as many outputs as possible during first creation, whether default or optional, to maximise reuse and minimise future channel changes. -- If the module uses a meta map, emit all files with it except `versions.yml`. +- If the module uses a meta map, emit all files with it. +- Specify a command to get a 'clean' version number (e.g. just `1.2`, not `samtools v1.2 (2026-01-02)`) in the `eval()` section of the 'versions' topic channel. - Give each file type its own `emit` entry, usually one per format. - Combine two mutually exclusive formats that serve the same purpose (such as `.bai` and `.csi` indexes for `.bam`) in one channel. - Compress output files where the tool supports compressed input, and reflect this in the path pattern. @@ -257,7 +271,7 @@ output: tuple val(meta), path("${prefix}.bam"), emit: bam, optional: true tuple val(meta), path("${prefix}.cram"), emit: cram, optional: true tuple val(meta), path("${prefix}.sam"), emit: sam, optional: true -path "versions.yml", +tuple val("${task.process}"), val('samtools'), eval('samtools --version | head -n 1 | cut -d ' ' -f 2'), emit: versions_samtools, topic: versions ``` ```nextflow @@ -280,7 +294,7 @@ tuple val(meta), path('*.html') , emit: html tuple val(meta), path('*.log') , emit: log tuple val(meta), path('*.fail.fastq.gz') , optional:true, emit: reads_fail tuple val(meta), path('*.merged.fastq.gz'), optional:true, emit: reads_merged -path "versions.yml" , emit: versions +tuple val("${task.process}"), val('fastp'), eval('fastp --version 2>&1 | sed -e "s/fastp //g"'), emit: versions_fastp, topic: versions ``` A module producing three mutually exclusive primary outputs: @@ -290,7 +304,7 @@ output: tuple val(meta), path("*.{vcf,vcf.gz,bcf,bcf.gz}"), emit: vcf tuple val(meta), path("*.tbi") , emit: tbi, optional: true tuple val(meta), path("*.csi") , emit: csi, optional: true -path "versions.yml" , emit: versions +tuple val("${task.process}"), val('bcftools'), eval("bcftools --version | sed '1!d; s/^.*bcftools //'"), topic: versions, emit: versions_bcftools ``` A module producing multiple files that all support compressed variants: @@ -304,7 +318,7 @@ tuple val(meta), path('*.{daa,daa.gz}') , optional: true, emit: daa tuple val(meta), path('*.{sam,sam.gz}') , optional: true, emit: sam tuple val(meta), path('*.{tsv,tsv.gz}') , optional: true, emit: tsv tuple val(meta), path('*.{paf,paf.gz}') , optional: true, emit: paf -path "versions.yml" , emit: versions +tuple val("${task.process}"), val('diamond'), eval('diamond --version 2>&1 | tail -n 1 | sed "s/^diamond version //"'), emit: versions_diamond, topic: versions ``` ::: @@ -322,7 +336,8 @@ Do not modify or remove this block. ### The `script` block -The `script` block defines the command the module runs. You will edit this section most. +The `script` block defines the command the module runs. +You will edit this section most. ```nextflow script: @@ -331,27 +346,24 @@ def prefix = task.ext.prefix ?: "${meta.id}" """ samtools \\ sort \\ - $args \\ + ${args} \\ -@ $task.cpus \\ -o ${prefix}.bam \\ -T $prefix \\ $bam - -cat <<-END_VERSIONS > versions.yml -"${task.process}": - drep: \$(samtools --version |& sed '1!d ; s/samtools //') -END_VERSIONS """ ``` -The boilerplate includes an example samtools command and a `versions.yml` HEREDOC. Replace both with commands for your tool. +The boilerplate includes an example samtools command. +Replace with the command for your tool. Two standard variables appear at the top of the block. Do not remove them: -- `args` — how pipeline developers inject optional parameters into the command. The value comes from `ext.args` in the process scope of a Nextflow configuration file (defined in `modules.config` for nf-core pipelines). See the "Using in pipelines" chapter for details. +- `args` — how pipeline developers inject optional parameters into the command. The value comes from `ext.args` in the process scope of a Nextflow configuration file (defined in `modules.config` for nf-core pipelines). See the ["Using in pipelines"](./8-using) chapter for details. - `prefix` — the default output file basename. It defaults to the `id` value of the primary input channel's meta map. -Replace the example command with your own command, split across multiple lines with escaped backslashes for readability. Reference all required input variables. +Replace the example command with your own command, split across multiple lines with escaped backslashes for readability. +Reference all required input variables. Every command must use the `$args` variable, and the CPU count where supported. @@ -366,34 +378,20 @@ def single_end = meta.single_end ? "--single" : "" elprep merge \\ input/ \\ output/${prefix}.bam \\ - $args \\ + ${args} \\ ${single_end} \\ ... """ ``` -Keep the `versions.yml` HEREDOC, but replace the command to emit a clean version string (for example `1.2.3`, not `v1.2.3`). Use standard UNIX tools like `cut` or `sed` for the clean-up. If the tool does not report its version on the command line, use a dedicated variable after `args` and `prefix`: - -```nextflow -script: -def args = task.ext.args ?: '' -def prefix = task.ext.prefix ?: "${meta.id}" -def VERSION='2.1.3' // WARN: Version information not provided by tool on CLI. Please update this string when bumping -""" -... - -cat <<-END_VERSIONS > versions.yml -"${task.process}": - scimap: $VERSION -END_VERSIONS -``` If you must chain tools (for example piping or compression), define additional `args2`, `args3`, and so on, one per command in the pipe. Do not use custom `meta` elements in modules. :::info{title="Behind the scenes" collapse} -Custom meta fields are not standardised across tools and create extra work for pipeline developers. All optional parameters for dynamic command construction should come from `ext.args`. +Custom meta fields are not standardised across tools and create extra work for pipeline developers. +All optional parameters for dynamic command construction should come from `ext.args`. ::: :::warning @@ -410,15 +408,11 @@ The `stub` block simulates the module's output during a `-dry-run` of a pipeline def prefix = task.ext.prefix ?: "${meta.id}" """ touch ${prefix}.bam - - cat <<-END_VERSIONS > versions.yml - "${task.process}": - drep: \$(samtools --version |& sed '1!d ; s/samtools //') - END_VERSIONS """ ``` -The stub block should mirror the `script` block, except instead of running the tool, it uses `touch` to create empty files matching every output name. For gzipped outputs, pipe an empty echo into `gzip`: +The stub block should mirror the `script` block, except instead of running the tool, it uses `touch` to create empty files matching every output name. +For gzipped outputs, pipe an empty echo into `gzip`: ```nextflow stub: @@ -426,11 +420,6 @@ def args = task.ext.args ?: '' def prefix = task.ext.prefix ?: "${meta.id}" """ echo "" | gzip > ${prefix}.txt.gz - -cat <<-END_VERSIONS > versions.yml -"${task.process}": - tool: \$(samtools --version |& sed '1!d ; s/samtools //') -END_VERSIONS """ ``` @@ -445,19 +434,14 @@ def prefix = task.ext.prefix ?: "${meta.id}" """ echo "$args" echo "" | gzip > ${prefix}.txt.gz - -cat <<-END_VERSIONS > versions.yml -"${task.process}": -tool: \$(samtools --version |& sed '1!d ; s/samtools //') -END_VERSIONS """ ``` -Keep the `versions.yml` HEREDOC command. It runs during the dry run. ## The `meta.yml` file -The `meta.yml` file documents the module with descriptions, keywords, and links to the tool's resources. This information powers the searchable [modules page](https://nf-co.re/modules), improves discoverability on external databases, and supports future automated linkage between modules. +The `meta.yml` file documents the module with descriptions, keywords, and links to the tool's resources. +This information powers the searchable [modules page](https://nf-co.re/modules), improves discoverability on external databases, and supports future automated linkage between modules. The file has four main sections: @@ -520,11 +504,27 @@ output: - edam: "http://edamontology.org/format_2573" - edam: "http://edamontology.org/format_3462" - - versions: - - "versions.yml": - type: file - description: File containing software versions - pattern: "versions.yml" + versions_drep: + - - ${task.process}: + type: string + description: The name of the process + - drep: + type: string + description: The name of the tool + - drep version 2>&1 | sed 's/^.*drep v//': + type: eval + description: The expression to obtain the version of the tool +topics: + versions: + - - ${task.process}: + type: string + description: The name of the process + - drep: + type: string + description: The name of the tool + - drep version 2>&1 | sed 's/^.*drep v//': + type: eval + description: The expression to obtain the version of the tool authors: - "@jfy133" @@ -726,11 +726,46 @@ output: type: file description: Index file for BAM file pattern: "*.{csi}" - - versions: - - versions.yml: - type: file - description: File containing software versions - pattern: "versions.yml" + versions_bwamem2: + - - ${task.process}: + type: string + description: The name of the process + - bwamem2: + type: string + description: The name of the tool + - bwa-mem2 version | grep -o -E "[0-9]+(\.[0-9]+)+": + type: eval + description: The expression to obtain the version of the tool + versions_samtools: + - - ${task.process}: + type: string + description: The name of the process + - samtools: + type: string + description: The name of the tool + - samtools version | sed '1!d;s/.* //': + type: eval + description: The expression to obtain the version of the tool +topics: + versions: + - - ${task.process}: + type: string + description: The name of the process + - bwamem2: + type: string + description: The name of the tool + - bwa-mem2 version | grep -o -E "[0-9]+(\.[0-9]+)+": + type: eval + description: The expression to obtain the version of the tool + - - ${task.process}: + type: string + description: The name of the process + - samtools: + type: string + description: The name of the tool + - samtools version | sed '1!d;s/.* //': + type: eval + description: The expression to obtain the version of the tool authors: - "@maxulysse" - "@matthdsm" @@ -787,4 +822,4 @@ If you later update a module originally written by someone else, add yourself to This chapter cannot cover every specification detail or edge case. Always refer to the [nf-core module specifications](https://nf-co.re/docs/specifications/components/modules/general) when writing a module. -The next chapter covers unit testing the module with nf-test. +The [next chapter](./6-testing) covers unit testing the module with nf-test. diff --git a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/6-testing.md b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/6-testing.md index 54e1a1b0da..04804c8594 100644 --- a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/6-testing.md +++ b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/6-testing.md @@ -7,7 +7,9 @@ shortTitle: "Chapter 6: Testing modules" This chapter describes how to unit test an nf-core module with the [nf-test](https://www.nf-test.com/) framework. :::warning -Setting up and debugging tests is a topic of its own. This chapter covers the basics to get you started. Tests are required for all nf-core modules, so do not be discouraged if you need multiple attempts to get them right. +Setting up and debugging tests is a topic of its own. +This chapter covers the basics to get you started. +Tests are required for all nf-core modules, so do not be discouraged if you need multiple attempts to get them right. ::: ## The `main.nf.test` file @@ -116,7 +118,8 @@ You rarely need to edit this section. You may need to extend it to: ### The `test` block -The test block declares inputs and assertions. You will spend most of your time here. +The test block declares inputs and assertions. +You will spend most of your time here. ```nextflow test("sarscov2 - bam") { @@ -146,11 +149,14 @@ A test block has three parts: - The `when` block (inputs). - The `then` block (assertions). -The boilerplate provides one test with a single input file plus a stub-run test. All nf-core modules need a stub-run test. You only need to update its title and inputs to match the first test. +The boilerplate provides one test with a single input file plus a stub-run test. +All nf-core modules need a stub-run test. +You only need to update its title and inputs to match the first test. #### `when` block -Update the test title to make it distinct. At a minimum include the organism of the test data and the file format of the primary input, for example: +Update the test title to make it distinct. +At a minimum include the organism of the test data and the file format of the primary input, for example: ```nextflow test("sarscov2 - fastq - pairedend") @@ -162,7 +168,9 @@ or test("sarscov2 - fastq - bakta annotation input") ``` -In the `when` block, declare each input channel using `input[index]` notation (`input[0]` for the first, `input[1]` for the second, and so on). The block accepts standard Nextflow code — channel factories, operators, and so on — as you would use in a pipeline. Make sure each `input[]` matches the channel structure of your module, including any meta map: +In the `when` block, declare each input channel using `input[index]` notation (`input[0]` for the first, `input[1]` for the second, and so on). +The block accepts standard Nextflow code — channel factories, operators, and so on — as you would use in a pipeline. +Make sure each `input[]` matches the channel structure of your module, including any meta map: ```nextflow when { @@ -198,7 +206,7 @@ then { { assert process.success }, { assert snapshot( process.out.metrics, - process.out.versions, + process.out.findAll { key, val -> key.startsWith('versions') } file(process.out.qc_report[0][1]).name, ).match() } @@ -206,13 +214,15 @@ then { } ``` -Here, `metrics` and `versions` have stable md5 sums and use the default snapshot check. The `qc_report` varies, so we only assert that the filename is unchanged. +Here, `metrics` have stable md5 sums and use the default snapshot check. +The `qc_report` varies, so we only assert that the filename is unchanged. +We use a special assertion to print the versions so they are readable in the test output file. For a full list of nf-test assertions, see the [nf-core documentation on assertions](https://nf-co.re/docs/developing/testing/assertions). Test outputs using these methods, in order of preference: -1. md5 sums. +1. `md5sums`. 2. String contents within a file. 3. File name or existence. From d2fc2a70a30d30718e90e5e08b23a2c37b83127c Mon Sep 17 00:00:00 2001 From: "James A. Fellows Yates" Date: Wed, 22 Apr 2026 09:59:03 +0200 Subject: [PATCH 06/10] Apply suggestions from code review Co-authored-by: James A. Fellows Yates --- .../3-what-is-a-module.md | 2 + .../writing-nf-core-modules/6-testing.md | 50 ++++++++++++------- .../writing-nf-core-modules/7-development.md | 27 ++++++---- 3 files changed, 51 insertions(+), 28 deletions(-) diff --git a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/3-what-is-a-module.md b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/3-what-is-a-module.md index e62fc7b153..651201802f 100644 --- a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/3-what-is-a-module.md +++ b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/3-what-is-a-module.md @@ -66,3 +66,5 @@ _Schematic diagram showing the relationship between the three main categories of Writing an nf-core module means creating these files and populating them with content that follows the nf-core specifications. The [next chapter](./4-boilerplate) goes into more detail about the contents of each file. + +The [next chapter](./4-boilerplate) goes into more detail about the contents of each file. diff --git a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/6-testing.md b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/6-testing.md index 04804c8594..ece3dafb00 100644 --- a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/6-testing.md +++ b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/6-testing.md @@ -226,13 +226,16 @@ Test outputs using these methods, in order of preference: 2. String contents within a file. 3. File name or existence. -If you vary assertion types per channel, always include `process.out.versions`. +If you vary assertion types per channel away from just `snapshot(process.out)`, always include `process.out.findAll { key, val -> key.startsWith('versions') }`. -Once you have completed one test, copy its structure for each additional test case. Write as many tests as needed to cover every configuration, and at a minimum test every mandatory and optional input and output channel once. +Once you have completed one test, copy its structure for each additional test case. +Write as many tests as needed to cover every configuration, and at a minimum test every mandatory and optional input and output channel once. ### The `setup` block (optional) -The `setup` block runs one or more modules before your new module. The outputs of these upstream modules can then feed into your test. The setup block is not included by default. +The `setup` block runs one or more modules before your new module. +The outputs of these upstream modules can then feed into your test. +The setup block is not included by default. ```nextflow setup { @@ -247,16 +250,18 @@ The `setup` block runs one or more modules before your new module. The outputs o } ``` -Placement options: +Placement options for the `setup` block: - Before all tests, to reuse the upstream output across every test. - Inside a test block, to run the setup only for that test. Outputs produced in a setup block are not asserted against. -Fill in the setup block like a test block, but include the script path of the upstream module. Use the same `input[0]`, `input[1]` syntax. +Fill in the setup block like a test block, but include the script path of the upstream module. +Use the same `input[0]`, `input[1]` syntax. -We discourage setup blocks because they increase runtime. They are useful when the module needs inputs too large for the nf-core/test-datasets repository, or when the upstream module runs quickly. +We generally discourage setup blocks because they increase runtime. +They are useful when the module needs inputs too large for the nf-core/test-datasets repository, or when the upstream module runs faster than downloading a file. :::tip{title="Examples" collapse} A global setup block, reused in every test: @@ -366,7 +371,7 @@ nextflow_process { { assert process.success }, { assert snapshot( process.out.vcf, - process.out.versions + process.out.findAll { key, val -> key.startsWith('versions') } ).match() } ) } @@ -378,7 +383,8 @@ nextflow_process { ## The `nextflow.config` file (optional) -The optional `nextflow.config` file is not generated by the boilerplate. It sits alongside `main.nf.test` and `main.nf.test.snap` in the `tests/` directory. +The optional `nextflow.config` file is not generated by the boilerplate. +It sits alongside `main.nf.test` and `main.nf.test.snap` in the `tests/` directory. Use it to provide extra settings, for example `ext.args` for test configurations, optional parameters that trigger extra outputs, or per-import settings when a module is imported more than once in setup blocks. @@ -445,7 +451,8 @@ Or loaded inside a single test: ## The `main.nf.test.snap` file -The `main.nf.test.snap` file is not part of the boilerplate. nf-test generates and updates it automatically the first time you run the test. +The `main.nf.test.snap` file is not part of the boilerplate. +nf-test generates and updates it automatically the first time you run the test. ```tree {6} ├── environment.yml @@ -463,19 +470,22 @@ Generate it with: nf-core modules test / ``` -Set the prompts, and the tests run twice. nf-test reports whether the outputs match and creates the snapshot on the first run. +Set the prompts, and the tests run twice. +nf-test reports whether the outputs match and creates the snapshot on the first run. -The snapshot records what every later test run is compared against, such as md5 sums or string matches. +The snapshot records what every later test run is compared against, such as `md5sums` or string matches. -You never need to edit this file manually. If you need to regenerate it: +You never need to edit this file manually. +If you need to regenerate it: ```bash nf-core modules test / --update ``` -Add `--verbose` to see the full nf-test output. +Add `--verbose` to see the full nf-test output including the Nextflow console log. -Inspect the snapshot after generation. Even if tests pass, look for unexpected values such as empty file md5 sums or missing assertions. +Inspect the snapshot after generation. +Even if tests pass, look for unexpected values such as empty file `md5sums` (for raw text files, this is `d41d8cd98f00b204e9800998ecf8427e`) or missing assertions. To inspect a module's working directory, note the hash shown next to each test name: @@ -484,11 +494,15 @@ To inspect a module's working directory, note the hash shown next to each test n │ PASSED (7.762s) ``` -Change into `.nf-test/tests//work` (use TAB to autocomplete) to find the Nextflow working directories for that test. Each subdirectory contains the outputs of one process run, so you can verify the contents of every output file. +Change into `.nf-test/tests//work` (use TAB to autocomplete) to find the Nextflow working directories for that test. +In the example above, the hash is `528b411a`. +Each subdirectory contains the outputs of one process run, so you can verify the contents of every output file. -Empty entries in the snapshot may be legitimate when an optional channel does not emit. Always verify this is expected for the test. +Empty entries in the snapshot may be legitimate when an optional channel does not emit output. +Always verify this is expected for the test. -Assertions outside the `snapshot()` function are not recorded in the snapshot. These are usually boolean checks such as file existence, and nf-test prints failures to the console instead. +Assertions outside the `snapshot()` function are not recorded in the snapshot. +These are usually boolean checks such as file existence, and nf-test prints failures to the console instead. :::tip{title="Examples" collapse} @@ -611,4 +625,4 @@ Assertions outside the `snapshot()` function are not recorded in the snapshot. T Once your snapshot is stable, your module is ready for use. -The next chapter summarises the development workflow and the steps for contributing to the [nf-core/modules](https://github.com/nf-core/modules) repository. +The [next chapter](./7-development) summarises the development workflow and the steps for contributing to the [nf-core/modules](https://github.com/nf-core/modules) repository. diff --git a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/7-development.md b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/7-development.md index 6b7b431435..95506500f7 100644 --- a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/7-development.md +++ b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/7-development.md @@ -6,11 +6,12 @@ shortTitle: "Chapter 7: Development workflow" This chapter outlines the workflow for contributing to the [nf-core/modules repository](https://github.com/nf-core/modules/). -This training used a fork of that repository, but you can apply everything you have learnt to your own private module repository or to local modules inside a pipeline. The nf-core community encourages you to contribute modules upstream so others can benefit. +This training used a fork of that repository, but you can apply everything you have learnt to your own private module repository or to local modules inside a pipeline. +The nf-core community encourages you to contribute modules upstream so others can benefit. ## Check -Before writing a module, check that someone has not already built it or started on it: +Before writing a module, check that someone has not already built or started on it: 1. Search the [modules page](https://nf-co.re/modules) on the nf-core website. 2. Search [open pull requests](https://github.com/nf-core/modules/pulls) on the nf-core/modules repository. @@ -47,22 +48,26 @@ Check the snapshot file looks correct before moving on. ## Lint -Standardisation is central to nf-core. The linting tool flags issues before submission: +Standardisation is central to nf-core. +The linting tool flags issues before submission: ```bash nf-core modules lint / ``` -The console output lists any issues to fix. Some issues can be auto-fixed with `--fix`, in which case the linter prints the command to run. +The console output lists any issues to fix. +Some issues can be auto-fixed with `--fix`, in which case the linter prints the command to run. -Check each fix carefully. The linter may add empty sections that you still need to populate. +Check each fix carefully. +The linter may add empty sections that you still need to populate. :::note Linting is optional for private custom modules, but still recommended to catch small issues. ::: :::warning -The linter cannot check every aspect of the [nf-core module specification](https://nf-co.re/docs/specifications/components/modules/general). Review the specification yourself before submission. +The linter cannot check every aspect of the [nf-core module specification](https://nf-co.re/docs/specifications/components/modules/general). +Review the specification yourself before submission. ::: ## Submit @@ -76,14 +81,16 @@ git push Open a [pull request](https://github.com/nf-core/modules/pulls) from your fork and branch to the nf-core/modules repository. -To attract a reviewer, ask in the nf-core Slack [#request-review](https://nfcore.slack.com/archives/CQY2U5QU9) channel. See the [join instructions](https://nf-co.re/join) if you are not already on Slack. +To attract a reviewer, ask in the nf-core Slack [#request-review](https://nfcore.slack.com/archives/CQY2U5QU9) channel. +See the [join instructions](https://nf-co.re/join) if you are not already on Slack. -If a reviewer asks for changes, do not close the PR. Push updates to the same branch, then request another review. +If a reviewer asks for changes, do not close the PR. +Push updates to the same branch, then request another review. Once the PR is approved, merge it and the module is available to everyone. If you get stuck, ask on the [#modules](https://nfcore.slack.com/archives/CJRH30T6V) channel or the [#nostupidquestions](https://nfcore.slack.com/archives/C043FMKUNLB) channel. -Thank you for contributing to the community. +Thank you for contributing to the community! -The final chapter covers how to use modules in your own pipelines. +The [final chapter](./8-using) covers how to use modules in your own pipelines. From 441cf173a104513a20cf6f2c2155609c53cbee2c Mon Sep 17 00:00:00 2001 From: "James A. Fellows Yates" Date: Wed, 22 Apr 2026 09:59:48 +0200 Subject: [PATCH 07/10] Apply suggestions from code review Co-authored-by: James A. Fellows Yates --- .../writing-nf-core-modules/8-using.md | 21 ++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/8-using.md b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/8-using.md index 2ff02f0038..0abac33fa1 100644 --- a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/8-using.md +++ b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/8-using.md @@ -12,13 +12,15 @@ It assumes you have already contributed your module to the nf-core/modules repos ### Pipelines using the nf-core template -If you generated your pipeline with [nf-core tools](https://nf-co.re/docs/nf-core-tools/pipelines/create) ([whether the pipeline is official or not](https://nf-co.re/docs/developing/pipelines/external-use)), you can install a module directly. Run the following from the root of the repository, or pass `--dir`: +If you generated your pipeline with [nf-core tools](https://nf-co.re/docs/nf-core-tools/pipelines/create) ([whether the pipeline is official or not](https://nf-co.re/docs/developing/pipelines/external-use)), you can install a module directly. +Run the following from the root of the repository, or pass `--dir`: ```bash nf-core modules install / ``` -This downloads the module files into the existing `modules/nf-core` directory. The console output prints an `include` line you can paste into your pipeline script. +This downloads the module files into the existing `modules/nf-core` directory. +The console output prints an `include` line you can paste into your pipeline script. ### Custom pipelines @@ -28,7 +30,8 @@ You can use the same command in a custom pipeline. On the first run you will be 2. Accept creation of an `nf-core.yaml` file. This supports adding more modules later. 3. Accept creation of a `modules.json` file. This tracks installed module versions for future updates. -The command creates `modules/nf-core//`, a `modules.json`, and an `.nf-core.yml` configuration file. The console output prints an `include` line for your pipeline script. +The command creates `modules/nf-core//`, a `modules.json`, and an `.nf-core.yml` configuration file. +The console output prints an `include` line for your pipeline script. :::warning If installing a second module returns `ERROR 'manifest.name'`, add a `nextflow.config` with a [`manifest` scope](https://www.nextflow.io/docs/latest/reference/config.html#manifest) that includes `name`, `description`, and `version`. @@ -58,7 +61,8 @@ Most pipelines need a few additional adjustments, covered below. Reconfigure your input channels to match nf-core conventions. -The most important convention is the use of [meta maps](https://nf-co.re/docs/developing/components/meta-map). If your pipeline does not use meta maps, add one before passing data to the module: +The most important convention is the use of [meta maps](https://nf-co.re/docs/developing/components/meta-map). +If your pipeline does not use meta maps, add one before passing data to the module: ```nextflow def val_interleave = false @@ -68,7 +72,8 @@ ch_input_for_samtoolsfasta = ch_input SAMTOOLS_FASTA (ch_input_for_samtoolsfasta, val_interleave) ``` -Here, the original pipeline channel contained only FASTA files. The module needs a meta map, so `.map {}` creates one with a single `id` attribute assigned from the file's [`simpleName`](https://www.nextflow.io/docs/latest/reference/stdlib.html#stdlib-types-path). +Here, the original pipeline channel contained only FASTA files. +The module needs a meta map, so `.map {}` creates one with a single `id` attribute assigned from the file's [`simpleName`](https://www.nextflow.io/docs/latest/reference/stdlib.html#stdlib-types-path). ### Tags and labels @@ -76,7 +81,8 @@ Here, the original pipeline channel contained only FASTA files. The module needs Most nf-core modules set a `tag` directive to produce a human-readable process description in the run log. -By default, all nf-core modules use the `id` attribute from the meta map as the tag value. Set `id` on every meta map passed to an nf-core module. +By default, all nf-core modules use the `id` attribute from the meta map as the tag value. +Set `id` on every meta map passed to an nf-core module that uses the meta map. #### Labels @@ -121,7 +127,8 @@ process{ } ``` -This hardcodes FastQC's `--quiet` flag for every invocation and sets the default output directory under `--outdir`. It also excludes the required `versions.yml` file from publication, because nf-core pipelines render versions through [MultiQC](https://multiqc.info/). +This hardcodes FastQC's `--quiet` flag for every invocation of the process and sets the default output directory under `--outdir`. +It also excludes the required `versions.yml` file from publication, because nf-core pipelines render versions through [MultiQC](https://multiqc.info/). Pass arguments dynamically through a closure to expose pipeline parameters to users: From 021b2727e06f6dcd83cbc8aff61ac84f406f28bb Mon Sep 17 00:00:00 2001 From: nf-core-bot Date: Wed, 22 Apr 2026 08:02:39 +0000 Subject: [PATCH 08/10] [automated] Fix code linting --- .../writing-nf-core-modules/4-boilerplate.md | 178 +++++++++--------- .../5-writing-modules.md | 2 - 2 files changed, 89 insertions(+), 91 deletions(-) diff --git a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/4-boilerplate.md b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/4-boilerplate.md index 04ca33d095..c54545f673 100644 --- a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/4-boilerplate.md +++ b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/4-boilerplate.md @@ -1,89 +1,89 @@ ---- -title: "Chapter 4: Generating boilerplate files" -subtitle: "How to generate the module file skeleton" -shortTitle: "Chapter 4: Boilerplate files" ---- - -This chapter explains how to generate an nf-core module template and what each file contains. - -## Preparation - -Fork and clone the [nf-core/modules GitHub repository](https://github.com/nf-core/modules) to your working environment, then create a new branch for your module: - -```bash -git switch master ## ensure you have the latest state of the repository -git switch -c -``` - -## Generate the boilerplate files - -From the root of the repository, run: - -```bash -nf-core modules create / -``` - -:::note -If your module does not have subcommands, use `` -::: - -### Naming conventions - -- All parts of the module name must be lowercase, alphanumeric, with no punctuation or special characters. -- Single-command tools use the tool name only. For example, for a tool executed with `fastp -i -o `, run `nf-core modules create fastp`. -- Tools with subcommands use `/`, even if you only plan to wrap one subcommand. For example, `samtools view`, run `nf-core modules create samtools/view`. -- For a third level of subcommand, append it to the subcommand name. For `samtools view flagstats`, run `nf-core modules create samtools/viewflagstat`. - -For example, to create a module for the tool `drep` with the subcommand `compare`: - -```bash -nf-core modules create drep/compare -``` - -### Prompts from `nf-core modules create` - -The command tries to pre-fill the boilerplate. -It searches [Bioconda](https://bioconda.github.io/) and [biocontainers](https://biocontainers.pro/) for the latest version of your tool and adds the container definitions automatically. - -You will then be prompted for: - -- **Your GitHub username.** -- **A process resource label.** These standardised tags map to default memory, CPU, and wall time. Choose the label that best matches your tool's typical requirements. The defaults for each label are defined in the pipeline template [base.config](https://github.com/nf-core/tools/blob/52e810986e382972ffad0aab28e94f828ffd509b/nf_core/pipeline-template/conf/base.config#L29-L54). Pipelines can override these values. -- **Whether the module should use a [meta map](https://nf-co.re/docs/developing/components/meta-map).** Answer yes in most cases. Meta maps carry sample metadata alongside files, which pipelines use to drive downstream processing decisions. - -:::tip -You can pass any of these values as command-line flags to skip the prompts. -::: - -## Output - -After the command completes, you will see the following files and directories: - -```tree {8-13} -modules/nf-core/drep/compare/ -├── environment.yml -├── main.nf -├── meta.yml -└── tests - └── main.nf.test -``` - -If you later create a second subcommand (`dereplicate`), the directory structure becomes: - -```tree {8-13} -modules/nf-core/drep/ -├── compare -│ ├── environment.yml -│ ├── main.nf -│ ├── meta.yml -│ └── tests -│ └── main.nf.test -└── dereplicate - ├── environment.yml - ├── main.nf - ├── meta.yml - └── tests - └── main.nf.test -``` - -The [next chapter](./5-writing-modules) walks through each generated file and explains what to change. +--- +title: "Chapter 4: Generating boilerplate files" +subtitle: "How to generate the module file skeleton" +shortTitle: "Chapter 4: Boilerplate files" +--- + +This chapter explains how to generate an nf-core module template and what each file contains. + +## Preparation + +Fork and clone the [nf-core/modules GitHub repository](https://github.com/nf-core/modules) to your working environment, then create a new branch for your module: + +```bash +git switch master ## ensure you have the latest state of the repository +git switch -c +``` + +## Generate the boilerplate files + +From the root of the repository, run: + +```bash +nf-core modules create / +``` + +:::note +If your module does not have subcommands, use `` +::: + +### Naming conventions + +- All parts of the module name must be lowercase, alphanumeric, with no punctuation or special characters. +- Single-command tools use the tool name only. For example, for a tool executed with `fastp -i -o `, run `nf-core modules create fastp`. +- Tools with subcommands use `/`, even if you only plan to wrap one subcommand. For example, `samtools view`, run `nf-core modules create samtools/view`. +- For a third level of subcommand, append it to the subcommand name. For `samtools view flagstats`, run `nf-core modules create samtools/viewflagstat`. + +For example, to create a module for the tool `drep` with the subcommand `compare`: + +```bash +nf-core modules create drep/compare +``` + +### Prompts from `nf-core modules create` + +The command tries to pre-fill the boilerplate. +It searches [Bioconda](https://bioconda.github.io/) and [biocontainers](https://biocontainers.pro/) for the latest version of your tool and adds the container definitions automatically. + +You will then be prompted for: + +- **Your GitHub username.** +- **A process resource label.** These standardised tags map to default memory, CPU, and wall time. Choose the label that best matches your tool's typical requirements. The defaults for each label are defined in the pipeline template [base.config](https://github.com/nf-core/tools/blob/52e810986e382972ffad0aab28e94f828ffd509b/nf_core/pipeline-template/conf/base.config#L29-L54). Pipelines can override these values. +- **Whether the module should use a [meta map](https://nf-co.re/docs/developing/components/meta-map).** Answer yes in most cases. Meta maps carry sample metadata alongside files, which pipelines use to drive downstream processing decisions. + +:::tip +You can pass any of these values as command-line flags to skip the prompts. +::: + +## Output + +After the command completes, you will see the following files and directories: + +```tree {8-13} +modules/nf-core/drep/compare/ +├── environment.yml +├── main.nf +├── meta.yml +└── tests + └── main.nf.test +``` + +If you later create a second subcommand (`dereplicate`), the directory structure becomes: + +```tree {8-13} +modules/nf-core/drep/ +├── compare +│ ├── environment.yml +│ ├── main.nf +│ ├── meta.yml +│ └── tests +│ └── main.nf.test +└── dereplicate + ├── environment.yml + ├── main.nf + ├── meta.yml + └── tests + └── main.nf.test +``` + +The [next chapter](./5-writing-modules) walks through each generated file and explains what to change. diff --git a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/5-writing-modules.md b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/5-writing-modules.md index 4448fc1536..dc3075ccf9 100644 --- a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/5-writing-modules.md +++ b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/5-writing-modules.md @@ -384,7 +384,6 @@ elprep merge \\ """ ``` - If you must chain tools (for example piping or compression), define additional `args2`, `args3`, and so on, one per command in the pipe. Do not use custom `meta` elements in modules. @@ -437,7 +436,6 @@ echo "" | gzip > ${prefix}.txt.gz """ ``` - ## The `meta.yml` file The `meta.yml` file documents the module with descriptions, keywords, and links to the tool's resources. From ea4601785f1a247fdd8db76a0c3c8ab0cff65167 Mon Sep 17 00:00:00 2001 From: "James A. Fellows Yates" Date: Wed, 22 Apr 2026 09:21:56 +0000 Subject: [PATCH 09/10] Try fixing display of image --- .../nf-core-module-file-relationship.png | Bin .../writing-nf-core-modules/3-what-is-a-module.md | 2 +- 2 files changed, 1 insertion(+), 1 deletion(-) rename {sites/docs/src/assets/images/tutorials/training => public/images/developing/tutorials/writing-nfcore-modules}/nf-core-module-file-relationship.png (100%) diff --git a/sites/docs/src/assets/images/tutorials/training/nf-core-module-file-relationship.png b/public/images/developing/tutorials/writing-nfcore-modules/nf-core-module-file-relationship.png similarity index 100% rename from sites/docs/src/assets/images/tutorials/training/nf-core-module-file-relationship.png rename to public/images/developing/tutorials/writing-nfcore-modules/nf-core-module-file-relationship.png diff --git a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/3-what-is-a-module.md b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/3-what-is-a-module.md index 651201802f..c998f00b5c 100644 --- a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/3-what-is-a-module.md +++ b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/3-what-is-a-module.md @@ -52,7 +52,7 @@ These files fall into three categories: The following diagram shows how these files relate to each other: -Diagram showing how the execution, documentation, and testing files in an nf-core module relate to each other. +Diagram showing how the execution, documentation, and testing files in an nf-core module relate to each other. _Schematic diagram showing the relationship between the three main categories of files in an nf-core module._ From fdc5ad58b1adc3e1b0ff24c2deabe351b5712077 Mon Sep 17 00:00:00 2001 From: "James A. Fellows Yates" Date: Wed, 22 Apr 2026 11:23:59 +0200 Subject: [PATCH 10/10] Update sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/3-what-is-a-module.md --- .../tutorials/writing-nf-core-modules/3-what-is-a-module.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/3-what-is-a-module.md b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/3-what-is-a-module.md index c998f00b5c..fa274072be 100644 --- a/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/3-what-is-a-module.md +++ b/sites/docs/src/content/docs/developing/tutorials/writing-nf-core-modules/3-what-is-a-module.md @@ -66,5 +66,3 @@ _Schematic diagram showing the relationship between the three main categories of Writing an nf-core module means creating these files and populating them with content that follows the nf-core specifications. The [next chapter](./4-boilerplate) goes into more detail about the contents of each file. - -The [next chapter](./4-boilerplate) goes into more detail about the contents of each file.