From 9a823e2c2095038adecdc2fb63975205cbfacfe2 Mon Sep 17 00:00:00 2001 From: Christopher Hakkaart Date: Tue, 14 Apr 2026 14:28:28 +1200 Subject: [PATCH 01/14] Revise Modules pages and create new section Signed-off-by: Christopher Hakkaart --- docs/index.md | 11 ++ docs/module.md | 162 +-------------------------- docs/modules/developing-modules.md | 172 +++++++++++++++++++++++++++++ docs/modules/module-registry.md | 80 ++++++++++++++ docs/modules/modules.md | 56 ++++++++++ docs/modules/using-modules.md | 140 +++++++++++++++++++++++ docs/reference/cli.md | 2 + 7 files changed, 466 insertions(+), 157 deletions(-) create mode 100644 docs/modules/developing-modules.md create mode 100644 docs/modules/module-registry.md create mode 100644 docs/modules/modules.md create mode 100644 docs/modules/using-modules.md diff --git a/docs/index.md b/docs/index.md index 877f0d8837..04ab8a0f73 100644 --- a/docs/index.md +++ b/docs/index.md @@ -87,6 +87,17 @@ sharing vscode ``` +```{toctree} +:hidden: +:caption: Modules +:maxdepth: 1 + +modules/modules +modules/using-modules +modules/developing-modules +modules/module-registry +``` + ```{toctree} :hidden: :caption: Software dependencies diff --git a/docs/module.md b/docs/module.md index 9a9667564e..d2b010e7bf 100644 --- a/docs/module.md +++ b/docs/module.md @@ -283,164 +283,12 @@ When {ref}`wave-page` is enabled, the contents of the module `resources/` direct ## Sharing modules -Modules are designed to be easy to share and re-use across different pipelines, which helps eliminate duplicate work and spread improvements throughout the community. There are several ways to share modules: +Nextflow makes modules easy to share and re-use across different pipelines, helping eliminate duplicate work and spread improvements throughout the community. +There are several ways to share modules: -- Use the Nextflow module registry (recommended, see below) -- Simply copy the module files into your pipeline repository +- Use the {ref}`Nextflow module registry ` (recommended) +- Copy the module files into your pipeline repository - Use [Git submodules](https://git-scm.com/book/en/v2/Git-Tools-Submodules) to fetch modules from other Git repositories without maintaining a separate copy - Use the [nf-core](https://nf-co.re/tools#modules) CLI to install and update modules with a standard approach used by the nf-core community -(module-registry)= - -## Registry-based modules - -:::{versionadded} 26.04.0 -::: - -Nextflow provides a module registry that enables you to install, publish, and manage modules from centralized registries. This system provides version management, integrity checking, and seamless integration with the Nextflow language. - -### Installing modules from a registry - -Use the `module install` command to download modules from a registry: - -```console -$ nextflow module install nf-core/fastqc -$ nextflow module install nf-core/fastqc -version 1.0.0 -``` - -Installed modules are stored in the `modules/` directory and can be included by name instead of a relative path: - -```nextflow -include { FASTQC } from 'nf-core/fastqc' - -workflow { - reads = Channel.fromFilePairs('data/*_{1,2}.fastq.gz') - FASTQC(reads) -} -``` - -### Running modules directly - -For ad-hoc tasks or testing, you can run a module directly without creating a workflow: - -```console -$ nextflow module run nf-core/fastqc --input 'data/*.fastq.gz' -``` - -This command accepts all standard `nextflow run` options (`-profile`, `-resume`, etc.) and automatically downloads the module if not already installed. - -### Discovering modules - -Search for available modules using the `module search` command: - -```console -$ nextflow module search alignment -$ nextflow module search "quality control" -limit 10 -``` - -List installed modules in your project: - -```console -$ nextflow module list -``` - -### Module checksum verification - -Nextflow automatically verifies module integrity using checksums. If you modify a module locally, Nextflow will detect the change and prevent accidental overwrites: - -```console -$ nextflow module install nf-core/fastqc -version 1.1.0 -Warning: Module nf-core/fastqc has local modifications. Use -force to override. -``` - -Use the `-force` flag to override local modifications when needed. - -### Removing modules - -Use the `module remove` command to uninstall a module: - -```console -$ nextflow module remove nf-core/fastqc -``` - -By default, both the module files and the `.module-info` file are removed. Use the flags below to control this behaviour: - -- `-keep-files`: Remove the `.module-info` file created at install but keep the rest of files -- `-force`: Force removal even if the module has no `.module-info` file (i.e. not installed from a registry) or has local modifications - -### Viewing module information - -Use the `module info` command to display metadata and a usage template for a module: - -```console -$ nextflow module info nf-core/fastqc -$ nextflow module info nf-core/fastqc -version 1.0.0 -``` - -The output includes the module description, authors, keywords, tools, inputs, outputs, and a ready-to-use command-line template. Use `-o json` to get machine-readable output. - -### Publishing modules - -To share your own modules, use the `module publish` command: - -```console -$ nextflow module publish myorg/my-module -``` - -The argument can be either a `scope/name` reference (for an already-installed module) or a local directory path containing the module files. - -Your module directory must include: - -- `main.nf`: The module entry point -- `meta.yml`: Module spec (name, description, version, etc.) -- `README.md`: Module documentation - -Authentication is required for publishing and can be provided via the `NXF_REGISTRY_TOKEN` environment variable or in your configuration: - -```groovy -registry { - apiKey = 'YOUR_REGISTRY_TOKEN' -} -``` - -Use `-dry-run` to validate your module structure without uploading: - -```console -$ nextflow module publish myorg/my-module -dry-run -``` - -### Registry configuration - -By default, Nextflow uses the public registry at `https://registry.nextflow.io`. You can configure alternative or additional registries: - -```groovy -registry { - url = [ - 'https://private.registry.myorg.com', - 'https://registry.nextflow.io' - ] - apiKey = '${MYORG_TOKEN}' -} -``` - -Registries are queried in the order specified until a module is found. The `apiKey` is used only for the primary (first) registry. - -### Module directory structure - -Registry modules follow a standard directory structure: - -``` -modules/ -└── scope/ - └── module-name/ - ├── .module-info # Integrity checksum (generated automatically) - ├── README.md # Documentation (required for publishing) - ├── main.nf # Module script (required) - ├── meta.yml # Module spec (required for publishing) - ├── resources/ # Optional: module binaries and resources - └── templates/ # Optional: process templates -``` - -The `modules/` directory should be committed to your Git repository to ensure reproducibility. - -See the {ref}`cli-page` documentation for more information about module commands. +See {ref}`modules-page` for an overview of the module registry, {ref}`using-modules-page` to learn how to install and manage registry modules, and {ref}`dev-modules-page` to learn how to create and publish your own modules. diff --git a/docs/modules/developing-modules.md b/docs/modules/developing-modules.md new file mode 100644 index 0000000000..81585c2141 --- /dev/null +++ b/docs/modules/developing-modules.md @@ -0,0 +1,172 @@ +(dev-modules-page)= + +# Developing modules + +Learn how to create modules for sharing through the Nextflow module registry. +For local module syntax (inclusion, aliases, templates, and binaries), see {ref}`module-page`. + +(dev-modules-creating)= + +## Creating a module + +Use the `module create` command to scaffold a new module with the required files: + +```console +$ nextflow module create myorg/my-module +``` + +If you omit the name, the command prompts you for details: + +```console +$ nextflow module create +``` + +The command creates a module directory with the following files: + +- `main.nf`: The module script containing your process definition. +- `meta.yml`: The module spec describing metadata, inputs, and outputs. +- `README.md`: Documentation for the module. + +See {ref}`cli-module-create` for the full command reference. + +(dev-modules-structure)= + +## Module structure + +Registry modules follow a standard directory structure: + +``` +modules/ +└── myorg/ + └── my-module/ + ├── .module-info # Integrity checksum (generated at install) + ├── README.md # Documentation (required for publishing) + ├── main.nf # Module script (required) + ├── meta.yml # Module spec (required for publishing) + ├── resources/ # Optional: module binaries and resources + └── templates/ # Optional: process templates +``` + +### main.nf + +The `main.nf` file contains the process (or workflow/function) definition. For example, a simple module wrapping FastQC: + +```nextflow +process FASTQC { + tag "$meta.id" + label 'process_medium' + + conda 'bioconda::fastqc=0.12.1' + container "${ workflow.containerEngine == 'singularity' + ? 'https://depot.galaxyproject.org/singularity/fastqc:0.12.1--hdfd78af_0' + : 'biocontainers/fastqc:0.12.1--hdfd78af_0' }" + + input: + tuple val(meta), path(reads) + + output: + tuple val(meta), path("*.html"), emit: html + tuple val(meta), path("*.zip") , emit: zip + + script: + """ + fastqc $reads --threads $task.cpus + """ +} +``` + +### meta.yml + +The `meta.yml` file describes the module's metadata, including its name, version, description, authors, and input/output specifications. Publishing requires this file, and the registry uses it to display module information and generate usage templates. + +### README.md + +The `README.md` file provides documentation for the module. It should describe what the module does, the tools it wraps, and any configuration requirements. + +### templates + +Include process script {ref}`templates ` alongside a module in the `templates/` directory. +See {ref}`module-templates` for details. + +### resources + +Modules can include binary scripts in the `resources/usr/bin/` directory that are locally scoped to the module's processes. See {ref}`module-binaries` for details. + +(dev-modules-spec)= + +## Generating a module spec + +Use the `module spec` command to generate or update the `meta.yml` file from the module's `main.nf`: + +```console +$ nextflow module spec myorg/my-module +``` + +Provide metadata fields directly to avoid `TODO` placeholders in the generated file: + +```console +$ nextflow module spec \ + -namespace myorg \ + -version 1.0.0 \ + -description "Quality control of raw sequencing reads" \ + -license MIT \ + -author "@myname" \ + ./modules/myorg/my-module +``` + +Use `-dry-run` to preview the generated spec without writing to disk: + +```console +$ nextflow module spec -dry-run myorg/my-module +``` + +If a `meta.yml` already exists, the command incorporates its content into the new file. + +See {ref}`cli-module-spec` for the full command reference. + +(dev-modules-validate)= + +## Validating a module + +Use the `module validate` command to check that a module is ready for publishing: + +```console +$ nextflow module validate myorg/my-module +``` + +The command verifies that: + +- All required files are present (`main.nf`, `meta.yml`, `README.md`). +- The module spec contains all required fields (name, version, description, license). + +Validate by path: + +```console +$ nextflow module validate ./modules/myorg/my-module +``` + +See {ref}`cli-module-validate` for the full command reference. + +(dev-modules-testing)= + +## Testing a module + +Before publishing, test your module by running it directly: + +```console +$ nextflow module run myorg/my-module --input 'test-data/*.fastq.gz' +``` + +The command executes the module as a standalone workflow, allowing you to verify that inputs are correctly declared, the process runs successfully, and outputs appear as expected. + +For more thorough testing, create a small wrapper workflow that exercises the module: + +```nextflow +include { MY_MODULE } from './modules/myorg/my-module' + +workflow { + input_ch = Channel.fromFilePairs('test-data/*_{1,2}.fastq.gz') + MY_MODULE(input_ch) + MY_MODULE.out.results.view() +} +``` diff --git a/docs/modules/module-registry.md b/docs/modules/module-registry.md new file mode 100644 index 0000000000..187dd0db11 --- /dev/null +++ b/docs/modules/module-registry.md @@ -0,0 +1,80 @@ +(module-registry-page)= + +# Nextflow module registry + +:::{versionadded} 26.04.0 +::: + +The [Nextflow module registry](https://registry.nextflow.io) is a central repository for sharing Nextflow modules. +It provides module discovery, version tracking, and integrity verification. + +## Publishing modules + +Use the `module publish` command to upload a module to the registry: + +```console +$ nextflow module publish myorg/my-module +``` + +Provide a `namespace/name` reference (for an already-installed module) or a local directory path containing the module files. + +The module directory must include the following files: + +- `main.nf`: The module script. +- `meta.yml`: The module spec with name, version, description, and license. +- `README.md`: Module documentation. + +Use `-dry-run` to validate the module structure without uploading: + +```console +$ nextflow module publish myorg/my-module -dry-run +``` + +Use the {ref}`module spec ` and {ref}`module validate ` commands to prepare and verify a module before publishing. +See {ref}`dev-modules-page` for a guide on creating modules. + +See {ref}`cli-module-publish` for the full command reference. + +## Authentication + +Publishing requires authentication. +Provide your registry token in one of two ways: + +**Environment variable:** + +```console +$ export NXF_REGISTRY_TOKEN=your-token +$ nextflow module publish myorg/my-module +``` + +**Nextflow configuration:** + +```groovy +registry { + apiKey = 'YOUR_REGISTRY_TOKEN' +} +``` + +## Registry configuration + +By default, Nextflow uses the public registry at `https://registry.nextflow.io`. +Configure alternative or additional registries in your Nextflow configuration: + +```groovy +registry { + url = [ + 'https://private.registry.myorg.com', + 'https://registry.nextflow.io' + ] + apiKey = '${MYORG_TOKEN}' +} +``` + +Nextflow queries registries in the order you specify until it finds a module. +The `apiKey` is used only for the primary (first) registry. + +Specify a registry when publishing: + +```console +$ nextflow module publish myorg/my-module -registry 'https://private.registry.myorg.com' +``` diff --git a/docs/modules/modules.md b/docs/modules/modules.md new file mode 100644 index 0000000000..04a47362fa --- /dev/null +++ b/docs/modules/modules.md @@ -0,0 +1,56 @@ +(modules-page)= + +# Overview + +## What are modules + +Nextflow modules are reusable units of pipeline logic -- processes, workflows, and functions -- that you can share across projects. +By packaging common tasks as modules, you avoid duplicating code and benefit from community improvements. + +There are two ways to use modules in Nextflow: local modules and registry modules. + +

Local modules

+ +Local modules are Nextflow scripts stored directly in your project. You include definitions from local modules using the `include` keyword with a relative path: + +```nextflow +include { FASTQC } from './modules/fastqc' +``` + +Local modules are well suited for project-specific logic that is not intended for reuse across projects. See {ref}`module-page` for details on module inclusion syntax, aliases, templates, and binaries. + +

Registry modules

+ +:::{versionadded} 26.04.0 +::: + +Centralized registries host registry modules, and you manage them with the `nextflow module` command. +They follow a standard structure with metadata (`meta.yml`), documentation (`README.md`), and a module script (`main.nf`) for version management, integrity checking, and discoverability. + +Install registry modules into your project and include them by name: + +```console +$ nextflow module install nf-core/fastqc +``` + +```nextflow +include { FASTQC } from 'nf-core/fastqc' +``` + +See {ref}`using-modules-page` for details on discovering, installing, and managing registry modules. + +## Versioning + +Registry modules use version strings to identify releases. Many modules use [Semantic Versioning](https://semver.org/): + +* MAJOR: Increment for backward-incompatible changes. +* MINOR: Increment for backward-compatible feature additions. +* PATCH: Increment for backward-compatible bug fixes. + +When installing a module, you can pin a specific version or let Nextflow use the latest available version: + +```console +$ nextflow module install nf-core/fastqc -version 1.0.0 +``` + +See {ref}`cli-module-install` for more information. diff --git a/docs/modules/using-modules.md b/docs/modules/using-modules.md new file mode 100644 index 0000000000..bf7777431a --- /dev/null +++ b/docs/modules/using-modules.md @@ -0,0 +1,140 @@ +(using-modules-page)= + +# Using modules + +:::{versionadded} 26.04.0 +::: + +The Nextflow module system allows you to discover, install, and manage reusable modules from centralized registries. +This page describes how to use registry modules in your pipelines. +For local module syntax (inclusion, aliases, templates, and binaries), see {ref}`module-page`. + +## Discovering modules + +Search for available modules using the `module search` command: + +```console +$ nextflow module search alignment +$ nextflow module search "quality control" -limit 10 +``` + +Results include module names, versions, descriptions, and download statistics. +Use `-output json` for machine-readable output. + +See {ref}`cli-module-search` for the full command reference. + +## Installing modules + +Use the `module install` command to download modules from a registry into your project: + +```console +$ nextflow module install nf-core/fastqc +$ nextflow module install nf-core/fastqc -version 1.0.0 +``` + +Nextflow stores installed modules in the `modules/` directory and creates a `.module-info` file alongside the module to record installation metadata such as the module checksum and registry URL. + +Once installed, include a module by name rather than a relative path: + +```nextflow +include { FASTQC } from 'nf-core/fastqc' + +workflow { + reads = Channel.fromFilePairs('data/*_{1,2}.fastq.gz') + FASTQC(reads) +} +``` + +:::{tip} +Commit the `modules/` directory to your Git repository to ensure reproducibility. +::: + +See {ref}`cli-module-install` for the full command reference. + +## Listing installed modules + +View all modules installed in your project with the `module list` command: + +```console +$ nextflow module list +``` + +The output shows each module's name, installed version, and whether it has been modified locally. +Use `-output json` for machine-readable output. + +See {ref}`cli-module-list` for the full command reference. + +## Viewing module information + +Use the `module info` command to view metadata and a usage template for a module: + +```console +$ nextflow module info nf-core/fastqc +$ nextflow module info nf-core/fastqc -version 1.0.0 +``` + +The output includes the module's description, authors, keywords, tools, input/output channels, and a generated usage template. +Use `-output json` for machine-readable output. + +See {ref}`cli-module-info` for the full command reference. + +## Running modules directly + +For ad-hoc tasks or testing, run a module directly without creating a wrapper workflow: + +```console +$ nextflow module run nf-core/fastqc --input 'data/*.fastq.gz' +``` + +The command automatically downloads the module if it is not already installed. +It accepts all standard Nextflow run options (`-profile`, `-resume`, etc.): + +```console +$ nextflow module run nf-core/salmon \ + --reads reads.fq \ + --index salmon_index \ + -profile docker \ + -resume +``` + +Nextflow infers command-line params (e.g., `--reads`) from the module's declared inputs. +Run `nextflow module info` to see the available inputs for a module. + +See {ref}`cli-module-run` for the full command reference. + +## Updating modules + +To update a module to a newer version, reinstall it with the desired version: + +```console +$ nextflow module install nf-core/fastqc -version 1.1.0 +``` + +If you have local modifications, Nextflow warns you and prevents the update. +Use the `-force` flag to override: + +```console +$ nextflow module install nf-core/fastqc -version 1.1.0 -force +``` + +## Checksum verification + +Nextflow automatically verifies module integrity using checksums stored in the `.module-info` file. When you modify a module locally, Nextflow detects the change and prevents accidental overwrites during reinstallation. + +Module integrity ensures that modules remain consistent with their registry versions unless you explicitly choose to override them. + +## Removing modules + +Use the `module remove` command to uninstall a module from your project: + +```console +$ nextflow module remove nf-core/fastqc +``` + +By default, Nextflow removes both the module files and the `.module-info` file. +Use the following flags to control this behavior: + +- `-keep-files`: Remove the `.module-info` file but keep the module files in the `modules/` directory. +- `-force`: Force removal even if the module has no `.module-info` file or has local modifications. + +See {ref}`cli-module-remove` for the full command reference. diff --git a/docs/reference/cli.md b/docs/reference/cli.md index 44ac9e31a6..bb295bf6ad 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -1152,6 +1152,8 @@ The `module` command provides a comprehensive system for managing registry-based **Subcommands** +(cli-module-create)= + `create [namespace/name]` : Create a new module with a basic `main.nf`, `meta.yml`, and `README.md`. From 7e73921b17defbb46dadf298e63966ea5f2b6866 Mon Sep 17 00:00:00 2001 From: Christopher Hakkaart Date: Tue, 14 Apr 2026 14:47:24 +1200 Subject: [PATCH 02/14] Improve overview page Signed-off-by: Christopher Hakkaart --- docs/modules/module-registry.md | 108 ++++++++++++++++++++++++++------ docs/modules/modules.md | 32 +++++++--- 2 files changed, 114 insertions(+), 26 deletions(-) diff --git a/docs/modules/module-registry.md b/docs/modules/module-registry.md index 187dd0db11..c1b0d70b6a 100644 --- a/docs/modules/module-registry.md +++ b/docs/modules/module-registry.md @@ -8,36 +8,81 @@ The [Nextflow module registry](https://registry.nextflow.io) is a central repository for sharing Nextflow modules. It provides module discovery, version tracking, and integrity verification. -## Publishing modules +The module registry shares its login, access tokens, and web interface with the {ref}`Nextflow plugin registry `. -Use the `module publish` command to upload a module to the registry: +(module-registry-claim)= -```console -$ nextflow module publish myorg/my-module -``` +## Claiming a namespace -Provide a `namespace/name` reference (for an already-installed module) or a local directory path containing the module files. +A namespace is an organizational prefix that groups related modules together (e.g., the namespace `nf-core` owns modules like `nf-core/fastqc` and `nf-core/samtools`). +As a namespace member, you can publish new modules and release new versions under that prefix. +Each namespace can have multiple members who share publishing rights. -The module directory must include the following files: +To claim a namespace: -- `main.nf`: The module script. -- `meta.yml`: The module spec with name, version, description, and license. -- `README.md`: Module documentation. +1. Open the [Nextflow module registry](https://registry.nextflow.io/) in a browser. -Use `-dry-run` to validate the module structure without uploading: +2. Log in to [Seqera](https://cloud.seqera.io/login) with your GitHub or Google account, or by providing an email address. -```console -$ nextflow module publish myorg/my-module -dry-run -``` + :::{note} + If you are logging in for the first time, Seqera sends an authentication link to your email address to complete the login process. + ::: -Use the {ref}`module spec ` and {ref}`module validate ` commands to prepare and verify a module before publishing. -See {ref}`dev-modules-page` for a guide on creating modules. +3. Go to **My Resources** and select the **My Namespaces** tab. -See {ref}`cli-module-publish` for the full command reference. +4. Enter your namespace name in the **Claim Namespace** field. + +5. Select **Claim Namespace**. + +The namespace shows as **PENDING REVIEW** until an administrator approves the request. +Admin approval is required only once. +Once approved, you can publish modules using the `nextflow module publish` command. + +(module-registry-access-token)= + +## Creating an access token + +An API access token is required to publish modules to the registry. + +To create an API access token: + +1. Open the [Nextflow module registry](https://registry.nextflow.io/) in a browser. + +2. Log in to [Seqera](https://cloud.seqera.io/login) with your GitHub or Google account, or by providing an email address. + + :::{note} + If you are logging in for the first time, Seqera sends an authentication link to your email address to complete the login process. + ::: + +3. Go to **My Resources** and select the **My access tokens** tab. + +4. Under **Create New Access Token**, enter a descriptive name for the **Token name** and select the token duration from the **Expiry** drop down. + +5. Select **Generate Token**. + +6. Copy the token and store it somewhere safe. + You cannot view it again after this step. + +Once you have your token, see [Publishing modules](#publish-modules) for instructions on how to use it. + +## Viewing your modules + +View the modules you have published in the registry: -## Authentication +1. Open the [Nextflow module registry](https://registry.nextflow.io/) in a browser. + +2. Log in and go to **My Resources**. + +3. Select the **My Modules** tab. + +The page lists all modules published under your namespaces, along with their versions and status. + +(publish-modules)= + +## Publishing modules + +Publishing requires a {ref}`claimed namespace ` and an {ref}`access token `. -Publishing requires authentication. Provide your registry token in one of two ways: **Environment variable:** @@ -55,6 +100,31 @@ registry { } ``` +Use the `module publish` command to upload a module to the registry: + +```console +$ nextflow module publish myorg/my-module +``` + +Provide a `namespace/name` reference (for an already-installed module) or a local directory path containing the module files. + +The module directory must include the following files: + +- `main.nf`: The module script. +- `meta.yml`: The module spec with name, version, description, and license. +- `README.md`: Module documentation. + +Use `-dry-run` to validate the module structure without uploading: + +```console +$ nextflow module publish myorg/my-module -dry-run +``` + +Use the {ref}`module spec ` and {ref}`module validate ` commands to prepare and verify a module before publishing. +See {ref}`dev-modules-page` for a guide on creating modules. + +See {ref}`cli-module-publish` for the full command reference. + ## Registry configuration By default, Nextflow uses the public registry at `https://registry.nextflow.io`. diff --git a/docs/modules/modules.md b/docs/modules/modules.md index 04a47362fa..f704320797 100644 --- a/docs/modules/modules.md +++ b/docs/modules/modules.md @@ -4,12 +4,15 @@ ## What are modules -Nextflow modules are reusable units of pipeline logic -- processes, workflows, and functions -- that you can share across projects. +Nextflow modules are reusable units of pipeline logic (i.e., processes, workflows, and functions) that you can share across projects. By packaging common tasks as modules, you avoid duplicating code and benefit from community improvements. -There are two ways to use modules in Nextflow: local modules and registry modules. +There are two ways to use modules in Nextflow: -

Local modules

+- [Local modules](#local-modules) +- [Registry modules](#registry-modules) + +### Local modules Local modules are Nextflow scripts stored directly in your project. You include definitions from local modules using the `include` keyword with a relative path: @@ -17,15 +20,28 @@ Local modules are Nextflow scripts stored directly in your project. You include include { FASTQC } from './modules/fastqc' ``` -Local modules are well suited for project-specific logic that is not intended for reuse across projects. See {ref}`module-page` for details on module inclusion syntax, aliases, templates, and binaries. +Local modules are well suited for project-specific logic that is not intended for reuse or sharing. See {ref}`module-page` for details on module inclusion syntax, aliases, templates, and binaries. -

Registry modules

+### Registry modules :::{versionadded} 26.04.0 ::: -Centralized registries host registry modules, and you manage them with the `nextflow module` command. -They follow a standard structure with metadata (`meta.yml`), documentation (`README.md`), and a module script (`main.nf`) for version management, integrity checking, and discoverability. +The [Nextflow module registry](https://registry.nextflow.io) hosts registry modules in a centralized repository. +You manage them with the `nextflow module` command. +Registry modules follow a standard structure with metadata (`meta.yml`), documentation (`README.md`), and a module script (`main.nf`) for version management, integrity checking, and discoverability. + +Key features of registry modules: + +- **Discoverability**: Search for modules by keyword or name and browse available versions. +- **Version management**: Pin specific versions or use the latest release, with automatic integrity checking via checksums. +- **Direct execution**: Run modules as standalone workflows for ad-hoc tasks or testing without writing a wrapper script. +- **Standard structure**: Each module includes a script (`main.nf`), metadata (`meta.yml`), and documentation (`README.md`), enabling consistent tooling and automation. + +:::{note} +Modules from the [nf-core](https://nf-co.re/) community are automatically mirrored to the Nextflow module registry under the `nf-core` namespace. +You can install and use them directly without any additional configuration. +::: Install registry modules into your project and include them by name: @@ -38,6 +54,8 @@ include { FASTQC } from 'nf-core/fastqc' ``` See {ref}`using-modules-page` for details on discovering, installing, and managing registry modules. +To create and publish your own modules, see {ref}`dev-modules-page`. +To learn about the registry itself (namespaces, access tokens, and configuration), see {ref}`module-registry-page`. ## Versioning From c79c2576b0db3a9773861413d29fb7bb46bc47ab Mon Sep 17 00:00:00 2001 From: Christopher Hakkaart Date: Tue, 14 Apr 2026 15:10:07 +1200 Subject: [PATCH 03/14] Add more details Signed-off-by: Christopher Hakkaart --- docs/modules/modules.md | 8 +++++--- docs/modules/using-modules.md | 3 ++- docs/plugins/plugin-registry.md | 4 ---- 3 files changed, 7 insertions(+), 8 deletions(-) diff --git a/docs/modules/modules.md b/docs/modules/modules.md index f704320797..327617b5be 100644 --- a/docs/modules/modules.md +++ b/docs/modules/modules.md @@ -53,9 +53,11 @@ $ nextflow module install nf-core/fastqc include { FASTQC } from 'nf-core/fastqc' ``` -See {ref}`using-modules-page` for details on discovering, installing, and managing registry modules. -To create and publish your own modules, see {ref}`dev-modules-page`. -To learn about the registry itself (namespaces, access tokens, and configuration), see {ref}`module-registry-page`. +For more registry information: + +- See {ref}`using-modules-page` for details on discovering, installing, and managing registry modules. +- See {ref}`dev-modules-page` for details on creating and publishing your own modules. +- See {ref}`module-registry-page` for details on namespaces, access tokens, and registry configuration. ## Versioning diff --git a/docs/modules/using-modules.md b/docs/modules/using-modules.md index bf7777431a..4d9a6e9216 100644 --- a/docs/modules/using-modules.md +++ b/docs/modules/using-modules.md @@ -119,7 +119,8 @@ $ nextflow module install nf-core/fastqc -version 1.1.0 -force ## Checksum verification -Nextflow automatically verifies module integrity using checksums stored in the `.module-info` file. When you modify a module locally, Nextflow detects the change and prevents accidental overwrites during reinstallation. +Nextflow automatically verifies module integrity using checksums stored in the `.module-info` file. +When you modify a module locally, Nextflow detects the change and prevents accidental overwrites during reinstallation. Module integrity ensures that modules remain consistent with their registry versions unless you explicitly choose to override them. diff --git a/docs/plugins/plugin-registry.md b/docs/plugins/plugin-registry.md index b0f5b96d55..ef1c691e0d 100644 --- a/docs/plugins/plugin-registry.md +++ b/docs/plugins/plugin-registry.md @@ -6,10 +6,6 @@ The [Nextflow plugin registry](https://registry.nextflow.io/) is a central repos Nextflow 25.10 and later can use the plugin registry as a drop-in replacement for the [legacy plugin index](https://github.com/nextflow-io/plugins) hosted on GitHub. See {ref}`migrate-plugin-page` for more information about migrating to the Nextflow plugin registry. -:::{note} -The Nextflow plugin registry is currently available as a public preview. Plugin developers can access the registry by contacting [info@nextflow.io](mailto:info@nextflow.io) to obtain access to the registry. -::: - See {ref}`gradle-plugin-publish` for instructions on how to publish plugins to the registry, including the {ref}`README.md requirement `. (plugin-registry-claim)= From 6d02920efcd11e9ae4e746eda3b9487098361f3e Mon Sep 17 00:00:00 2001 From: Christopher Hakkaart Date: Tue, 14 Apr 2026 17:30:12 +1200 Subject: [PATCH 04/14] Make reference cli page consistent Signed-off-by: Christopher Hakkaart --- docs/modules/modules.md | 24 +-- docs/reference/cli.md | 332 ++++++++++++++++++++++------------------ 2 files changed, 189 insertions(+), 167 deletions(-) diff --git a/docs/modules/modules.md b/docs/modules/modules.md index 327617b5be..001ef78f12 100644 --- a/docs/modules/modules.md +++ b/docs/modules/modules.md @@ -2,8 +2,6 @@ # Overview -## What are modules - Nextflow modules are reusable units of pipeline logic (i.e., processes, workflows, and functions) that you can share across projects. By packaging common tasks as modules, you avoid duplicating code and benefit from community improvements. @@ -12,7 +10,7 @@ There are two ways to use modules in Nextflow: - [Local modules](#local-modules) - [Registry modules](#registry-modules) -### Local modules +## Local modules Local modules are Nextflow scripts stored directly in your project. You include definitions from local modules using the `include` keyword with a relative path: @@ -22,7 +20,7 @@ include { FASTQC } from './modules/fastqc' Local modules are well suited for project-specific logic that is not intended for reuse or sharing. See {ref}`module-page` for details on module inclusion syntax, aliases, templates, and binaries. -### Registry modules +## Registry modules :::{versionadded} 26.04.0 ::: @@ -53,24 +51,8 @@ $ nextflow module install nf-core/fastqc include { FASTQC } from 'nf-core/fastqc' ``` -For more registry information: +For more registry modules information: - See {ref}`using-modules-page` for details on discovering, installing, and managing registry modules. - See {ref}`dev-modules-page` for details on creating and publishing your own modules. - See {ref}`module-registry-page` for details on namespaces, access tokens, and registry configuration. - -## Versioning - -Registry modules use version strings to identify releases. Many modules use [Semantic Versioning](https://semver.org/): - -* MAJOR: Increment for backward-incompatible changes. -* MINOR: Increment for backward-compatible feature additions. -* PATCH: Increment for backward-compatible bug fixes. - -When installing a module, you can pin a specific version or let Nextflow use the latest available version: - -```console -$ nextflow module install nf-core/fastqc -version 1.0.0 -``` - -See {ref}`cli-module-install` for more information. diff --git a/docs/reference/cli.md b/docs/reference/cli.md index bb295bf6ad..b82236aca7 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -1158,16 +1158,6 @@ The `module` command provides a comprehensive system for managing registry-based : Create a new module with a basic `main.nf`, `meta.yml`, and `README.md`. -: **Examples:** - - ```console - # Create a module (it will prompt for details) - $ nextflow module create - - # Create a module with the given name - $ nextflow module create myorg/my-module - ``` - (cli-module-info)= `info [options] [namespace/name]` @@ -1182,19 +1172,6 @@ The `module` command provides a comprehensive system for managing registry-based `-o, -output` (`text`) : Output mode for info results. Options: `text` (default), `json`. -: **Examples:** - - ```console - # Display information for latest version - $ nextflow module info nf-core/fastqc - - # Display information for specific version - $ nextflow module info nf-core/fastqc -version 1.0.0 - - # Get results as JSON - $ nextflow module info nf-core/fastqc -output json - ``` - (cli-module-install)= `install [options] [namespace/name]` @@ -1210,19 +1187,6 @@ The `module` command provides a comprehensive system for managing registry-based `-force` : Force reinstall even if the module exists locally with modifications. Without this flag, Nextflow prevents overwriting locally modified modules. -: **Examples:** - - ```console - # Install latest version - $ nextflow module install nf-core/fastqc - - # Install specific version - $ nextflow module install nf-core/fastqc -version 1.0.0 - - # Force reinstall over local modifications - $ nextflow module install nf-core/fastqc -force - ``` - (cli-module-list)= `list [options]` @@ -1234,16 +1198,6 @@ The `module` command provides a comprehensive system for managing registry-based `-o, -output` (`table`) : Output mode for list results. Options: `table` (default), `json`. -: **Examples:** - - ```console - # Display installed modules in formatted table - $ nextflow module list - - # Output as JSON - $ nextflow module list -output 'json' - ``` - (cli-module-publish)= `publish [options] [namespace/name | path]` @@ -1260,21 +1214,6 @@ The `module` command provides a comprehensive system for managing registry-based `-registry` : Specify the registry to publish the module (default: `https://registry.nextflow.io`) -: **Examples:** - - ```console - # Validate module structure without publishing - $ nextflow module publish myorg/my-module -dry-run - - # Publish to nextflow registry - $ export NXF_REGISTRY_TOKEN=your-token - $ nextflow module publish myorg/my-module - - # Publish to a custom registry - $ export NXF_REGISTRY_TOKEN=your-token - $ nextflow module publish myorg/my-module -registry 'https://custom.registry.com' - ``` - (cli-module-remove)= `remove [options] [namespace/name]` @@ -1289,16 +1228,6 @@ The `module` command provides a comprehensive system for managing registry-based `-keep-files` : Remove the `.module-info` but keep local files in the `modules/` directory. -: **Examples:** - - ```console - # Remove module completely - $ nextflow module remove nf-core/fastqc - - # Remove from config but keep local files - $ nextflow module remove nf-core/fastqc -keep-files - ``` - (cli-module-run)= `run [options] [namespace/name] [-- ]` @@ -1311,20 +1240,6 @@ The `module` command provides a comprehensive system for managing registry-based `-version` : Specify the module version to run (e.g., `1.0.0`). If not specified, uses the latest version. -: **Examples:** - - ```console - # Run module with inputs - $ nextflow module run nf-core/fastqc --input 'data/*.fastq.gz' - - # Run specific version with Nextflow options - $ nextflow module run nf-core/fastqc \ - --input 'data/*.fastq.gz' \ - -version 1.0.0 \ - -profile docker \ - -resume - ``` - (cli-module-search)= `search [options] [query]` @@ -1339,19 +1254,6 @@ The `module` command provides a comprehensive system for managing registry-based `-o, -output` (`simple`) : Output mode for search results. Options: `simple` (default), `json`. -: **Examples:** - - ```console - # Search for alignment-related modules - $ nextflow module search alignment - - # Search with limited results - $ nextflow module search "quality control" -limit 10 - - # Get results as JSON - $ nextflow module search bwa -output json - ``` - (cli-module-spec)= `spec [options] ` @@ -1379,29 +1281,6 @@ The `module` command provides a comprehensive system for managing registry-based `-dry-run` : Print the generated spec to stdout without writing any file. -: **Examples:** - - ```console - # Generate spec for local module by name - $ nextflow module spec nf-core/fastqc - - # Generate spec for local module by path (namespace required) - $ nextflow module spec -namespace nf-core ./modules/my-module - - # Provide additional fields to avoid TODO placeholders - $ nextflow module spec \ - -namespace nf-core \ - -version 1.0.0 \ - -description "Quality control of raw sequencing reads" \ - -license MIT \ - -author "@drpatelh" \ - -author "@joseespinosa" \ - ./modules/nf-core/fastqc - - # Print module spec without saving it - $ nextflow module spec -dry-run -namespace nf-core ./modules/my-module - ``` - (cli-module-validate)= `validate [options] ` @@ -1409,15 +1288,176 @@ The `module` command provides a comprehensive system for managing registry-based : Validate a module before publishing to the registry. : Verifies that all required files are present (`main.nf`, `meta.yml`, `README.md`) and that the module spec contains all required fields (name, version, description, license). -: **Examples:** +**Examples** - ```console - # Validate module by name - $ nextflow module validate myorg/my-module +Create a module and prompt for details: - # Validate module by path - $ nextflow module validate ./modules/myorg/my-module - ``` +```console +$ nextflow module create +``` + +Create a module with a given name: + +```console +$ nextflow module create myorg/my-module +``` + +Search for alignment-related modules: + +```console +$ nextflow module search alignment +``` + +Search and limit the number of results: + +```console +$ nextflow module search "quality control" -limit 10 +``` + +Search and output results as JSON: + +```console +$ nextflow module search bwa -output json +``` + +Display information for a module: + +```console +$ nextflow module info nf-core/fastqc +``` + +Display information for a specific module version: + +```console +$ nextflow module info nf-core/fastqc -version 1.0.0 +``` + +Display module information as JSON: + +```console +$ nextflow module info nf-core/fastqc -output json +``` + +Install the latest version of a module: + +```console +$ nextflow module install nf-core/fastqc +``` + +Install a specific version of a module: + +```console +$ nextflow module install nf-core/fastqc -version 1.0.0 +``` + +Force a reinstall over local modifications: + +```console +$ nextflow module install nf-core/fastqc -force +``` + +List installed modules in a formatted table: + +```console +$ nextflow module list +``` + +List installed modules as JSON: + +```console +$ nextflow module list -output json +``` + +Run a module with inputs: + +```console +$ nextflow module run nf-core/fastqc --input 'data/*.fastq.gz' +``` + +Run a specific version with Nextflow options: + +```console +$ nextflow module run nf-core/fastqc \ + --input 'data/*.fastq.gz' \ + -version 1.0.0 \ + -profile docker \ + -resume +``` + +Remove a module completely: + +```console +$ nextflow module remove nf-core/fastqc +``` + +Remove a module from config but keep local files: + +```console +$ nextflow module remove nf-core/fastqc -keep-files +``` + +Generate a spec for a local module by name: + +```console +$ nextflow module spec nf-core/fastqc +``` + +Generate a spec for a local module by path (namespace required): + +```console +$ nextflow module spec -namespace nf-core ./modules/my-module +``` + +Provide additional fields to avoid TODO placeholders: + +```console +$ nextflow module spec \ + -namespace nf-core \ + -version 1.0.0 \ + -description "Quality control of raw sequencing reads" \ + -license MIT \ + -author "@drpatelh" \ + -author "@joseespinosa" \ + ./modules/nf-core/fastqc +``` + +Print a module spec without saving it: + +```console +$ nextflow module spec -dry-run -namespace nf-core ./modules/my-module +``` + +Validate a module by name: + +```console +$ nextflow module validate myorg/my-module +``` + +Validate a module by path: + +```console +$ nextflow module validate ./modules/myorg/my-module +``` + +Validate a module structure without publishing: + +```console +$ nextflow module publish myorg/my-module -dry-run +``` + +Publish a module to the Nextflow registry: + +```console +$ export NXF_REGISTRY_TOKEN=your-token +$ nextflow module publish myorg/my-module +``` + +Publish a module to a custom registry: + +```console +$ export NXF_REGISTRY_TOKEN=your-token +$ nextflow module publish myorg/my-module -registry 'https://custom.registry.com' +``` (cli-plugin)= @@ -1805,29 +1845,29 @@ $ nextflow secrets [OPTIONS] **Examples** -- Set a secret: +Set a secret: - ```console - $ nextflow secrets set FOO "Hello world" - ``` +```console +$ nextflow secrets set FOO "Hello world" +``` -- List secrets: +List secrets: - ```console - $ nextflow secrets list - ``` +```console +$ nextflow secrets list +``` -- Get a secret: +Get a secret: - ```console - $ nextflow secrets get FOO - ``` +```console +$ nextflow secrets get FOO +``` -- Delete a secret: +Delete a secret: - ```console - $ nextflow secrets delete FOO - ``` +```console +$ nextflow secrets delete FOO +``` ### `self-update` @@ -1845,7 +1885,7 @@ The `self-update` command directs the `nextflow` CLI to update itself to the lat **Examples** -Update Nextflow. +Update Nextflow: ```console $ nextflow self-update @@ -1895,7 +1935,7 @@ The `view` command is used to inspect the pipelines that are already stored in t **Examples** -Viewing the contents of a downloaded pipeline. +View the contents of a downloaded pipeline: ```console $ nextflow view nextflow-io/hello From da203aa3d1894ca0d70e6215faa1596c545d9e79 Mon Sep 17 00:00:00 2001 From: Christopher Hakkaart Date: Tue, 14 Apr 2026 17:32:11 +1200 Subject: [PATCH 05/14] Small fix Signed-off-by: Christopher Hakkaart --- docs/reference/cli.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/reference/cli.md b/docs/reference/cli.md index b82236aca7..ff99872401 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -1448,14 +1448,14 @@ $ nextflow module publish myorg/my-module -dry-run Publish a module to the Nextflow registry: ```console -$ export NXF_REGISTRY_TOKEN=your-token +$ export NXF_REGISTRY_TOKEN= $ nextflow module publish myorg/my-module ``` Publish a module to a custom registry: ```console -$ export NXF_REGISTRY_TOKEN=your-token +$ export NXF_REGISTRY_TOKEN= $ nextflow module publish myorg/my-module -registry 'https://custom.registry.com' ``` From f98511f4c5d787a920ca01dbabd02e4321643421 Mon Sep 17 00:00:00 2001 From: Christopher Hakkaart Date: Wed, 15 Apr 2026 10:32:43 +1200 Subject: [PATCH 06/14] Check commands Signed-off-by: Christopher Hakkaart --- docs/modules/developing-modules.md | 12 +++------- docs/modules/module-registry.md | 6 ++--- docs/modules/modules.md | 8 +++---- docs/modules/using-modules.md | 36 +++++++++++++++++------------- 4 files changed, 31 insertions(+), 31 deletions(-) diff --git a/docs/modules/developing-modules.md b/docs/modules/developing-modules.md index 81585c2141..dacb1f2af7 100644 --- a/docs/modules/developing-modules.md +++ b/docs/modules/developing-modules.md @@ -3,7 +3,7 @@ # Developing modules Learn how to create modules for sharing through the Nextflow module registry. -For local module syntax (inclusion, aliases, templates, and binaries), see {ref}`module-page`. +For information about modules syntax, see {ref}`module-page`. (dev-modules-creating)= @@ -39,7 +39,7 @@ Registry modules follow a standard directory structure: modules/ └── myorg/ └── my-module/ - ├── .module-info # Integrity checksum (generated at install) + ├── .module-info # Integrity checksum (generated at install/creation) ├── README.md # Documentation (required for publishing) ├── main.nf # Module script (required) ├── meta.yml # Module spec (required for publishing) @@ -137,13 +137,7 @@ $ nextflow module validate myorg/my-module The command verifies that: - All required files are present (`main.nf`, `meta.yml`, `README.md`). -- The module spec contains all required fields (name, version, description, license). - -Validate by path: - -```console -$ nextflow module validate ./modules/myorg/my-module -``` +- The module spec contains all required fields (name, version, description, and license). See {ref}`cli-module-validate` for the full command reference. diff --git a/docs/modules/module-registry.md b/docs/modules/module-registry.md index c1b0d70b6a..d77567f8cf 100644 --- a/docs/modules/module-registry.md +++ b/docs/modules/module-registry.md @@ -65,7 +65,7 @@ To create an API access token: Once you have your token, see [Publishing modules](#publish-modules) for instructions on how to use it. -## Viewing your modules +## Viewing modules View the modules you have published in the registry: @@ -88,7 +88,7 @@ Provide your registry token in one of two ways: **Environment variable:** ```console -$ export NXF_REGISTRY_TOKEN=your-token +$ export NXF_REGISTRY_TOKEN= $ nextflow module publish myorg/my-module ``` @@ -96,7 +96,7 @@ $ nextflow module publish myorg/my-module ```groovy registry { - apiKey = 'YOUR_REGISTRY_TOKEN' + apiKey = '' } ``` diff --git a/docs/modules/modules.md b/docs/modules/modules.md index 001ef78f12..aa9997d116 100644 --- a/docs/modules/modules.md +++ b/docs/modules/modules.md @@ -51,8 +51,8 @@ $ nextflow module install nf-core/fastqc include { FASTQC } from 'nf-core/fastqc' ``` -For more registry modules information: +For more information about registry modules, see: -- See {ref}`using-modules-page` for details on discovering, installing, and managing registry modules. -- See {ref}`dev-modules-page` for details on creating and publishing your own modules. -- See {ref}`module-registry-page` for details on namespaces, access tokens, and registry configuration. +- {ref}`using-modules-page` for details on discovering, installing, and managing registry modules. +- {ref}`dev-modules-page` for details on creating and publishing your own modules. +- {ref}`module-registry-page` for details on namespaces, access tokens, and registry configuration. diff --git a/docs/modules/using-modules.md b/docs/modules/using-modules.md index 4d9a6e9216..bbb7c0edc1 100644 --- a/docs/modules/using-modules.md +++ b/docs/modules/using-modules.md @@ -7,7 +7,7 @@ The Nextflow module system allows you to discover, install, and manage reusable modules from centralized registries. This page describes how to use registry modules in your pipelines. -For local module syntax (inclusion, aliases, templates, and binaries), see {ref}`module-page`. +For information about modules syntax, see {ref}`module-page`. ## Discovering modules @@ -18,7 +18,7 @@ $ nextflow module search alignment $ nextflow module search "quality control" -limit 10 ``` -Results include module names, versions, descriptions, and download statistics. +Results include module names and descriptions. Use `-output json` for machine-readable output. See {ref}`cli-module-search` for the full command reference. @@ -29,9 +29,14 @@ Use the `module install` command to download modules from a registry into your p ```console $ nextflow module install nf-core/fastqc -$ nextflow module install nf-core/fastqc -version 1.0.0 +$ nextflow module install nf-core/fastqc -version 0.0.0-0c7146d ``` +:::{note} +Modules mirrored from nf-core do not follow standard semantic versioning. +Instead, they use the format `0.0.0-`, where the suffix is a short portion of the nf-core module's commit hash. +::: + Nextflow stores installed modules in the `modules/` directory and creates a `.module-info` file alongside the module to record installation metadata such as the module checksum and registry URL. Once installed, include a module by name rather than a relative path: @@ -70,10 +75,10 @@ Use the `module info` command to view metadata and a usage template for a module ```console $ nextflow module info nf-core/fastqc -$ nextflow module info nf-core/fastqc -version 1.0.0 +$ nextflow module info nf-core/fastqc -version 0.0.0-0c7146d ``` -The output includes the module's description, authors, keywords, tools, input/output channels, and a generated usage template. +The output includes the module's version, URL, description, authors, maintainers, keywords, tools, input/output channels, and a generated usage template. Use `-output json` for machine-readable output. See {ref}`cli-module-info` for the full command reference. @@ -83,23 +88,24 @@ See {ref}`cli-module-info` for the full command reference. For ad-hoc tasks or testing, run a module directly without creating a wrapper workflow: ```console -$ nextflow module run nf-core/fastqc --input 'data/*.fastq.gz' +$ nextflow module run nf-core/fastqc --meta.id=test_sample --reads sample1_R1.fastq.gz ``` +:::{tip} +Run `nextflow module info` to see the available inputs for a module. +::: + The command automatically downloads the module if it is not already installed. It accepts all standard Nextflow run options (`-profile`, `-resume`, etc.): ```console -$ nextflow module run nf-core/salmon \ - --reads reads.fq \ - --index salmon_index \ +$ nextflow module run nf-core/fastqc \ + --meta.id=test_sample \ + --reads sample1_R1.fastq.gz \ -profile docker \ -resume ``` -Nextflow infers command-line params (e.g., `--reads`) from the module's declared inputs. -Run `nextflow module info` to see the available inputs for a module. - See {ref}`cli-module-run` for the full command reference. ## Updating modules @@ -107,14 +113,14 @@ See {ref}`cli-module-run` for the full command reference. To update a module to a newer version, reinstall it with the desired version: ```console -$ nextflow module install nf-core/fastqc -version 1.1.0 +$ nextflow module install nf-core/fastqc -version 0.0.0-c9h0bv4 ``` If you have local modifications, Nextflow warns you and prevents the update. Use the `-force` flag to override: ```console -$ nextflow module install nf-core/fastqc -version 1.1.0 -force +$ nextflow module install nf-core/fastqc -version 0.0.0-c9h0bv4 -force ``` ## Checksum verification @@ -133,7 +139,7 @@ $ nextflow module remove nf-core/fastqc ``` By default, Nextflow removes both the module files and the `.module-info` file. -Use the following flags to control this behavior: +Use flags to control this behavior: - `-keep-files`: Remove the `.module-info` file but keep the module files in the `modules/` directory. - `-force`: Force removal even if the module has no `.module-info` file or has local modifications. From fd8867b2d4f2706e1ab48f32f30e64e5dbc467fa Mon Sep 17 00:00:00 2001 From: Christopher Hakkaart Date: Wed, 15 Apr 2026 11:55:33 +1200 Subject: [PATCH 07/14] Tidy docs Signed-off-by: Christopher Hakkaart --- docs/modules/module-registry.md | 66 +++++++++++++++++++++++---------- 1 file changed, 46 insertions(+), 20 deletions(-) diff --git a/docs/modules/module-registry.md b/docs/modules/module-registry.md index d77567f8cf..2adaef75ed 100644 --- a/docs/modules/module-registry.md +++ b/docs/modules/module-registry.md @@ -14,7 +14,7 @@ The module registry shares its login, access tokens, and web interface with the ## Claiming a namespace -A namespace is an organizational prefix that groups related modules together (e.g., the namespace `nf-core` owns modules like `nf-core/fastqc` and `nf-core/samtools`). +A namespace is an organizational prefix (e.g., the namespace `nf-core` owns modules like `nf-core/fastqc` and `nf-core/samtools`). As a namespace member, you can publish new modules and release new versions under that prefix. Each namespace can have multiple members who share publishing rights. @@ -30,9 +30,11 @@ To claim a namespace: 3. Go to **My Resources** and select the **My Namespaces** tab. -4. Enter your namespace name in the **Claim Namespace** field. +4. Select **Claim Namespace** -5. Select **Claim Namespace**. +4. Enter your **Namespace name**, **Organization**, **Project URL**, and a **Description**. + +5. Select **Submit Request**. The namespace shows as **PENDING REVIEW** until an administrator approves the request. Admin approval is required only once. @@ -56,7 +58,7 @@ To create an API access token: 3. Go to **My Resources** and select the **My access tokens** tab. -4. Under **Create New Access Token**, enter a descriptive name for the **Token name** and select the token duration from the **Expiry** drop down. +4. Under **Create New Access Token**, enter a descriptive name for the **Token name** and select a token duration from the **Expiry** drop down. 5. Select **Generate Token**. @@ -89,14 +91,13 @@ Provide your registry token in one of two ways: ```console $ export NXF_REGISTRY_TOKEN= -$ nextflow module publish myorg/my-module ``` **Nextflow configuration:** ```groovy registry { - apiKey = '' + apiKey = '${MYORG_TOKEN}' } ``` @@ -106,45 +107,70 @@ Use the `module publish` command to upload a module to the registry: $ nextflow module publish myorg/my-module ``` -Provide a `namespace/name` reference (for an already-installed module) or a local directory path containing the module files. - The module directory must include the following files: - `main.nf`: The module script. - `meta.yml`: The module spec with name, version, description, and license. - `README.md`: Module documentation. -Use `-dry-run` to validate the module structure without uploading: +Provide a `namespace/name` reference (for an already-installed module) or a local directory path containing the module files. + + :::{tip} + Before publishing, verify your module is ready: -```console -$ nextflow module publish myorg/my-module -dry-run -``` + - Run {ref}`module validate ` to check that all required files and fields are present. + - Run {ref}`module spec ` to generate or update `meta.yml` from your `main.nf`. + - Add `-dry-run` to `module publish` to validate the structure without uploading. -Use the {ref}`module spec ` and {ref}`module validate ` commands to prepare and verify a module before publishing. -See {ref}`dev-modules-page` for a guide on creating modules. + See {ref}`dev-modules-page` for a complete guide on creating modules. + ::: See {ref}`cli-module-publish` for the full command reference. ## Registry configuration By default, Nextflow uses the public registry at `https://registry.nextflow.io`. -Configure alternative or additional registries in your Nextflow configuration: +Configure alternative or additional registries using the `registry` scope in your Nextflow configuration. + +### Use a private registry + +Replace the default registry with your organization's private registry: + +```groovy +registry { + url = 'https://registry.myorg.com' + apiKey = '${MYORG_TOKEN}' +} +``` + +Nextflow uses this registry for all module operations (search, install, and publish). + +### Use multiple registries + +Specify multiple registries as a list. +Nextflow queries them in order when searching for or installing a module: ```groovy registry { url = [ - 'https://private.registry.myorg.com', - 'https://registry.nextflow.io' + 'https://registry.myorg.com', + 'https://private.registry.nextflow.io' ] apiKey = '${MYORG_TOKEN}' } ``` -Nextflow queries registries in the order you specify until it finds a module. -The `apiKey` is used only for the primary (first) registry. +In this example, Nextflow searches the private registry first and falls back to the public registry. + +:::{note} +The `apiKey` authenticates with the primary (first) registry. +::: -Specify a registry when publishing: +:::{tip} +Override the target registry with the `-registry` flag: ```console +$ export NXF_REGISTRY_TOKEN= $ nextflow module publish myorg/my-module -registry 'https://private.registry.myorg.com' ``` +::: From af89f8150fa8691ee9acffb7351330fa9c41956c Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Wed, 22 Apr 2026 11:12:12 -0500 Subject: [PATCH 08/14] Update links Signed-off-by: Ben Sherman --- docs/migrations/26-04.md | 2 +- docs/modules/module-registry.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/migrations/26-04.md b/docs/migrations/26-04.md index e184efa224..42afdddb28 100644 --- a/docs/migrations/26-04.md +++ b/docs/migrations/26-04.md @@ -44,7 +44,7 @@ include { BWA_MEM } from 'nf-core/bwa/mem' All [nf-core modules](https://github.com/nf-core/modules) are automatically synced to the registry from GitHub. They are available as `nf-core/` as shown above. ::: -See {ref}`module-registry` and the {ref}`cli-module` command reference for details. +See the {ref}`Modules overview ` and {ref}`cli-module` command reference for details.

Record types

diff --git a/docs/modules/module-registry.md b/docs/modules/module-registry.md index 2adaef75ed..5d3db87923 100644 --- a/docs/modules/module-registry.md +++ b/docs/modules/module-registry.md @@ -65,7 +65,7 @@ To create an API access token: 6. Copy the token and store it somewhere safe. You cannot view it again after this step. -Once you have your token, see [Publishing modules](#publish-modules) for instructions on how to use it. +Once you have your token, see [Publishing modules](#publishing-modules) for instructions on how to use it. ## Viewing modules From ba8002dabf0d920502378a3546f97f1541945261 Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Wed, 22 Apr 2026 11:29:57 -0500 Subject: [PATCH 09/14] Trim examples in command reference Signed-off-by: Ben Sherman --- docs/modules/developing-modules.md | 16 ++-- docs/modules/module-registry.md | 15 ++-- docs/modules/using-modules.md | 16 +++- docs/reference/cli.md | 136 +++-------------------------- 4 files changed, 38 insertions(+), 145 deletions(-) diff --git a/docs/modules/developing-modules.md b/docs/modules/developing-modules.md index dacb1f2af7..6a33f7cb4c 100644 --- a/docs/modules/developing-modules.md +++ b/docs/modules/developing-modules.md @@ -57,9 +57,7 @@ process FASTQC { label 'process_medium' conda 'bioconda::fastqc=0.12.1' - container "${ workflow.containerEngine == 'singularity' - ? 'https://depot.galaxyproject.org/singularity/fastqc:0.12.1--hdfd78af_0' - : 'biocontainers/fastqc:0.12.1--hdfd78af_0' }" + container 'biocontainers/fastqc:0.12.1--hdfd78af_0' input: tuple val(meta), path(reads) @@ -83,15 +81,15 @@ The `meta.yml` file describes the module's metadata, including its name, version The `README.md` file provides documentation for the module. It should describe what the module does, the tools it wraps, and any configuration requirements. +### resources + +Modules can include binary scripts in the `resources/usr/bin/` directory that are locally scoped to the module's processes. See {ref}`module-binaries` for details. + ### templates Include process script {ref}`templates ` alongside a module in the `templates/` directory. See {ref}`module-templates` for details. -### resources - -Modules can include binary scripts in the `resources/usr/bin/` directory that are locally scoped to the module's processes. See {ref}`module-binaries` for details. - (dev-modules-spec)= ## Generating a module spec @@ -102,7 +100,7 @@ Use the `module spec` command to generate or update the `meta.yml` file from the $ nextflow module spec myorg/my-module ``` -Provide metadata fields directly to avoid `TODO` placeholders in the generated file: +Provide required fields directly when generating from scratch to avoid `TODO` placeholders in the generated file: ```console $ nextflow module spec \ @@ -159,7 +157,7 @@ For more thorough testing, create a small wrapper workflow that exercises the mo include { MY_MODULE } from './modules/myorg/my-module' workflow { - input_ch = Channel.fromFilePairs('test-data/*_{1,2}.fastq.gz') + input_ch = channel.fromFilePairs('test-data/*_{1,2}.fastq.gz') MY_MODULE(input_ch) MY_MODULE.out.results.view() } diff --git a/docs/modules/module-registry.md b/docs/modules/module-registry.md index 5d3db87923..27c885a5f6 100644 --- a/docs/modules/module-registry.md +++ b/docs/modules/module-registry.md @@ -113,17 +113,16 @@ The module directory must include the following files: - `meta.yml`: The module spec with name, version, description, and license. - `README.md`: Module documentation. -Provide a `namespace/name` reference (for an already-installed module) or a local directory path containing the module files. +Provide a `namespace/name` reference or a relative path to the module directory. - :::{tip} - Before publishing, verify your module is ready: +:::{tip} +Before publishing, verify your module is ready: - - Run {ref}`module validate ` to check that all required files and fields are present. - - Run {ref}`module spec ` to generate or update `meta.yml` from your `main.nf`. - - Add `-dry-run` to `module publish` to validate the structure without uploading. +- Run {ref}`module spec ` to generate or update `meta.yml` from your `main.nf`. +- Run {ref}`module validate ` to check that all required files and spec fields are present. - See {ref}`dev-modules-page` for a complete guide on creating modules. - ::: +See {ref}`dev-modules-page` for a complete guide on creating modules. +::: See {ref}`cli-module-publish` for the full command reference. diff --git a/docs/modules/using-modules.md b/docs/modules/using-modules.md index 9c31e36556..114ef3d71b 100644 --- a/docs/modules/using-modules.md +++ b/docs/modules/using-modules.md @@ -88,7 +88,7 @@ See {ref}`cli-module-view` for the full command reference. For ad-hoc tasks or testing, run a module directly without creating a wrapper workflow: ```console -$ nextflow module run nf-core/fastqc --meta.id=test_sample --reads sample1_R1.fastq.gz +$ nextflow module run nf-core/fastqc --meta.id test_sample --reads sample1_R1.fastq.gz ``` :::{tip} @@ -100,10 +100,18 @@ It accepts all standard Nextflow run options (`-profile`, `-resume`, etc.): ```console $ nextflow module run nf-core/fastqc \ - --meta.id=test_sample \ + --meta.id test_sample \ --reads sample1_R1.fastq.gz \ - -profile docker \ - -resume + -with-docker +``` + +Run a local module by specifying a path starting with `./` or `../`: + +```console +$ nextflow module run ./modules/local/fastqc/main.nf \ + --meta.id test_sample \ + --reads sample1_R1.fastq.gz \ + -with-docker ``` See {ref}`cli-module-run` for the full command reference. diff --git a/docs/reference/cli.md b/docs/reference/cli.md index 2e16aef73b..bc2c41275e 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -1294,168 +1294,63 @@ The `module` command provides a comprehensive system for managing registry-based **Examples** -Create a module and prompt for details: - -```console -$ nextflow module create -``` - -Create a module with a given name: - -```console -$ nextflow module create myorg/my-module -``` - -Search for alignment-related modules: +Search for modules related to "alignment": ```console $ nextflow module search alignment ``` -Search and limit the number of results: - -```console -$ nextflow module search "quality control" -limit 10 -``` - -Search and output results as JSON: - -```console -$ nextflow module search bwa -output json -``` - View information for a module: ```console $ nextflow module view nf-core/fastqc ``` -View information for a specific module version: - -```console -$ nextflow module view nf-core/fastqc -version 1.0.0 -``` - -View module information as JSON: - -```console -$ nextflow module view nf-core/fastqc -output json -``` - -Install the latest version of a module: +Install a module: ```console $ nextflow module install nf-core/fastqc ``` -Install a specific version of a module: - -```console -$ nextflow module install nf-core/fastqc -version 1.0.0 -``` - -Force a reinstall over local modifications: - -```console -$ nextflow module install nf-core/fastqc -force -``` - -List installed modules in a formatted table: +List installed modules: ```console $ nextflow module list ``` -List installed modules as JSON: - -```console -$ nextflow module list -output json -``` - -Run a remote module: +Run a module: ```console $ nextflow module run nf-core/fastqc \ - --input 'data/*.fastq.gz' -``` - -Run remote module with specific version and run options: - -```console -$ nextflow module run nf-core/fastqc \ - -version 1.0.0 \ --input 'data/*.fastq.gz' \ - -with-conda \ - -resume + -with-docker Run a local module: ```console -$ nextflow module run ./modules/nf-core/fastqc/main.nf \ - --input 'data/*.fastq.gz' +$ nextflow module run ./modules/local/fastqc/main.nf \ + --input 'data/*.fastq.gz' \ + -with-docker ``` -Remove a module completely: +Remove a module: ```console $ nextflow module remove nf-core/fastqc ``` -Remove a module from config but keep local files: - -```console -$ nextflow module remove nf-core/fastqc -keep-files -``` - -Generate a spec for a local module by name: - -```console -$ nextflow module spec nf-core/fastqc -``` - -Generate a spec for a local module by path (namespace required): - -```console -$ nextflow module spec -namespace nf-core ./modules/my-module -``` - -Provide additional fields to avoid TODO placeholders: +Generate a spec for a local module: ```console -$ nextflow module spec \ - -namespace nf-core \ - -version 1.0.0 \ - -description "Quality control of raw sequencing reads" \ - -license MIT \ - -author "@drpatelh" \ - -author "@joseespinosa" \ - ./modules/nf-core/fastqc +$ nextflow module spec -namespace myorg ./modules/myorg/my-module ``` -Print a module spec without saving it: - -```console -$ nextflow module spec -dry-run -namespace nf-core ./modules/my-module -``` - -Validate a module by name: - -```console -$ nextflow module validate myorg/my-module -``` - -Validate a module by path: +Validate a local module: ```console $ nextflow module validate ./modules/myorg/my-module ``` -Validate a module structure without publishing: - -```console -$ nextflow module publish myorg/my-module -dry-run -``` - Publish a module to the Nextflow registry: ```console @@ -1463,13 +1358,6 @@ $ export NXF_REGISTRY_TOKEN= $ nextflow module publish myorg/my-module ``` -Publish a module to a custom registry: - -```console -$ export NXF_REGISTRY_TOKEN= -$ nextflow module publish myorg/my-module -registry 'https://custom.registry.com' -``` - (cli-plugin)= ### `plugin` From 21999e26710f90aac41fcf65ad6b00b8fc1626fb Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Wed, 22 Apr 2026 16:33:28 -0500 Subject: [PATCH 10/14] Incorporate old module docs into new module docs Signed-off-by: Ben Sherman --- docs/guides/gradle-plugin.md | 4 - docs/index.md | 1 - docs/migrations/24-10.md | 2 +- docs/migrations/dsl1.md | 2 +- docs/module.md | 294 ------------------ docs/modules/developing-modules.md | 112 +++++-- docs/modules/modules.md | 37 ++- docs/modules/using-modules.md | 52 ++-- docs/plugins/developing-plugins.md | 4 - docs/process.md | 7 +- docs/reference/cli.md | 6 + docs/reference/feature-flags.md | 2 +- docs/reference/syntax.md | 10 +- docs/script.md | 2 +- docs/strict-syntax.md | 2 +- docs/wave.md | 12 +- docs/workflow.md | 15 +- .../nextflow/script/dsl/FeatureFlagDsl.java | 2 +- 18 files changed, 180 insertions(+), 386 deletions(-) delete mode 100644 docs/module.md diff --git a/docs/guides/gradle-plugin.md b/docs/guides/gradle-plugin.md index 23e9052cf6..75de793466 100644 --- a/docs/guides/gradle-plugin.md +++ b/docs/guides/gradle-plugin.md @@ -6,10 +6,6 @@ Nextflow provides a new `plugin create` command that simplifies the creation of The [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle) configures default dependencies needed for Nextflow integration and defines Gradle tasks for building, testing, and publishing Nextflow plugins. The Gradle plugin is versioned and published to the [Gradle Plugin Portal](https://plugins.gradle.org/), allowing developers to manage it like any other dependency. As the plugin ecosystem evolves, the Gradle plugin will enable easier maintenance and adoption of improvements. This page introduces [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle) and how to use it. -:::{note} -The Nextflow Gradle plugin and plugin registry are currently available as a public preview. See the {ref}`Migrating to the Nextflow plugin registry ` for more information. -::: - (gradle-plugin-create)= ## Creating a plugin diff --git a/docs/index.md b/docs/index.md index 04ab8a0f73..be9e7bc359 100644 --- a/docs/index.md +++ b/docs/index.md @@ -80,7 +80,6 @@ process process-typed workflow workflow-typed -module notifications secrets sharing diff --git a/docs/migrations/24-10.md b/docs/migrations/24-10.md index 9d6fe18c5f..2152c0e7db 100644 --- a/docs/migrations/24-10.md +++ b/docs/migrations/24-10.md @@ -51,7 +51,7 @@ The `task` variable in the process definition has two new properties, `task.prev ## Deprecations -- The use of `addParams` and `params` clauses in include declarations is deprecated. See {ref}`module-params` for details. +- The use of `addParams` and `params` clauses in include declarations is deprecated. See {ref}`strict-syntax-page` for details. ## Miscellaneous diff --git a/docs/migrations/dsl1.md b/docs/migrations/dsl1.md index a3b532d8f6..0a1094ab69 100644 --- a/docs/migrations/dsl1.md +++ b/docs/migrations/dsl1.md @@ -100,7 +100,7 @@ Similarly, in DSL2, process outputs can be consumed by multiple consumers automa In DSL1, the entire Nextflow pipeline must be defined in a single file. For example, `main.nf`. This restriction becomes cumbersome as a pipeline grows and hinders the sharing and reuse of pipeline components. -DSL2 introduces the concept of "module scripts" (or "modules" for short), which are Nextflow scripts that can be "included" by other scripts. While modules are not essential to migrating to DSL2, nor are they mandatory in DSL2, modules can help you organize a large pipeline into multiple smaller files and take advantage of modules created by others. See {ref}`module-page` to learn more about modules. +DSL2 introduces the concept of "module scripts" (or "modules" for short), which are Nextflow scripts that can be "included" by other scripts. While modules are not essential to migrating to DSL2, nor are they mandatory in DSL2, modules can help you organize a large pipeline into multiple smaller files and take advantage of modules created by others. See {ref}`modules-page` to learn more about modules. :::{note} DSL2 scripts cannot exceed 64 KB in size. Split large DSL1 scripts into modules to avoid this limit. diff --git a/docs/module.md b/docs/module.md deleted file mode 100644 index d2b010e7bf..0000000000 --- a/docs/module.md +++ /dev/null @@ -1,294 +0,0 @@ -(module-page)= - -# Modules - -Nextflow scripts can include **definitions** (workflows, processes, and functions) from other scripts. When a script is included in this way, it is referred to as a **module**. Modules can be included by other modules or pipeline scripts and can even be shared across workflows. - -:::{note} -Modules were introduced in DSL2. If you are still using DSL1, see the {ref}`dsl1-page` page to learn how to migrate your Nextflow pipelines to DSL2. -::: - -## Module inclusion - -You can include any definition from a module into a Nextflow script using the `include` keyword. - -For example: - -```nextflow -include { cat } from './some/module' - -workflow { - data = channel.fromPath('/some/data/*.txt') - cat(data) -} -``` - -The above snippet imports a process named `cat`, defined in the module, into the main execution context. This way, `cat` can be invoked in the `workflow` scope. - -Nextflow implicitly looks for the script file `./some/module.nf`, resolving the path against the *including* script location. - -Module includes are subject to the following rules: - -- Relative paths must begin with the `./` prefix. -- Include statements are not allowed from within a workflow. They must occur at the script level. - -(module-directory)= - -## Module directory - -:::{versionadded} 22.10.0 -::: - -A module can be defined as a directory with the same name as the module and with a script named `main.nf`. For example: - -``` -some -└── module - └── main.nf -``` - -When defined as a directory, the module must be included by specifying the module directory path: - -```nextflow -include { hello } from './some/module' -``` - -Module directories allow the use of module scoped binary scripts. See [Module binaries] for details. - -## Multiple inclusions - -A Nextflow script can include any number of modules, and an `include` statement can import any number of definitions from a module. Multiple definitions can be included from the same module by using the syntax shown below: - -```nextflow -include { cat; wc } from './some/module' - -workflow { - data = channel.fromPath('/some/data/*.txt') - cat(data) - wc(data) -} -``` - -(module-aliases)= - -## Module aliases - -When including definition from a module, it's possible to specify an *alias* with the `as` keyword. Aliasing allows you to avoid module name clashes, by assigning them different names in the including context. For example: - -```nextflow -include { cat as cat_alpha } from './some/module' -include { cat as cat_beta } from './other/module' - -workflow { - cat_alpha(some_data) - cat_beta(other_data) -} -``` - -You can also include the same definition multiple times under different names: - -```nextflow -include { cat as cat_alpha; cat as cat_beta } from './some/module' - -workflow { - cat_alpha(some_data) - cat_beta(other_data) -} -``` - -(module-params)= - -## Module parameters - -:::{deprecated} 24.07.0-edge -As a best practice, parameters should be used in the entry workflow and passed to workflows, processes, and functions as explicit inputs. -::: - -A module can define parameters using the same syntax as a Nextflow workflow script: - -```nextflow -params.message = 'Hello' -params.target = 'world!' - -def sayHello() { - println "$params.message $params.target" -} -``` - -When including a module, the module will first use parameters from the including context. For example: - -```nextflow -params.message = 'Hola' -params.target = 'Mundo' - -include { sayHello } from './some/module' - -workflow { - sayHello() -} -``` - -The above snippet prints: - -``` -Hola Mundo -``` - -:::{note} -The module inherits the parameters defined *before* the `include` statement, therefore any parameters set afterwards will not be used by the module. -::: - -:::{tip} -It is best to define all pipeline parameters *before* any `include` statements. -::: - -The `addParams` option can be used to pass parameters to the module without adding them to the including scope. - -```nextflow -params.message = 'Hola' -params.target = 'Mundo' - -include { sayHello } from './some/module' addParams(message: 'Ciao') - -workflow { - sayHello() -} -``` - -The above snippet prints: - -``` -Ciao Mundo -``` - -Alternatively, the `params` option can be used to pass parameters to module without adding them to the including scope, *and* without inheriting any parameters from the including scope. - -```nextflow -params.message = 'Hola' -params.target = 'Mundo' - -include { sayHello } from './some/module' params(message: 'Ciao') - -workflow { - sayHello() -} -``` - -The above snippet prints: - -``` -Ciao world! -``` - -(module-templates)= - -## Module templates - -Process script {ref}`templates ` can be included alongside a module in the `templates` directory. - -For example, suppose we have a project L with a module that defines two processes, P1 and P2, both of which use templates. The template files can be made available in the local `templates` directory: - -``` -Project L -|── myModules.nf -└── templates - |── P1-template.sh - └── P2-template.sh -``` - -Then, we have a second project A with a workflow that includes P1 and P2: - -``` -Pipeline A -└── main.nf -``` - -Finally, we have a third project B with a workflow that also includes P1 and P2: - -``` -Pipeline B -└── main.nf -``` - -With the possibility to keep the template files inside the project L, A and B can use the modules defined in L without any changes. A future project C would do the same, just cloning L (if not available on the system) and including its module. - -Beside promoting the sharing of modules across pipelines, there are several advantages to keeping the module template under the script path: - -1. Modules are self-contained -2. Modules can be tested independently from the pipeline(s) that import them -3. Modules can be made into libraries - -Having multiple template locations enables a structured project organization. If a project has several modules, and they all use templates, the project could group module scripts and their templates as needed. For example: - -``` -baseDir -|── main.nf -|── Phase0-Modules - |── mymodules1.nf - |── mymodules2.nf - └── templates - |── P1-template.sh - |── P2-template.sh -|── Phase1-Modules - |── mymodules3.nf - |── mymodules4.nf - └── templates - |── P3-template.sh - └── P4-template.sh -└── Phase2-Modules - |── mymodules5.nf - |── mymodules6.nf - └── templates - |── P5-template.sh - |── P6-template.sh - └── P7-template.sh -``` - -(module-binaries)= - -## Module binaries - -:::{versionadded} 22.10.0 -::: - -Modules can define binary scripts that are locally scoped to the processes defined by the tasks. - -To enable this feature, set the following flag in your pipeline script or configuration file: - -```nextflow -nextflow.enable.moduleBinaries = true -``` - -The binary scripts must be placed in the module directory named `/resources/usr/bin`: - -``` - -|── main.nf -└── resources - └── usr - └── bin - |── your-module-script1.sh - └── another-module-script2.py -``` - -Those scripts will be made accessible like any other command in the task environment, provided they have been granted the Linux execute permissions. - -:::{note} -This feature requires the use of a local or shared file system for the pipeline work directory, or {ref}`wave-page` when using cloud-based executors. -::: - -:::{note} -When {ref}`wave-page` is enabled, the contents of the module `resources/` directory are automatically included in the provisioned container. Wave treats the `resources/` directory as the container root. For example, `resources/some/data/file.txt` becomes `/some/data/file.txt` in the container. -::: - -## Sharing modules - -Nextflow makes modules easy to share and re-use across different pipelines, helping eliminate duplicate work and spread improvements throughout the community. -There are several ways to share modules: - -- Use the {ref}`Nextflow module registry ` (recommended) -- Copy the module files into your pipeline repository -- Use [Git submodules](https://git-scm.com/book/en/v2/Git-Tools-Submodules) to fetch modules from other Git repositories without maintaining a separate copy -- Use the [nf-core](https://nf-co.re/tools#modules) CLI to install and update modules with a standard approach used by the nf-core community - -See {ref}`modules-page` for an overview of the module registry, {ref}`using-modules-page` to learn how to install and manage registry modules, and {ref}`dev-modules-page` to learn how to create and publish your own modules. diff --git a/docs/modules/developing-modules.md b/docs/modules/developing-modules.md index 6a33f7cb4c..134dcce0a0 100644 --- a/docs/modules/developing-modules.md +++ b/docs/modules/developing-modules.md @@ -2,8 +2,7 @@ # Developing modules -Learn how to create modules for sharing through the Nextflow module registry. -For information about modules syntax, see {ref}`module-page`. +Learn how to create modules and share them through the Nextflow module registry. (dev-modules-creating)= @@ -43,13 +42,15 @@ modules/ ├── README.md # Documentation (required for publishing) ├── main.nf # Module script (required) ├── meta.yml # Module spec (required for publishing) - ├── resources/ # Optional: module binaries and resources - └── templates/ # Optional: process templates + ├── resources/ # Optional: module resources + └── templates/ # Optional: process script templates ``` +Local modules that are not intended for publishing do not need to follow this structure, although it is recommended as a best practice. + ### main.nf -The `main.nf` file contains the process (or workflow/function) definition. For example, a simple module wrapping FastQC: +The `main.nf` file contains the process definition. For example, a simple module wrapping FastQC: ```nextflow process FASTQC { @@ -73,22 +74,91 @@ process FASTQC { } ``` +Registry modules are subject to the following constraints: + +- The module must contain a single script named `main.nf` +- The module must define a single process +- The module must not define any named workflows +- The module may define an entry workflow to override the default behavior of `module run`. +- The module may define any number of functions + +Local modules can define any number of processes, workflows, and functions. As a best practice, each process and named workflow should be defined in its own script. + ### meta.yml -The `meta.yml` file describes the module's metadata, including its name, version, description, authors, and input/output specifications. Publishing requires this file, and the registry uses it to display module information and generate usage templates. +The `meta.yml` file contains the module's metadata, including its name, version, description, authors, and input/output specifications. The registry uses this file to display module information and generate usage templates. ### README.md The `README.md` file provides documentation for the module. It should describe what the module does, the tools it wraps, and any configuration requirements. +(module-resources)= + ### resources -Modules can include binary scripts in the `resources/usr/bin/` directory that are locally scoped to the module's processes. See {ref}`module-binaries` for details. +Modules can include resource files in the `resources/` directory. + +When running the module with {ref}`wave-page`, the contents of `resources/` are mounted into the root directory of the task container. + +For example, given a module with the following structure: + +``` +my-module/ +├── main.nf +└── resources/ + ├── data/ + | └── file.txt + └── usr/ + └── bin/ + └── hello.sh +``` + +The process script can use these files as follows: + +```nextflow +process hello { + container 'quay.io/nextflow/bash' + + script: + """ + cat /data/file.txt + hello.sh + """ +} +``` + +Module resources can be used without Wave or containerization, with the following limitations: + +- The `nextflow.enable.moduleBinaries` feature flag must be enabled in the pipeline script. + +- The pipeline work directory must be in a local or shared file system. Remote object storage is not supported without Wave. + +- Only executable scripts in `resources/usr/bin/` are made accessible to the process script. ### templates -Include process script {ref}`templates ` alongside a module in the `templates/` directory. -See {ref}`module-templates` for details. +Modules can include process script {ref}`templates ` in the `templates/` directory. + +For example, given a module with the following structure: + +``` +my-module/ +|── main.nf +└── templates/ + └── hello.sh +``` + +The module's process can use the script template as follows: + +```nextflow +process hello { + input: + val STR + + script: + template 'hello.sh' +} +``` (dev-modules-spec)= @@ -100,7 +170,13 @@ Use the `module spec` command to generate or update the `meta.yml` file from the $ nextflow module spec myorg/my-module ``` -Provide required fields directly when generating from scratch to avoid `TODO` placeholders in the generated file: +Use `-dry-run` to preview the generated spec without writing to disk: + +```console +$ nextflow module spec -dry-run myorg/my-module +``` + +When generating the module spec for the first time, provide required fields directly to avoid `TODO` placeholders in the generated file: ```console $ nextflow module spec \ @@ -112,13 +188,7 @@ $ nextflow module spec \ ./modules/myorg/my-module ``` -Use `-dry-run` to preview the generated spec without writing to disk: - -```console -$ nextflow module spec -dry-run myorg/my-module -``` - -If a `meta.yml` already exists, the command incorporates its content into the new file. +When updating an existing module spec, it is incorporated into the new file. See {ref}`cli-module-spec` for the full command reference. @@ -149,7 +219,7 @@ Before publishing, test your module by running it directly: $ nextflow module run myorg/my-module --input 'test-data/*.fastq.gz' ``` -The command executes the module as a standalone workflow, allowing you to verify that inputs are correctly declared, the process runs successfully, and outputs appear as expected. +The command executes the module as a standalone run, allowing you to verify that inputs are correctly declared, the process runs successfully, and the correct outputs are produced. For more thorough testing, create a small wrapper workflow that exercises the module: @@ -157,8 +227,8 @@ For more thorough testing, create a small wrapper workflow that exercises the mo include { MY_MODULE } from './modules/myorg/my-module' workflow { - input_ch = channel.fromFilePairs('test-data/*_{1,2}.fastq.gz') - MY_MODULE(input_ch) - MY_MODULE.out.results.view() + input_ch = channel.fromPath('test-data/*.fastq.gz') + results_ch = MY_MODULE(input_ch) + results_ch.view() } ``` diff --git a/docs/modules/modules.md b/docs/modules/modules.md index aa9997d116..f756c6d653 100644 --- a/docs/modules/modules.md +++ b/docs/modules/modules.md @@ -2,43 +2,54 @@ # Overview -Nextflow modules are reusable units of pipeline logic (i.e., processes, workflows, and functions) that you can share across projects. -By packaging common tasks as modules, you avoid duplicating code and benefit from community improvements. +Nextflow scripts can include **definitions** (workflows, processes, and functions) from other scripts. +When a script is included in this way, it is referred to as a **module**. +Modules can be re-used within a pipeline and can be shared across projects. +By packaging definitions as modules, you avoid duplicating code and benefit from community improvements. There are two ways to use modules in Nextflow: - [Local modules](#local-modules) - [Registry modules](#registry-modules) +See {ref}`using-modules-page` to learn how to install and use modules in a project. +See {ref}`dev-modules-page` to learn how to create and publish your own modules. +See {ref}`module-registry-page` to learn how to use the Nextflow module registry. + ## Local modules -Local modules are Nextflow scripts stored directly in your project. You include definitions from local modules using the `include` keyword with a relative path: +Local modules are stored and maintained directly in your project. You include definitions from local modules using the `include` keyword with a relative path: ```nextflow -include { FASTQC } from './modules/fastqc' +include { CAT } from './modules/cat' + +workflow { + data = channel.fromPath('data/*.txt') + CAT(data) +} ``` -Local modules are well suited for project-specific logic that is not intended for reuse or sharing. See {ref}`module-page` for details on module inclusion syntax, aliases, templates, and binaries. +The above snippet imports a process named `CAT` from the *included module* into the *including script*. The include source `./modules/cat` refers to the script `./modules/cat.nf` relative to the including script path. + +Local modules are well suited for project-specific components that are not intended for sharing. ## Registry modules :::{versionadded} 26.04.0 ::: -The [Nextflow module registry](https://registry.nextflow.io) hosts registry modules in a centralized repository. -You manage them with the `nextflow module` command. -Registry modules follow a standard structure with metadata (`meta.yml`), documentation (`README.md`), and a module script (`main.nf`) for version management, integrity checking, and discoverability. +Registry modules are sourced from the [Nextflow registry](https://registry.nextflow.io) or a compatible module registry. +Like local modules, registry modules are stored in your project, but they are installed and managed using the `nextflow module` command. Key features of registry modules: - **Discoverability**: Search for modules by keyword or name and browse available versions. - **Version management**: Pin specific versions or use the latest release, with automatic integrity checking via checksums. - **Direct execution**: Run modules as standalone workflows for ad-hoc tasks or testing without writing a wrapper script. -- **Standard structure**: Each module includes a script (`main.nf`), metadata (`meta.yml`), and documentation (`README.md`), enabling consistent tooling and automation. +- **Standard structure**: Each module includes a script (`main.nf`), spec (`meta.yml`), and documentation (`README.md`), providing a consistent structure for tooling and automation. :::{note} Modules from the [nf-core](https://nf-co.re/) community are automatically mirrored to the Nextflow module registry under the `nf-core` namespace. -You can install and use them directly without any additional configuration. ::: Install registry modules into your project and include them by name: @@ -50,9 +61,3 @@ $ nextflow module install nf-core/fastqc ```nextflow include { FASTQC } from 'nf-core/fastqc' ``` - -For more information about registry modules, see: - -- {ref}`using-modules-page` for details on discovering, installing, and managing registry modules. -- {ref}`dev-modules-page` for details on creating and publishing your own modules. -- {ref}`module-registry-page` for details on namespaces, access tokens, and registry configuration. diff --git a/docs/modules/using-modules.md b/docs/modules/using-modules.md index 114ef3d71b..f32ecaa287 100644 --- a/docs/modules/using-modules.md +++ b/docs/modules/using-modules.md @@ -6,8 +6,7 @@ ::: The Nextflow module system allows you to discover, install, and manage reusable modules from centralized registries. -This page describes how to use registry modules in your pipelines. -For information about modules syntax, see {ref}`module-page`. +This page describes how to use modules in your pipelines. ## Discovering modules @@ -39,17 +38,6 @@ Instead, they use the format `0.0.0-`, where the suffix is a short portion Nextflow stores installed modules in the `modules/` directory and creates a `.module-info` file alongside the module to record installation metadata such as the module checksum and registry URL. -Once installed, include a module by name rather than a relative path: - -```nextflow -include { FASTQC } from 'nf-core/fastqc' - -workflow { - reads = Channel.fromFilePairs('data/*_{1,2}.fastq.gz') - FASTQC(reads) -} -``` - :::{tip} Commit the `modules/` directory to your Git repository to ensure reproducibility. ::: @@ -83,6 +71,26 @@ Use `-output json` for machine-readable output. See {ref}`cli-module-view` for the full command reference. +## Including modules + +Modules installed from a registry can be included by name: + +```nextflow +include { FASTQC } from 'nf-core/fastqc' + +workflow { + reads = channel.fromPath('data/*.fastq') + reads = reads.map { fastq -> tuple([id: fastq.baseName], fastq) } + FASTQC(reads) +} +``` + +Local modules must be included by relative path: + +```nextflow +include { FASTQC } from './modules/local/fastqc' +``` + ## Running modules directly For ad-hoc tasks or testing, run a module directly without creating a wrapper workflow: @@ -114,6 +122,10 @@ $ nextflow module run ./modules/local/fastqc/main.nf \ -with-docker ``` +:::{note} +The local module should define a single process, or else the command will fail. +::: + See {ref}`cli-module-run` for the full command reference. ## Updating modules @@ -124,20 +136,14 @@ To update a module to a newer version, reinstall it with the desired version: $ nextflow module install nf-core/fastqc -version 0.0.0-c9h0bv4 ``` -If you have local modifications, Nextflow warns you and prevents the update. -Use the `-force` flag to override: +Nextflow automatically verifies module integrity using a checksum stored in the `.module-info` file. +If the module has local modifications, it will not be updated. +Use the `-force` flag to overwrite local changes: ```console $ nextflow module install nf-core/fastqc -version 0.0.0-c9h0bv4 -force ``` -## Checksum verification - -Nextflow automatically verifies module integrity using checksums stored in the `.module-info` file. -When you modify a module locally, Nextflow detects the change and prevents accidental overwrites during reinstallation. - -Module integrity ensures that modules remain consistent with their registry versions unless you explicitly choose to override them. - ## Removing modules Use the `module remove` command to uninstall a module from your project: @@ -150,6 +156,6 @@ By default, Nextflow removes both the module files and the `.module-info` file. Use flags to control this behavior: - `-keep-files`: Remove the `.module-info` file but keep the module files in the `modules/` directory. -- `-force`: Force removal even if the module has no `.module-info` file or has local modifications. +- `-force`: Remove the module directory even if it has no `.module-info` file or has local modifications. See {ref}`cli-module-remove` for the full command reference. diff --git a/docs/plugins/developing-plugins.md b/docs/plugins/developing-plugins.md index 1c01085e4a..7a636d15ac 100644 --- a/docs/plugins/developing-plugins.md +++ b/docs/plugins/developing-plugins.md @@ -12,10 +12,6 @@ The [Nextflow plugin template](https://github.com/nextflow-io/nf-plugin-template You can use the `nextflow plugin create` sub-command to create plugins using the plugin template. See {ref}`gradle-plugin-create` for more information. -:::{note} -The Nextflow Gradle plugin is currently available as a public preview. See {ref}`Migrating to the Nextflow plugin registry ` for more information. -::: - (dev-plugins-structure)= ### Structure diff --git a/docs/process.md b/docs/process.md index ffb623c3d5..cedbc33143 100644 --- a/docs/process.md +++ b/docs/process.md @@ -166,12 +166,15 @@ process hello { input: val STR + output: + stdout + script: template 'hello.sh' } workflow { - channel.of('this', 'that') | hello + hello('Hello!').view() } ``` @@ -189,7 +192,7 @@ echo "process completed" Variables prefixed with the dollar character (`$`) are interpreted as Nextflow variables when the template script is executed by Nextflow and Bash variables when executed directly. For example, the above script can be executed from the command line by providing each input as an environment variable: ```bash -STR='Hello!' bash templates/my_script.sh +STR='Hello!' bash templates/hello.sh ``` The following caveats should be considered: diff --git a/docs/reference/cli.md b/docs/reference/cli.md index bc2c41275e..70480dbffa 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -1339,6 +1339,12 @@ Remove a module: $ nextflow module remove nf-core/fastqc ``` +Create a module with a given name: + +```console +$ nextflow module create myorg/my-module +``` + Generate a spec for a local module: ```console diff --git a/docs/reference/feature-flags.md b/docs/reference/feature-flags.md index 594066aa50..2eea415904 100644 --- a/docs/reference/feature-flags.md +++ b/docs/reference/feature-flags.md @@ -20,7 +20,7 @@ Feature flags with the `nextflow.preview` prefix can cause pipelines run with ne : Defines the DSL version to use (`1` or `2`). `nextflow.enable.moduleBinaries` -: When `true`, enables the use of modules with binary scripts. See {ref}`module-binaries` for more information. +: When `true`, enables the use of module-scoped executable scripts via {ref}`module resources `. `nextflow.enable.strict` : :::{deprecated} 26.04.0 diff --git a/docs/reference/syntax.md b/docs/reference/syntax.md index 64985f0a9a..99682ac7df 100644 --- a/docs/reference/syntax.md +++ b/docs/reference/syntax.md @@ -73,6 +73,8 @@ A feature flag declaration is an assignment. The target should be a valid {ref}` nextflow.preview.recursion = true ``` +(syntax-include)= + ### Include An include declaration consists of an *include source* and one or more *include clauses*: @@ -1053,6 +1055,8 @@ Compound expressions are evaluated in the following order: The following legacy features were excluded from this page because they are deprecated: -- The `addParams` and `params` clauses of include declarations. See {ref}`module-params` for more information. -- The `when:` section of a process definition. See {ref}`process-when` for more information. -- The `shell:` section of a process definition. See {ref}`process-shell` for more information. +- The `addParams` and `params` clauses of include declarations. +- The `when:` section of a process definition. +- The `shell:` section of a process definition. + +See {ref}`strict-syntax-page` for more information. diff --git a/docs/script.md b/docs/script.md index d4c80ee07b..d25141749d 100644 --- a/docs/script.md +++ b/docs/script.md @@ -561,4 +561,4 @@ workflow { } ``` -See {ref}`workflow-page`, {ref}`process-page`, and {ref}`module-page` for more information about how to use these features in your Nextflow scripts. +See {ref}`workflow-page`, {ref}`process-page`, and {ref}`modules-page` for more information about how to use these features in your Nextflow scripts. diff --git a/docs/strict-syntax.md b/docs/strict-syntax.md index 00037698e8..8e6d949847 100644 --- a/docs/strict-syntax.md +++ b/docs/strict-syntax.md @@ -260,7 +260,7 @@ The following patterns are still supported but have been restricted. That is, so ### Include declarations -In Nextflow DSL2, include declarations can have an `addParams` or `params` clause as described in {ref}`module-params`: +In Nextflow DSL2, include declarations can have an `addParams` or `params` clause: ```nextflow params.message = 'Hola' diff --git a/docs/wave.md b/docs/wave.md index 44e882d307..ea733ed61c 100644 --- a/docs/wave.md +++ b/docs/wave.md @@ -48,22 +48,16 @@ tower { Wave can build and provision container images on-demand for your Nextflow pipelines. -To enable this feature, add the Dockerfile of the container to be built in the {ref}`module directory ` where the pipeline process is defined. When Wave is enabled, it automatically uses the Dockerfile to build the required container, upload to the registry, and it uses the container to carry out the tasks defined in the module. +To enable this feature, add the Dockerfile of the container image to the module directory where the process is defined. When Wave is enabled, it automatically uses the Dockerfile to build the required container, upload to the registry, and it uses the container to carry out the tasks defined in the module. -:::{tip} -Make sure the process does not declare a `container` directive, otherwise it will take precedence over the Dockerfile definition. -::: - -If a process uses a `container` directive and you still want to build the container using the Dockerfile provided in the module directory, add the following setting to the pipeline config file: +If a process has a `container` directive, it will take precedence over the Dockerfile. Use the `wave.strategy` config option to prioritize the Dockerfile over the `container` directive: ```groovy wave.strategy = ['dockerfile','container'] ``` -This setting instructs Wave to prioritize the module Dockerfile over process `container` directives. - :::{note} -When building containers, Wave does not support `ADD`, `COPY`, or any other Dockerfile commands that access files in the host file system. To include custom scripts or files, use the module `resources/` directory instead. Wave automatically includes its contents in the container build context. See {ref}`module-binaries` for more details. +When building container images, Wave does not support `ADD`, `COPY`, or any other Dockerfile commands that access files in the host file system. To include custom scripts or files, use the module `resources/` directory instead. Wave automatically includes its contents in the container build context. See {ref}`module-resources` for more details. ::: ### Build Conda based containers diff --git a/docs/workflow.md b/docs/workflow.md index 6e1b2489c1..18aa2807b0 100644 --- a/docs/workflow.md +++ b/docs/workflow.md @@ -540,11 +540,20 @@ workflow { } ``` -Processes and workflows have a few extra rules for how they can be called: +Processes and workflows can only be called by workflows. A given process or workflow can only be called once in a given workflow. To use a process or workflow multiple times in the same workflow, include it from another script with multiple aliases: -- Processes and workflows can only be called by workflows +```nextflow +// workflow `hello_bye` is defined in `./modules/hello_bye/main.nf` +include { hello_bye as hello_bye1 } from './modules/hello_bye' +include { hello_bye as hello_bye2 } from './modules/hello_bye' -- A given process or workflow can only be called once in a given workflow. To use a process or workflow multiple times in the same workflow, use {ref}`module-aliases`. +workflow { + data1 = channel.fromPath('data1/*.txt') + data2 = channel.fromPath('data2/*.txt') + hello_bye1(data1) + hello_bye2(data2) +} +``` The "return value" of a process or workflow call is the process outputs or workflow emits, respectively. The return value can be assigned to a variable or passed into another call: diff --git a/modules/nf-lang/src/main/java/nextflow/script/dsl/FeatureFlagDsl.java b/modules/nf-lang/src/main/java/nextflow/script/dsl/FeatureFlagDsl.java index ce9f6f27e3..1e9ee17738 100644 --- a/modules/nf-lang/src/main/java/nextflow/script/dsl/FeatureFlagDsl.java +++ b/modules/nf-lang/src/main/java/nextflow/script/dsl/FeatureFlagDsl.java @@ -33,7 +33,7 @@ Defines the DSL version (`1` or `2`). @FeatureFlag("nextflow.enable.moduleBinaries") @Description(""" - When `true`, enables the use of modules with executable scripts i.e. [module binaries](https://nextflow.io/docs/latest/module.html#module-binaries). + When `true`, enables the use of module-scoped executable scripts via [module resources](https://nextflow.io/docs/latest/module.html#module-resources). """) public boolean moduleBinaries; From fb43f80861e6a3e96dd81de1dab987a53fa36cd7 Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Wed, 22 Apr 2026 17:05:48 -0500 Subject: [PATCH 11/14] Remove extraneous content in CLI usage docs Signed-off-by: Ben Sherman --- docs/cli.md | 139 +++------------------------------------------------- 1 file changed, 8 insertions(+), 131 deletions(-) diff --git a/docs/cli.md b/docs/cli.md index 5c6362d3bf..29c67ecf82 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -250,149 +250,26 @@ $ nextflow drop nextflow-io/hello See {ref}`cli-drop` for more information. -### Secret management - -The `secrets` command manages secure pipeline secrets. - -Use this to store credentials securely, reference them in pipelines without exposing values, and manage sensitive data centrally across your organization. - -```console -$ nextflow secrets list -$ nextflow secrets set AWS_ACCESS_KEY_ID -$ nextflow secrets delete AWS_ACCESS_KEY_ID -``` - -See {ref}`cli-secrets` for more information. - ## Module management :::{versionadded} 26.04.0 ::: -The `module` command enables working with reusable, registry-based modules. The Nextflow module system allows you to install, run, search, and publish standardized modules from registries, eliminating duplicate work and sharing improvements with the community. - -Use these commands to discover modules in registries, install them into your project, run them directly without creating a workflow, and publish your own modules for others to use. - -### Searching for modules - -The `module search` command queries the module registry to discover available modules by keyword or name. - -Use this to find modules for specific tasks, explore available tools, or discover community modules. - -```console -$ nextflow module search alignment -$ nextflow module search "quality control" -limit 10 -$ nextflow module search bwa -output json -``` - -Results include module names, versions, descriptions, and download statistics. Use `-limit` to control the number of results and `-output json` for JSON-formatted output. - -See {ref}`cli-module-search` for more information. - -### Installing modules - -The `module install` command downloads modules from a registry and makes them available in your workflow. Modules are stored locally in the `modules/` directory. An additional `.module-info` file is created during to store installation information such as the module checksum at installation and the registry URL. - -Use this to add reusable modules to your pipeline, manage module versions, or update modules to newer versions. - -```console -$ nextflow module install nf-core/fastqc -$ nextflow module install nf-core/fastqc -version 1.0.0 -``` - -The installed module will be available in `modules/nf-core/fastqc`. - -Use the `-force` flag to reinstall a module even if local modifications exist. - -See {ref}`cli-module-install` for more information. - -### Listing modules - -The `module list` command displays all modules currently installed in your project, showing their versions and integrity status. - -Use this to review installed modules, check module versions, or detect local modifications. - -```console -$ nextflow module list -$ nextflow module list -output json -``` - -The output shows each module's name, installed version, and whether it has been modified locally. Use `-o json` for JSON-formatted output. +The `module` command allows you to discover, install, and run modules through a centralized module registry. See {ref}`using-modules-page` and the {ref}`cli-module` command reference for more information. -See {ref}`cli-module-list` for more information. +## Secret management -### Viewing module information - -The `module view` command displays detailed metadata and usage information for a specific module from the registry. - -Use this to understand module requirements, view input/output specifications, see available tools, or generate usage templates before installing or running a module. - -```console -$ nextflow module view nf-core/fastqc -$ nextflow module view nf-core/fastqc -version 1.0.0 -$ nextflow module view nf-core/fastqc -output json -``` - -The output includes the module's version, description, authors, keywords, tools, input/output channels, and a generated usage template showing how to run the module. Use `-json` for machine-readable output suitable for programmatic access. - -See {ref}`cli-module-view` for more information. - -### Running modules directly - -The `module run` command executes a module directly from the registry without requiring a wrapper workflow. This provides immediate access to module functionality for ad-hoc tasks or testing. - -Use this to quickly run a module, test module functionality, or execute one-off data processing tasks. - -```console -$ nextflow module run nf-core/fastqc --input 'data/*.fastq.gz' -$ nextflow module run nf-core/fastqc --input 'data/*.fastq.gz' -version 1.0.0 -``` - -The command accepts all standard Nextflow execution options (`-profile`, `-resume`, etc.): - -```console -$ nextflow module run nf-core/salmon \ - --reads reads.fq \ - --index salmon_index \ - -profile docker \ - -resume -``` - -Process inputs can be specified like params on the command line. For example, `--reads reads.fq` corresponds to the `reads` input in the `nf-core/salmon` module. Run `nextflow module view nf-core/salmon` to see the available params for the module. - -See {ref}`cli-module-run` for more information. - -### Removing modules - -The `module remove` command deletes modules from your project, removing local files and configuration entries. - -Use this to clean up unused modules, free disk space, or remove deprecated modules from your pipeline. - -```console -$ nextflow module remove nf-core/fastqc -$ nextflow module remove nf-core/fastqc -keep-files -``` - -By default, both local files and configuration entries are removed. Use `-keep-files` to remove the configuration entry and `.module-info` while keeping local files. - -See {ref}`cli-module-remove` for more information. - -### Publishing modules - -The `module publish` command uploads modules to a registry, making them available for others to install and use. +The `secrets` command manages secure pipeline secrets. -Use this to share your modules with the community, contribute to module libraries, or distribute modules within your organization. +Use this to store credentials securely, reference them in pipelines without exposing values, and manage sensitive data centrally across your organization. ```console -$ nextflow module publish myorg/my-module -$ nextflow module publish myorg/my-module -dry-run +$ nextflow secrets list +$ nextflow secrets set AWS_ACCESS_KEY_ID +$ nextflow secrets delete AWS_ACCESS_KEY_ID ``` -Publishing requires authentication via the `NXF_REGISTRY_TOKEN` environment variable or the `registry.apiKey` config option. The module must include `main.nf`, `meta.yml`, and `README.md` files. - -Use the {ref}`cli-module-spec` and {ref}`cli-module-validate` commands to prepare and validate a module before publishing. - -See {ref}`cli-module-publish` for more information. +See {ref}`cli-secrets` for more information. ## Configuration and validation From f38692786e65619d9d637206ce3f4df5fd2c20e5 Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Wed, 22 Apr 2026 17:05:59 -0500 Subject: [PATCH 12/14] Add redirect for old module page Signed-off-by: Ben Sherman --- docs/conf.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/conf.py b/docs/conf.py index 47d5739387..9420b4d7ed 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -58,7 +58,8 @@ 'flux.md': 'tutorials/flux.md', 'developer/plugins.md': 'plugins/developing-plugins.md', 'plugins.md': 'plugins/plugins.md', - 'channel.md': 'workflow.md' + 'channel.md': 'workflow.md', + 'module.md': 'modules/modules.md' } # Add any paths that contain templates here, relative to this directory. From 03862ede3631d073631a605de1b68accc1f23753 Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Wed, 22 Apr 2026 17:06:26 -0500 Subject: [PATCH 13/14] Fix links to module subcommands Signed-off-by: Ben Sherman --- docs/modules/developing-modules.md | 8 ++++---- docs/modules/module-registry.md | 2 +- docs/modules/using-modules.md | 12 ++++++------ docs/script.md | 2 +- 4 files changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/modules/developing-modules.md b/docs/modules/developing-modules.md index 134dcce0a0..79aec71737 100644 --- a/docs/modules/developing-modules.md +++ b/docs/modules/developing-modules.md @@ -26,7 +26,7 @@ The command creates a module directory with the following files: - `meta.yml`: The module spec describing metadata, inputs, and outputs. - `README.md`: Documentation for the module. -See {ref}`cli-module-create` for the full command reference. +See {ref}`module create ` for the full command reference. (dev-modules-structure)= @@ -76,7 +76,7 @@ process FASTQC { Registry modules are subject to the following constraints: -- The module must contain a single script named `main.nf` +- The module must contain a single script called `main.nf` - The module must define a single process - The module must not define any named workflows - The module may define an entry workflow to override the default behavior of `module run`. @@ -190,7 +190,7 @@ $ nextflow module spec \ When updating an existing module spec, it is incorporated into the new file. -See {ref}`cli-module-spec` for the full command reference. +See {ref}`module spec ` for the full command reference. (dev-modules-validate)= @@ -207,7 +207,7 @@ The command verifies that: - All required files are present (`main.nf`, `meta.yml`, `README.md`). - The module spec contains all required fields (name, version, description, and license). -See {ref}`cli-module-validate` for the full command reference. +See {ref}`module validate ` for the full command reference. (dev-modules-testing)= diff --git a/docs/modules/module-registry.md b/docs/modules/module-registry.md index 27c885a5f6..5cdefdd6c8 100644 --- a/docs/modules/module-registry.md +++ b/docs/modules/module-registry.md @@ -124,7 +124,7 @@ Before publishing, verify your module is ready: See {ref}`dev-modules-page` for a complete guide on creating modules. ::: -See {ref}`cli-module-publish` for the full command reference. +See {ref}`module publish ` for the full command reference. ## Registry configuration diff --git a/docs/modules/using-modules.md b/docs/modules/using-modules.md index f32ecaa287..ec43945a53 100644 --- a/docs/modules/using-modules.md +++ b/docs/modules/using-modules.md @@ -20,7 +20,7 @@ $ nextflow module search "quality control" -limit 10 Results include module names and descriptions. Use `-output json` for machine-readable output. -See {ref}`cli-module-search` for the full command reference. +See {ref}`module search ` for the full command reference. ## Installing modules @@ -42,7 +42,7 @@ Nextflow stores installed modules in the `modules/` directory and creates a `.mo Commit the `modules/` directory to your Git repository to ensure reproducibility. ::: -See {ref}`cli-module-install` for the full command reference. +See {ref}`module install ` for the full command reference. ## Listing installed modules @@ -55,7 +55,7 @@ $ nextflow module list The output shows each module's name, installed version, and whether it has been modified locally. Use `-output json` for machine-readable output. -See {ref}`cli-module-list` for the full command reference. +See {ref}`module list ` for the full command reference. ## Viewing module information @@ -69,7 +69,7 @@ $ nextflow module view nf-core/fastqc -version 0.0.0-0c7146d The output includes the module's version, URL, description, authors, maintainers, keywords, tools, input/output channels, and a generated usage template. Use `-output json` for machine-readable output. -See {ref}`cli-module-view` for the full command reference. +See {ref}`module view ` for the full command reference. ## Including modules @@ -126,7 +126,7 @@ $ nextflow module run ./modules/local/fastqc/main.nf \ The local module should define a single process, or else the command will fail. ::: -See {ref}`cli-module-run` for the full command reference. +See {ref}`module run ` for the full command reference. ## Updating modules @@ -158,4 +158,4 @@ Use flags to control this behavior: - `-keep-files`: Remove the `.module-info` file but keep the module files in the `modules/` directory. - `-force`: Remove the module directory even if it has no `.module-info` file or has local modifications. -See {ref}`cli-module-remove` for the full command reference. +See {ref}`module remove ` for the full command reference. diff --git a/docs/script.md b/docs/script.md index d25141749d..3f606b2e9d 100644 --- a/docs/script.md +++ b/docs/script.md @@ -561,4 +561,4 @@ workflow { } ``` -See {ref}`workflow-page`, {ref}`process-page`, and {ref}`modules-page` for more information about how to use these features in your Nextflow scripts. +See {ref}`Workflows `, {ref}`Processes `, and {ref}`Modules ` for more information about how to use these features in your Nextflow scripts. From b112542aded78d35eed2a1ef7690f5a11e742c99 Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Thu, 23 Apr 2026 10:26:07 -0500 Subject: [PATCH 14/14] Apply suggestions from review Signed-off-by: Ben Sherman --- docs/modules/developing-modules.md | 12 +-- docs/modules/using-modules.md | 2 + docs/workflow.md | 150 +++++++++-------------------- 3 files changed, 54 insertions(+), 110 deletions(-) diff --git a/docs/modules/developing-modules.md b/docs/modules/developing-modules.md index 79aec71737..910df03573 100644 --- a/docs/modules/developing-modules.md +++ b/docs/modules/developing-modules.md @@ -38,12 +38,12 @@ Registry modules follow a standard directory structure: modules/ └── myorg/ └── my-module/ - ├── .module-info # Integrity checksum (generated at install/creation) - ├── README.md # Documentation (required for publishing) - ├── main.nf # Module script (required) - ├── meta.yml # Module spec (required for publishing) - ├── resources/ # Optional: module resources - └── templates/ # Optional: process script templates + ├── .module-info # Integrity checksum + ├── README.md # Documentation + ├── main.nf # Module script + ├── meta.yml # Module spec + ├── resources/ # Optional: Module resources + └── templates/ # Optional: Process script templates ``` Local modules that are not intended for publishing do not need to follow this structure, although it is recommended as a best practice. diff --git a/docs/modules/using-modules.md b/docs/modules/using-modules.md index ec43945a53..4a9dec2c23 100644 --- a/docs/modules/using-modules.md +++ b/docs/modules/using-modules.md @@ -91,6 +91,8 @@ Local modules must be included by relative path: include { FASTQC } from './modules/local/fastqc' ``` +See {ref}`syntax-include` for a full description of the include syntax. + ## Running modules directly For ad-hoc tasks or testing, run a module directly without creating a wrapper workflow: diff --git a/docs/workflow.md b/docs/workflow.md index 18aa2807b0..46146d50a9 100644 --- a/docs/workflow.md +++ b/docs/workflow.md @@ -346,8 +346,8 @@ A *named workflow* is a workflow that can be called by other workflows: ```nextflow workflow my_workflow { - hello() - bye( hello.out.collect() ) + ch_hello = hello() + bye( ch_hello.collect() ) } workflow { @@ -368,8 +368,8 @@ workflow my_workflow { data2 main: - hello(data1, data2) - bye(hello.out) + ch_hello = hello(data1, data2) + bye(hello) } ``` @@ -385,31 +385,34 @@ The `emit:` section declares the outputs of a named workflow: ```nextflow workflow my_workflow { + take: + data + main: - hello(data) - bye(hello.out) + ch_bye = bye(hello(data)) emit: - bye.out + ch_bye } ``` -When calling the workflow, the output can be accessed using the `out` property, i.e. `my_workflow.out`. - If an output is assigned to a name, the name can be used to reference the output from the calling workflow. For example: ```nextflow workflow my_workflow { main: - hello(data) - bye(hello.out) + ch_hello = hello(data) + ch_bye = bye(ch_hello) emit: - my_data = bye.out + my_data = ch_bye } -``` -The result of the above workflow can be accessed using `my_workflow.out.my_data`. +workflow { + result = my_workflow() + result.my_data.view() +} +``` :::{note} Every output must be assigned to a name when multiple outputs are declared. @@ -421,9 +424,7 @@ Every output must be assigned to a name when multiple outputs are declared. Workflows consist of *dataflow* logic, in which processes are connected to each other through *dataflow channels* and *dataflow values*. -(dataflow-type-channel)= - -### Channels +### Channels and values A *dataflow channel* (or simply *channel*) is an asynchronous sequence of values. @@ -439,7 +440,21 @@ channel emits 2 channel emits 3 ``` -**Factories** +A *dataflow value* is an asynchronous value. + +Dataflow values can be created using the {ref}`channel.value ` factory, and they are created by processes (under {ref}`certain conditions `). + +A dataflow value cannot be accessed directly, but only through an operator or process. For example: + +```nextflow +channel.value(1).view { v -> "dataflow value is ${v}" } +``` + +```console +dataflow value is 1 +``` + +### Factories A channel can be created by factories in the `channel` namespace. For example, the `channel.fromPath()` factory creates a channel from a file name or glob pattern, similar to the `files()` function: @@ -449,7 +464,7 @@ channel.fromPath('input/*.txt').view() See {ref}`channel-factory` for the full list of channel factories. -**Operators** +### Operators Channel operators, or *operators* for short, are functions that consume and produce channels. Because channels are asynchronous, operators are necessary to manipulate the values in a channel. Operators are particularly useful for implementing glue logic between processes. @@ -473,27 +488,7 @@ Commonly used operators include: - {ref}`operator-view`: print each channel value to standard output -See {ref}`operator-page` for the full set of operators. - -(dataflow-type-value)= - -### Values - -A *dataflow value* is an asynchronous value. - -Dataflow values can be created using the {ref}`channel.value ` factory, and they are created by processes (under {ref}`certain conditions `). - -A dataflow value cannot be accessed directly, but only through an operator or process. For example: - -```nextflow -channel.value(1).view { v -> "dataflow value is ${v}" } -``` - -```console -dataflow value is 1 -``` - -See {ref}`stdlib-types-value` for the set of available methods for dataflow values. +See {ref}`operator-page` for the full set of operators. See {ref}`stdlib-types-value` for the set of available methods for dataflow values. (workflow-process-invocation)= @@ -540,10 +535,9 @@ workflow { } ``` -Processes and workflows can only be called by workflows. A given process or workflow can only be called once in a given workflow. To use a process or workflow multiple times in the same workflow, include it from another script with multiple aliases: +Processes and workflows can only be called by workflows. A given process or workflow can only be called once in a given workflow. To use a process or workflow multiple times in the same workflow, {ref}`include ` it from another script with multiple aliases: ```nextflow -// workflow `hello_bye` is defined in `./modules/hello_bye/main.nf` include { hello_bye as hello_bye1 } from './modules/hello_bye' include { hello_bye as hello_bye2 } from './modules/hello_bye' @@ -623,10 +617,6 @@ workflow { Process named outputs are defined using the `emit` option on a process output. See {ref}`naming process outputs ` for more information. ::: -:::{note} -Process and workflow outputs can also be accessed by index (e.g., `hello.out[0]`, `hello.out[1]`, etc.). As a best practice, multiple outputs should be accessed by name. -::: - Workflows can be composed in the same way: ```nextflow @@ -634,36 +624,25 @@ workflow flow1 { take: data - main: - tick(data) - tack(tick.out) - emit: - tack.out + tack(tick(data)) } workflow flow2 { take: data - main: - tick(data) - tock(tick.out) - emit: - tock.out + tock(tick(data)) } workflow { data = channel.fromPath('/some/path/*.txt') - flow1(data) - flow2(flow1.out) + flow2(flow1(data)) } ``` -:::{note} The same process can be called in different workflows without using an alias, like `tick` in the above example, which is used in both `flow1` and `flow2`. The workflow call stack determines the *fully qualified process name*, which is used to distinguish the different process calls, i.e. `flow1:tick` and `flow2:tick` in the above example. -::: :::{tip} The fully qualified process name can be used as a {ref}`process selector ` in a Nextflow configuration file, and it takes priority over the simple process name. @@ -671,55 +650,18 @@ The fully qualified process name can be used as a {ref}`process selector ` is enabled. Using these operators will prevent the type checker from validating your code. +:::{deprecated} 26.04.0 +These operators are not supported when {ref}`static typing ` is enabled. Use standard method calls and assignments instead. ::: -**Pipe `|`** - -The `|` *pipe* operator can be used to chain processes, operators, and workflows: - -```nextflow -process greet { - input: - val data - - output: - val result - - exec: - result = "$data world" -} - -workflow { - channel.of('Hello', 'Hola', 'Ciao') - | greet - | map { v -> v.toUpperCase() } - | view -} -``` +The following operators have a special meaning when used with process and workflow calls in a workflow: -The above snippet defines a process named `greet` and invokes it with the input channel. The result is then piped to the {ref}`operator-map` operator, which converts each string to uppercase, and finally to the {ref}`operator-view` operator which prints it. +- The `|` *pipe* operator can be used to chain processes, operators, and workflows. +- The `&` *and* operator can be used to call multiple processes in parallel with the same channel(s). -The same code can also be written as: - -```nextflow -workflow { - ch_input = channel.of('Hello', 'Hola', 'Ciao') - ch_greet = greet(ch_input) - ch_greet - .map { v -> v.toUpperCase() } - .view() -} -``` - -**And `&`** - -The `&` *and* operator can be used to call multiple processes in parallel with the same channel(s): +For example: ```nextflow process greet {