Skip to content

Add NVMe disk allocation and diskMountPath support#6879

Merged
pditommaso merged 4 commits intomasterfrom
pditommaso/nvme-disk-mount-path
Mar 2, 2026
Merged

Add NVMe disk allocation and diskMountPath support#6879
pditommaso merged 4 commits intomasterfrom
pditommaso/nvme-disk-mount-path

Conversation

@pditommaso
Copy link
Member

@pditommaso pditommaso commented Mar 2, 2026

Summary

  • Add diskMountPath config option to seqera.executor.machineRequirement for specifying the container mount path (default: /tmp), applicable to all disk allocation strategies
  • Add NVMe disk allocation support (diskAllocation = 'nvme') which uses instance store disks instead of EBS volumes, with validation that EBS-specific options are not set
  • Update documentation for the new diskMountPath option in executor and config reference docs

Test plan

  • Unit tests for NVMe allocation without disk size
  • Unit tests for NVMe allocation with disk size
  • Unit tests for NVMe disk in machine requirement
  • Unit tests for diskMountPath across all allocation types (node, task, nvme)
  • Unit tests for EBS option validation with NVMe allocation
  • Run ./gradlew :plugins:nf-seqera:test to verify all tests pass

🤖 Generated with Claude Code

@pditommaso @claude
Add NVMe disk allocation and diskMountPath support for Seqera executor
eeaea42 
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
@pditommaso pditommaso requested a review from a team as a code owner March 2, 2026 17:58
@netlify
Copy link

netlify bot commented Mar 2, 2026 •
edited
Loading

✅ Deploy Preview for nextflow-docs-staging canceled.

Name Link
🔨 Latest commit 0cc86ab
🔍 Latest deploy log https://app.netlify.com/projects/nextflow-docs-staging/deploys/69a60e0247a7ae00088680bf

@pditommaso pditommaso marked this pull request as draft March 2, 2026 17:59
pditommaso and others added 2 commits March 2, 2026 21:16
@pditommaso @claude
docs: clarify diskMountPath default varies by allocation strategy
9a6b7e1 
For task/node allocation the default is /tmp; for nvme allocation
the default is the server-configured NVMe mount path (typically /scratch).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
@pditommaso
Bump sched dep [ci fast]
abfd667 
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
@pditommaso pditommaso marked this pull request as ready for review March 2, 2026 22:17
@pditommaso
Bump version 0.13.0 [ci skip]
0cc86ab 
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
@pditommaso pditommaso merged commit 6644d3c into master Mar 2, 2026
6 checks passed
@pditommaso pditommaso deleted the pditommaso/nvme-disk-mount-path branch March 2, 2026 22:25
pditommaso added a commit that referenced this pull request Mar 3, 2026
@pditommaso @claude
Add NVMe disk allocation and diskMountPath support (#6879) [ci fast]
7d772ea 
* Add NVMe disk allocation and diskMountPath support for Seqera executor

For task/node allocation the default is /tmp; for nvme allocation
the default is the server-configured NVMe mount path (typically /scratch).


Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
pditommaso added a commit that referenced this pull request Mar 17, 2026
@pditommaso @claude
@jorgee @ewels @bentsherman @LarytheLord @PeterKneale @christopher-hakkaart @rzelle-lallemand @colinbrislawn @fntlnz @adamrtalbot @edmundmiller
Module system (#6650)
a0ac594 
* Add Nextflow Development Constitution

Introduce comprehensive development constitution documenting core principles
and practices for Nextflow development including modular architecture,
test-driven quality assurance, dataflow programming model, licensing
compliance, DCO requirements, semantic versioning, and Groovy code standards.

The constitution codifies existing best practices from CLAUDE.md and
CONTRIBUTING.md to provide clear governance and quality standards for
the project.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

* Module adr v1

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

* Module adr v2

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

* ADR: Nextflow remote modules system v2.1

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

* Add detailed CLI command descriptions to module system ADR [ci skip]

- Add comprehensive documentation for all module CLI commands
- Add `nextflow module run` command for standalone module execution
- Remove `module update` command to simplify the design
- Use single-dash prefix for Nextflow options, double-dash for module inputs
- Remove @ prefix from scope in CLI commands (keep only in DSL syntax)

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

* Simplify module spec: unify dependencies under requires

- Remove separate `dependencies` and `components` fields
- Expand `requires` to include:
  - `nextflow`: version constraint (unchanged)
  - `plugins`: array with name@constraint syntax
  - `modules`: array of module dependencies
  - `workflows`: array of workflow dependencies
- Unified version constraint syntax: `[scope/]name[@constraint]`
- Mark `components` as deprecated (use requires.modules)
- Update all examples in ADR and schema

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

* Add structured tool arguments to module spec schema

- Add `args` property to tools section for type-safe argument configuration
- Define toolArgSpec with flag, type, description, default, enum, required
- Support implicit variable `tools.<toolname>.args.<argname>` returning
  formatted flag+value (e.g., "-K 100000000")
- Support `tools.<toolname>.args` to return all args concatenated
- Document deprecation of ext.args/ext.args2/ext.args3 pattern
- Update ADR with Tool Arguments Configuration section and appendix

[ci skip]

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

* Update adr/20251114-module-system.md [ci skip]

Co-authored-by: Jorge Ejarque <jorgee@users.noreply.github.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

* Update adr/20251114-module-system.md [ci skip]

Co-authored-by: Jorge Ejarque <jorgee@users.noreply.github.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

* Update adr/20251114-module-system.md [ci skip]

Co-authored-by: Jorge Ejarque <jorgee@users.noreply.github.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

* Update adr/20251114-module-system.md [ci skip]

Expand deprecation notice to cover all ext.* custom directives

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

* Fix inconsistencies in module system ADR [ci skip]

- Use consistent module path format with version: modules/@scope/name@version/
- Fix directory structure example: samtools-view -> samtools/view
- Standardize on 'license' spelling (American English)
- Fix author -> authors (plural array format)

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

* Fix registry API path in comparison table [ci skip]

Remove @ prefix from scope in API path to match API definition

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

* Fix checksum format: sha256- -> sha256: [ci skip]

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

* Update module API to match registry implementation [ci skip]

- Update API endpoints to match seqeralabs/plugin-registry#266
- Use /api/modules base path (no v1 prefix)
- Use single {name} parameter with namespace (e.g., "nf-core/fastqc")
- Add separate /releases endpoint
- Simplify publish to single POST endpoint

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

* Update adr [ci skip]

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

* Simplify module storage and verification model [ci skip]

- Remove checksums from nextflow.config (registry is source of truth)
- Use single version per module locally (no version in directory path)
- Add .checksum file in module directory for integrity verification
- Simplify freeze command to only pin transitive dependency versions
- Checksum mismatch reports warning instead of automatic re-download
- Remove resolved open question about checksum verification

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

* Update adr/20251114-module-system.md [ci skip]

Co-authored-by: Jorge Ejarque <jorgee@users.noreply.github.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

* Clarify module resolution rules for version changes vs local modifications [ci skip]

- Added Resolution Rules table clearly specifying behavior for each scenario:
  - Version change (local unmodified): automatically replace with declared version
  - Local modification (checksum mismatch): warn and protect local changes
  - Use -force flag to override locally modified modules
- Updated install command behavior to reflect checksum verification
- Updated Technical Details with expanded resolution flow

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

* ADR v2.3: Fix resolution rules consistency and add version entry [ci skip]

- Fixed Resolution Rules table: locally modified modules will NOT be
  replaced unless -force is used (was incorrectly saying "will be replaced")
- Added Version 2.3 changelog entry documenting:
  - Resolution Rules table with clear behavior matrix
  - Local modification protection with -force flag
  - Simplified storage model (single version per module)
  - .checksum file for fast integrity verification

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

* Fix Version 2.1 date typo: 2025 -> 2024 [ci skip]

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

* Add module system client specification [ci skip]

Specification for Nextflow module system client implementation based on
ADR 20251114-module-system.md. Covers:

P1 (Core):
- Install and use registry modules via @scope/name syntax
- Run modules directly from CLI without wrapper workflow
- Structured tool arguments replacing ext.args pattern

P2 (Important):
- Module version management and freeze command
- Module integrity protection with checksum validation

P3 (Nice to have):
- Remove module command
- Search and discover modules
- Publish module to registry

Registry backend is out of scope (assumed implemented).

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

* Remove transitive dependency resolution from module system [ci skip]

- Remove `freeze` command from CLI
- Remove transitive dependency install behavior
- Remove orphaned transitive dependency removal from `remove` command
- Update rationale and consequences sections
- Simplify dependency resolution flow
- Update ADR to version 2.4

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

* Update adr/20251114-module-system.md [ci skip]

Co-authored-by: Jorge Ejarque <jorgee@users.noreply.github.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

* Update adr/20251114-module-system.md [ci skip]

Co-authored-by: Phil Ewels <phil.ewels@seqera.io>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

* Update modules plan [ci skip]

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

* Remove tools config [ci skip]

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

* Simplify module system requires definition [ci skip]

- Remove plugins, modules, subworkflows from requires block
- Spec now focused on process modules only
- Dependencies managed via nextflow.config

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

* Replace tool arguments with module parameters in ADR [ci skip]

- Module parameters defined in meta.yaml params section with name, type, description, and example attributes
- Removed tools args property and specification
- Updated schema and examples throughout

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

* Update specs + adr [ci skip]

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

* Resolve some conflicts with nextflow-io/schemas#10

Signed-off-by: Ben Sherman <bentshermann@gmail.com>

* Update adr/20251114-module-system.md [ci skip]

Co-authored-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

* Extract module parameters to separate spec [ci skip]

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

* docs: clarify path name for staged task inputs (#6869)

Co-authored-by: Ben Sherman <bentshermann@gmail.com>

* Add devcontainer (#6792)

Co-authored-by: Phil Ewels <phil.ewels@seqera.io>
Co-authored-by: Ben Sherman <bentshermann@gmail.com>

* Fix vertical alignment in timeline report (#6794)

* Ensure main script is first in the WorkflowRun lineage record (#6845)

* Add whitespace rules to .editorconfig (#5606)

* docs: Update your first script to use outputs (#6500)

* Update AWS CLI install docs to bypass conda install TOS prompt (#6685)

* Clarify onError and onComplete handler descriptions (#6709) [ci fast]

* Use explicit memory units for LSF executor (#5217)

Signed-off-by: Ben Sherman <bentshermann@gmail.com>

* Treat LSF job status UNKWN as HOLD (#5756)

* Allow boolean params to implicitly default to false (#6764)

* Sched core implementation alpha1 (#6242)


Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Lorenzo Fontana <fontanalorenz@gmail.com>

* Revert version 26.01.1-edge

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

* Bump copyright 2026

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

* Handle Azure Batch ActiveJobAndScheduleQuotaReached with retry (#6874)

* Handle Azure Batch ActiveJobAndScheduleQuotaReached with retry (#5575)

Add retry logic when Azure Batch returns HTTP 409 with
ActiveJobAndScheduleQuotaReached error code during job creation,
instead of failing immediately. Configurable via
azure.batch.maxJobQuotaRetries (default 3) and
azure.batch.jobQuotaRetryDelay (default 2 min).

Generated by Claude Code

Signed-off-by: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com>

* Refactor job quota retry to use Failsafe RetryPolicy

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

---------

Signed-off-by: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Co-authored-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

* [ci fast] Add agent output mode (NXF_AGENT_MODE)

Add AI agent-friendly output mode activated via NXF_AGENT_MODE=1.

When enabled, Nextflow replaces the interactive ANSI progress display with
minimal, structured tagged lines optimized for AI agent context windows:

  [PIPELINE] name version | profile=X
  [WORKDIR] /path/to/work
  [PROCESS hash] name (tag)
  [WARN] deduplicated warning
  [ERROR] name with exit/cmd/stderr/workdir
  [SUCCESS|FAILED] completed=N failed=N cached=N

Key changes:
- LogObserver interface as common abstraction for AnsiLogObserver and AgentLogObserver
- Session holds a single logObserver field instead of two concrete fields
- DefaultObserverFactory creates both observer types using the same pattern
- AgentLogObserver filters hash-prefixed log lines to avoid duplicating [PROCESS] output
- Workflow output (view operator) passes through to stdout in agent mode
- Renamed env var from NXF_AGENT to NXF_AGENT_MODE
- Added NXF_AGENT_MODE to environment variables documentation

* Bump nf-amazon@3.7.1

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

* Bump nf-azure@1.22.0

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

* Bump nf-google@1.26.1

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

* Bump nf-tower@1.21.0

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

* Bump nf-seqera@0.12.0

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

* Bump nf-tower@1.21.0 nextflowVersion

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

* Bump nf-seqera@0.12.0 nextflowVersion and README

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

* Fix AgentLogObserver println clash with Groovy GDK method

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

* Add Summary section to plugin README files

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

* [release] [ci fast] Nextflow version 26.02.0-edge

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

* Fix docs [ci skip]

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

* Add logStreamId to Tower task record (#6877)

Propagate the log stream identifier from TraceRecord to the Tower
task record so that Seqera Platform can link tasks to their cloud
log streams.

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

* Don't include container in task trace when containers are not enabled (#6824)

* Propagate errors from workflow outputs (#6876)

* Fix type detection of CLI params in v2 config parser (#6765)

* Add NVMe disk allocation and diskMountPath support (#6879) [ci fast]

* Add NVMe disk allocation and diskMountPath support for Seqera executor

For task/node allocation the default is /tmp; for nvme allocation
the default is the server-configured NVMe mount path (typically /scratch).


Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

* Version 2.7: rename `.checksum` to `.module-info`, remove `@` prefix from module scopes, remove version pinning from `nextflow.config`

Signed-off-by: Ben Sherman <bentshermann@gmail.com>

* Module System Implementation (#6768)

Signed-off-by: jorgee <jorge.ejarque@seqera.io>

* minor edits

Signed-off-by: Ben Sherman <bentshermann@gmail.com>

* Cleanup module run output

Signed-off-by: Ben Sherman <bentshermann@gmail.com>

* Fix meta.yaml -> meta.yml, module manifest -> module spec

Signed-off-by: Ben Sherman <bentshermann@gmail.com>

* Fix failing test

Signed-off-by: Ben Sherman <bentshermann@gmail.com>

* fix tests

Signed-off-by: jorgee <jorge.ejarque@seqera.io>

* modify module list to skip subdirectories of directories that already contains the .module-info. This folders can't be a module

Signed-off-by: jorgee <jorge.ejarque@seqera.io>

* Update docs

Signed-off-by: Ben Sherman <bentshermann@gmail.com>

* Apply suggestion from @pditommaso [ci skip]

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

---------

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com>
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Jorge Ejarque <jorgee@users.noreply.github.com>
Co-authored-by: Phil Ewels <phil.ewels@seqera.io>
Co-authored-by: Ben Sherman <bentshermann@gmail.com>
Co-authored-by: Abi <llawlietbagsum@gmail.com>
Co-authored-by: Peter Kneale <PeterKneale@gmail.com>
Co-authored-by: Chris Hakkaart <chris.hakkaart@seqera.io>
Co-authored-by: Rintze Zelle <78232505+rzelle-lallemand@users.noreply.github.com>
Co-authored-by: Colin J. Brislawn <cbrisl@gmail.com>
Co-authored-by: Lorenzo Fontana <fontanalorenz@gmail.com>
Co-authored-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com>
Co-authored-by: Edmund Miller <20095261+edmundmiller@users.noreply.github.com>
Co-authored-by: jorgee <jorge.ejarque@seqera.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@pditommaso