From dd54e1f30e86953c818a1052f32559aa6fba65b0 Mon Sep 17 00:00:00 2001 From: Jen Reeve Date: Thu, 6 Nov 2025 16:02:22 +1300 Subject: [PATCH 01/12] quick draft --- .../Supported_Applications/Nextflow.md | 76 +++++++++++++++++++ 1 file changed, 76 insertions(+) create mode 100644 docs/Scientific_Computing/Supported_Applications/Nextflow.md diff --git a/docs/Scientific_Computing/Supported_Applications/Nextflow.md b/docs/Scientific_Computing/Supported_Applications/Nextflow.md new file mode 100644 index 000000000..c0aa53057 --- /dev/null +++ b/docs/Scientific_Computing/Supported_Applications/Nextflow.md @@ -0,0 +1,76 @@ + +[//]: <> (APPS PAGE BOILERPLATE START) +{% set app_name = page.title | trim %} +{% set app = applications[app_name] %} +{% include "partials/app_header.html" %} +[//]: <> (APPS PAGE BOILERPLATE END) + +[Nextflow](https://www.nextflow.io/) is a workflow system for creating scalable, portable, and reproducible workflows. It uses a dataflow programming model that simplifies writing parallel and distributed pipelines by allowing you to focus on data flow and computation. Nextflow can deploy workflows on a variety of execution platforms, including your local machine, HPC schedulers, and cloud. Additionally, Nextflow supports a range of compute environments, software container runtimes, and package managers, allowing workflows to be executed in reproducible and isolated environments. + +## Running Nextflow in an interactive Slurm session + +## Submitting a Nextflow workflow as a batch job + +The following `sbatch` will submit a Nextflow workflow as a Slurm job. All of the Nextflow processes will run on the same compute node, so request enough resources to run the most intensive process in the workflow and enough time for the entire workflow to complete. + +```sl +#!/bin/bash -e + +#SBATCH --job-name my job +#SBATCH --account nesi12345 +#SBATCH --time 01:00:00 +#SBATCH --mem 2G + +# load Nextflow and set environmental variables +module load Nextflow/25.10.0 +export NXF_APPTAINER_CACHEDIR=/nesi/nobackup/nesi12345/apptainer_cache +export NXF_OFFLINE='true' +export NXF_PLUGINS_DIR=/nesi/project/nesi12345/user/PROJECT_DIR/.nextflow/plugins + +# run nextflow +nextflow run NEXTFLOW_WORKFLOW \ + -profile local,apptainer \ + --outdir /nesi/project/nesi12345/user/PROJECT_DIR/NEXTFLOW_WORKFLOW/out \ + -w /nesi/nobackup/nesi12345/user/PROJECT_DIR/NEXTFLOW_WORKFLOW/work \ + -with-trace \ + -with-report \ + -with-dag +``` + +## Configurations + +Below is an example `nextflow.config` file with some configuration settings that will help when running Nextflow via NeSI. + +```json +// enable use of apptainer to run containers +apptainer { + apptainer.pullTimeout = '1h' +} + +// Default settings for all profiles +process { + stageInMode = 'symlink' + cache = 'lenient' +} + +// Specific profiles to use in different contexts +profiles { + debug { + cleanup = false + } + local { + executor = 'local' + } + slurm { + executor = 'slurm' + executor.queue = 'genoa,milan' + executor.queueSize = 100 + } +} + +cleanup = true +``` + +## Community pipelines - nf-core + +[nf-core](https://nf-co.re/) is a global community collaborating to build open-source Nextflow components and pipelines. Many standard pipelines and tools have nf-core pipelines or modules which allow you to skip straight to running the pipeline. From 0575942bff87d56bb5049e818acac4ea8b241d26 Mon Sep 17 00:00:00 2001 From: Jen Reeve Date: Fri, 7 Nov 2025 10:43:58 +1300 Subject: [PATCH 02/12] slurm executor --- .../Supported_Applications/Nextflow.md | 38 +++++++++++++++---- 1 file changed, 31 insertions(+), 7 deletions(-) diff --git a/docs/Scientific_Computing/Supported_Applications/Nextflow.md b/docs/Scientific_Computing/Supported_Applications/Nextflow.md index c0aa53057..081b9d6c5 100644 --- a/docs/Scientific_Computing/Supported_Applications/Nextflow.md +++ b/docs/Scientific_Computing/Supported_Applications/Nextflow.md @@ -5,8 +5,6 @@ {% include "partials/app_header.html" %} [//]: <> (APPS PAGE BOILERPLATE END) -[Nextflow](https://www.nextflow.io/) is a workflow system for creating scalable, portable, and reproducible workflows. It uses a dataflow programming model that simplifies writing parallel and distributed pipelines by allowing you to focus on data flow and computation. Nextflow can deploy workflows on a variety of execution platforms, including your local machine, HPC schedulers, and cloud. Additionally, Nextflow supports a range of compute environments, software container runtimes, and package managers, allowing workflows to be executed in reproducible and isolated environments. - ## Running Nextflow in an interactive Slurm session ## Submitting a Nextflow workflow as a batch job @@ -61,16 +59,42 @@ profiles { local { executor = 'local' } - slurm { - executor = 'slurm' - executor.queue = 'genoa,milan' - executor.queueSize = 100 - } } cleanup = true ``` +## Time/compute intensive processes + +If any of your individual processes regularly take longer than 30 minutes, you can flag them to be submitted as additional Slurm jobs separate from the head job. To do this, give these processes a [label](https://www.nextflow.io/docs/latest/reference/process.html#label) such as `'slurm_array'` and add the following to your `nextflow.config` file: + +```json +process { + withLabel: slurm_array { + executor = 'slurm' + queue = 'genoa,milan' + queueSize = 10 + jobName = { "$task.name - $task.hash" } + slurm_array = 100 + } +} +``` + +This will allow the main Nextflow Slurm job to submit this process to Slurm as separate tasks. `queueSize` limits the number of these additional Slurm jobs that can be queued simultaneously, while `array` tells Nextflow to group up to 100 processes with label and submit them as a job array. You can provide more specific labels and set up the configuration to request different resources for different processes. + +!!! warning "Avoid many short jobs" + This will put a major burden on the Slurm scheduler for no improvement in your computational speed. Do not use the Nextflow `slurm` executor for jobs which take less than 30 minutes to complete. + + + +## Improving efficiency + +Nextflow provides tools that can assist you in making efficient use of the HPC resources. Running a pipeline with the CLI option `-with-report` will produce [a HTML execution report containing CPU and memory utilization information](https://www.nextflow.io/docs/latest/reports.html#execution-report) for each individual process as well as each process type. This information can be used to ensure processes are only getting the resources they need. + ## Community pipelines - nf-core [nf-core](https://nf-co.re/) is a global community collaborating to build open-source Nextflow components and pipelines. Many standard pipelines and tools have nf-core pipelines or modules which allow you to skip straight to running the pipeline. + +!!! warning "Nextflow plugins" + nf-core pipelines expect to use nf-plugins in their base configuration. If you want to use these plugins, you will need to manually download them and store them in a plugin cache directory that you can specify with the `NXF_PLUGINS_DIR` environmental variable (as in the example `.sl` above) + \ No newline at end of file From 9aa2d72da357c9ed2114d37030e433fd55109185 Mon Sep 17 00:00:00 2001 From: Jen Reeve Date: Fri, 7 Nov 2025 10:48:36 +1300 Subject: [PATCH 03/12] adding nextflow ref to cylc --- .../Supported_Applications/Cylc.md | 33 +++++++++++++++---- 1 file changed, 26 insertions(+), 7 deletions(-) diff --git a/docs/Scientific_Computing/Supported_Applications/Cylc.md b/docs/Scientific_Computing/Supported_Applications/Cylc.md index 0e6440db4..99eeadff2 100644 --- a/docs/Scientific_Computing/Supported_Applications/Cylc.md +++ b/docs/Scientific_Computing/Supported_Applications/Cylc.md @@ -20,8 +20,8 @@ running a monolithic, parallel executable is that each task will require less resources that the complete problem, it is thus easier for each task to slip into the queue and start running. -See the NeSI  [Snakemake](https://snakemake-on-nesi.sschmeier.com/) page -for another, possible choice. +See the NeSI [Snakemake](https://snakemake-on-nesi.sschmeier.com/) and [Nextflow](https://nextflow.io/) pages +for other possible choices. In this article, we show how you can create a simple workflow and run it on NeSI's platform. Consult the [Cylc @@ -42,7 +42,7 @@ filesystem, so this is easy): - run **`ssh-keygen`** to generate a public/private key pair with **no passphrase** (when it asks for a passphrase, just hit enter) - add your own public key to your authorized keys - file: **`cat .ssh/id_rsa.pub >> .ssh/authorized_keys`**  + file: **`cat .ssh/id_rsa.pub >> .ssh/authorized_keys`** - check that your **keys, authorized\_keys file, ssh directory, **and** home directory** all have sufficiently secure file permissions. If not, `ssh` will silently revert to requiring @@ -57,39 +57,50 @@ being asked for a passphrase. ## How to install Cylc Create a new conda environment and install Cylc with the commands: + ``` sh module purge module load Miniforge3 export CYLC_HOME=/nesi/project/nesi99999/$USER/environment/cylc-env conda create --prefix $CYLC_HOME python=3.12 ``` -where `CYLC_HOME` points to the installation path of your choice. Adjust `nesi99999` to fit your project number. It is not recommended to install in your home directory. + +where `CYLC_HOME` points to the installation path of your choice. Adjust `nesi99999` to fit your project number. It is not recommended to install in your home directory. Then type + ``` sh conda init ``` + Close and start a new shell. Now activate the new environment + ``` sh module purge && module load Miniforge3 conda activate /nesi/project/nesi99999/$USER/environment/cylc-env # or whereever you installed cylc-env conda install -c conda-forge cylc-flow ``` + Check that `cylc` was successfully installed + ``` sh cylc --version ``` -should return + +should return + ``` output 8.6.0 ``` + or a later version. ## Create a cylc wrapper script In order to allow `cylc` to be invoked through `SLURM` the following steps are required + ``` sh mkdir $CYLC_HOME/wrapper cylc get-resources cylc $CYLC_HOME/wrapper @@ -99,20 +110,23 @@ sed -i "s|CYLC_HOME_ROOT=\"\${CYLC_HOME_ROOT:-/opt}\"|CYLC_HOME_ROOT=\"\${CYLC_H ``` Make sure the wrapper script is found when opening a new shell + ``` sh export CYLC_HOME=/nesi/project/nesi99999/$USER/environment/cylc-env # adapt export PATH=$CYLC_HOME/wrapper:$PATH ``` - ## Setting up Cylc on Mahuika In order to allow Cylc to submit tasks to the SLURM scheduler, we need to configure platforms. Edit (or create) your global configuration file: + ``` sh mkdir -p ~/.cylc/flow/ vim ~/.cylc/flow/global.cylc ``` + or use the editor of your choice, and add the following lines + ``` sh [platforms] [[mahuika-slurm]] @@ -131,15 +145,20 @@ cylc vip . ``` This will submit a job to SLURM + ``` sh squeue --me ``` -should return + +should return + ``` output JOBID USER ACCOUNT NAME CPUS MIN_MEM PARTITI START_TIME TIME_LEFT STATE NODELIST(REASON) 2974409 pletzera nesi99999 a.1.slurm/ru 1 512M milan,g N/A 1:00 PENDING (Priority) ``` + You casn also monitor the tasks with the command + ``` sh cylc tui slurm ``` From adf231ae4d78363bc814edf137e56b1d5e2b92ec Mon Sep 17 00:00:00 2001 From: Jen Reeve Date: Fri, 7 Nov 2025 11:03:27 +1300 Subject: [PATCH 04/12] interactive session info --- .../Supported_Applications/Nextflow.md | 75 ++++++++++++------- 1 file changed, 46 insertions(+), 29 deletions(-) diff --git a/docs/Scientific_Computing/Supported_Applications/Nextflow.md b/docs/Scientific_Computing/Supported_Applications/Nextflow.md index 081b9d6c5..cc447cae4 100644 --- a/docs/Scientific_Computing/Supported_Applications/Nextflow.md +++ b/docs/Scientific_Computing/Supported_Applications/Nextflow.md @@ -5,8 +5,54 @@ {% include "partials/app_header.html" %} [//]: <> (APPS PAGE BOILERPLATE END) +## Configurations + +Below is an example `nextflow.config` file with some configuration settings that will help when running Nextflow via NeSI. + +```json +// enable use of apptainer to run containers +apptainer { + apptainer.pullTimeout = '1h' +} + +// Default settings for all profiles +process { + stageInMode = 'symlink' + cache = 'lenient' +} + +// Specific profiles to use in different contexts +profiles { + debug { + cleanup = false + } + local { + executor = 'local' + } +} + +cleanup = true +``` + ## Running Nextflow in an interactive Slurm session +Running Nextflow in an interactive session can be especially helpful when setting up or debugging a pipeline. To do so, request an interactive session with appropriate resources for the pipeline: + +``` +srun --account nesi12345 --job-name "InteractiveJob" --cpus-per-task 8 --mem-per-cpu 1500 --time 24:00:00 --pty bash +``` + +Once your interactive session has launched, load the Nextflow module with `module load Nextflow/25.10.1` (or your required version) and proceed. Parallelization of Nextflow processes will occur within this job. + +There are several environmental variables that can be helpful to avoid loading containers and plugins into your persistent project space: + +``` +export NXF_APPTAINER_CACHEDIR=/nesi/nobackup/nesi12345/apptainer_cache +export NXF_PLUGINS_DIR=/nesi/project/nesi12345/user/PROJECT_DIR/.nextflow/plugins +``` + +You can confirm that Nextflow is loaded and ready using the `nextflow run hello` command to test Nextflow's Hello World pipeline. + ## Submitting a Nextflow workflow as a batch job The following `sbatch` will submit a Nextflow workflow as a Slurm job. All of the Nextflow processes will run on the same compute node, so request enough resources to run the most intensive process in the workflow and enough time for the entire workflow to complete. @@ -35,35 +81,6 @@ nextflow run NEXTFLOW_WORKFLOW \ -with-dag ``` -## Configurations - -Below is an example `nextflow.config` file with some configuration settings that will help when running Nextflow via NeSI. - -```json -// enable use of apptainer to run containers -apptainer { - apptainer.pullTimeout = '1h' -} - -// Default settings for all profiles -process { - stageInMode = 'symlink' - cache = 'lenient' -} - -// Specific profiles to use in different contexts -profiles { - debug { - cleanup = false - } - local { - executor = 'local' - } -} - -cleanup = true -``` - ## Time/compute intensive processes If any of your individual processes regularly take longer than 30 minutes, you can flag them to be submitted as additional Slurm jobs separate from the head job. To do this, give these processes a [label](https://www.nextflow.io/docs/latest/reference/process.html#label) such as `'slurm_array'` and add the following to your `nextflow.config` file: From c1878fd1258acafee3fb002752217e9e5bc76fe6 Mon Sep 17 00:00:00 2001 From: Jen Reeve Date: Fri, 7 Nov 2025 11:06:48 +1300 Subject: [PATCH 05/12] code block fix --- .../Supported_Applications/Nextflow.md | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/docs/Scientific_Computing/Supported_Applications/Nextflow.md b/docs/Scientific_Computing/Supported_Applications/Nextflow.md index cc447cae4..fa1994f70 100644 --- a/docs/Scientific_Computing/Supported_Applications/Nextflow.md +++ b/docs/Scientific_Computing/Supported_Applications/Nextflow.md @@ -38,7 +38,7 @@ cleanup = true Running Nextflow in an interactive session can be especially helpful when setting up or debugging a pipeline. To do so, request an interactive session with appropriate resources for the pipeline: -``` +```bash srun --account nesi12345 --job-name "InteractiveJob" --cpus-per-task 8 --mem-per-cpu 1500 --time 24:00:00 --pty bash ``` @@ -46,7 +46,7 @@ Once your interactive session has launched, load the Nextflow module with `modul There are several environmental variables that can be helpful to avoid loading containers and plugins into your persistent project space: -``` +```bash export NXF_APPTAINER_CACHEDIR=/nesi/nobackup/nesi12345/apptainer_cache export NXF_PLUGINS_DIR=/nesi/project/nesi12345/user/PROJECT_DIR/.nextflow/plugins ``` @@ -101,8 +101,6 @@ This will allow the main Nextflow Slurm job to submit this process to Slurm as s !!! warning "Avoid many short jobs" This will put a major burden on the Slurm scheduler for no improvement in your computational speed. Do not use the Nextflow `slurm` executor for jobs which take less than 30 minutes to complete. - - ## Improving efficiency @@ -114,4 +112,3 @@ Nextflow provides tools that can assist you in making efficient use of the HPC r !!! warning "Nextflow plugins" nf-core pipelines expect to use nf-plugins in their base configuration. If you want to use these plugins, you will need to manually download them and store them in a plugin cache directory that you can specify with the `NXF_PLUGINS_DIR` environmental variable (as in the example `.sl` above) - \ No newline at end of file From 395ab211951fe471178675940e60616ac003621e Mon Sep 17 00:00:00 2001 From: Jen Reeve Date: Mon, 10 Nov 2025 10:49:50 +1300 Subject: [PATCH 06/12] working on benchmarking table --- .../Supported_Applications/Nextflow.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/docs/Scientific_Computing/Supported_Applications/Nextflow.md b/docs/Scientific_Computing/Supported_Applications/Nextflow.md index fa1994f70..7fa40f28b 100644 --- a/docs/Scientific_Computing/Supported_Applications/Nextflow.md +++ b/docs/Scientific_Computing/Supported_Applications/Nextflow.md @@ -112,3 +112,21 @@ Nextflow provides tools that can assist you in making efficient use of the HPC r !!! warning "Nextflow plugins" nf-core pipelines expect to use nf-plugins in their base configuration. If you want to use these plugins, you will need to manually download them and store them in a plugin cache directory that you can specify with the `NXF_PLUGINS_DIR` environmental variable (as in the example `.sl` above) + +## UNTESTED Running with MPI + +There is a Nextflow plugin called nf-ignite which might be helpful for splitting large pipelines that only have small tasks. [More information here.](https://github.com/nextflow-io/nf-ignite?tab=readme-ov-file#slurm) + +## Comparison of submission methods + +!!! note "Small test run" + The following data come from a very small pipeline execution and needs to be repeated with a larger dataset for better information. + + +The nf-core rnaseq pipeline (v3.21.0) was run using the `test_full` dataset (v3.10) with three methods: a single Slurm job, a Slurm job capable of launching additional jobs, and a Slurm job split 4 ways by MPI. + +| Type of run | Number of cores / grid specification | Run time (hrs:mins:secs) | Total CPU | +|-----------------------|------------------------------------------|------------------------------|--------------------------------| +| Single Slurm job | 4 cores | 00:05:37 | 16:42.462 | +| Slurm job with additional | max\_nodes=1; cmds\_per\_node=4 | 00:06:13 | 17:16.920 | +| Slurm job with MPI | max\_nodes=3; cmds\_per\_node=4 | 00:09:41 | 42:18.727 | From ab07cb29c123c611f74c7c6a5b42bdee2a204b01 Mon Sep 17 00:00:00 2001 From: Jen Reeve Date: Thu, 13 Nov 2025 10:46:42 +1300 Subject: [PATCH 07/12] updates to benchmarking/mpi --- .../Supported_Applications/Nextflow.md | 84 +++++++++++++++++-- 1 file changed, 75 insertions(+), 9 deletions(-) diff --git a/docs/Scientific_Computing/Supported_Applications/Nextflow.md b/docs/Scientific_Computing/Supported_Applications/Nextflow.md index 7fa40f28b..2d7206b38 100644 --- a/docs/Scientific_Computing/Supported_Applications/Nextflow.md +++ b/docs/Scientific_Computing/Supported_Applications/Nextflow.md @@ -102,6 +102,31 @@ This will allow the main Nextflow Slurm job to submit this process to Slurm as s !!! warning "Avoid many short jobs" This will put a major burden on the Slurm scheduler for no improvement in your computational speed. Do not use the Nextflow `slurm` executor for jobs which take less than 30 minutes to complete. +## Running with MPI + +In addition to being able to use Nextflow to submit job arrays, you can also [run Nextflow across several nodes using MPI (Message-Passing Interface)](https://github.com/nextflow-io/nf-ignite?tab=readme-ov-file#slurm). + +```sl +#!/bin/bash -e + +#SBATCH --job-name=nextflow-mpi +#SBATCH --time=24:00:00 +#SBATCH --ntasks=3 +#SBATCH --cpus-per-task=12 +#SBATCH --tasks-per-node=1 +#SBATCH --mem=200GB +#SBATCH --account=nesi12345 +#SBATCH --output="%x-%j.out" + +# load nextflow +module load Nextflow/25.10.0 +export NXF_CLUSTER_SEED=$(shuf -i 0-16777216 -n 1) + +# run nextflow +srun nextflow run \ + -with-mpi +``` + ## Improving efficiency Nextflow provides tools that can assist you in making efficient use of the HPC resources. Running a pipeline with the CLI option `-with-report` will produce [a HTML execution report containing CPU and memory utilization information](https://www.nextflow.io/docs/latest/reports.html#execution-report) for each individual process as well as each process type. This information can be used to ensure processes are only getting the resources they need. @@ -113,20 +138,61 @@ Nextflow provides tools that can assist you in making efficient use of the HPC r !!! warning "Nextflow plugins" nf-core pipelines expect to use nf-plugins in their base configuration. If you want to use these plugins, you will need to manually download them and store them in a plugin cache directory that you can specify with the `NXF_PLUGINS_DIR` environmental variable (as in the example `.sl` above) -## UNTESTED Running with MPI - -There is a Nextflow plugin called nf-ignite which might be helpful for splitting large pipelines that only have small tasks. [More information here.](https://github.com/nextflow-io/nf-ignite?tab=readme-ov-file#slurm) - ## Comparison of submission methods -!!! note "Small test run" - The following data come from a very small pipeline execution and needs to be repeated with a larger dataset for better information. - - -The nf-core rnaseq pipeline (v3.21.0) was run using the `test_full` dataset (v3.10) with three methods: a single Slurm job, a Slurm job capable of launching additional jobs, and a Slurm job split 4 ways by MPI. +The [nf-core rnaseq pipeline (v3.21.0)](https://nf-co.re/rnaseq/3.21.0) was run using the `test_full` dataset (v3.10) with three methods: a single Slurm job, a Slurm job capable of launching additional jobs, and a Slurm job split 4 ways by MPI. | Type of run | Number of cores / grid specification | Run time (hrs:mins:secs) | Total CPU | |-----------------------|------------------------------------------|------------------------------|--------------------------------| | Single Slurm job | 4 cores | 00:05:37 | 16:42.462 | | Slurm job with additional | max\_nodes=1; cmds\_per\_node=4 | 00:06:13 | 17:16.920 | | Slurm job with MPI | max\_nodes=3; cmds\_per\_node=4 | 00:09:41 | 42:18.727 | + +### Local only + +Requested 1 task, 200GB memory, 16 CPUs per task, 12 hours +Duration : 11h 16m 47s +CPU hours : 319.6 + +```bash +nn_seff +Cluster: hpc +Job ID: 3034402 +State: ['COMPLETED'] +Cores: 16 +Tasks: 1 +Nodes: 1 +Job Wall-time: 94.0% 11:17:04 of 12:00:00 time limit +CPU Efficiency: 76.6% 5-18:19:39 of 7-12:33:04 core-walltime +Mem Efficiency: 53.8% 107.69 GB of 200.00 GB +``` + +### MPI + +Requested 3 tasks, 200GB memory, 12 CPUs per task, 24 hours. + +### Labeled processes + +Requested 1 task, 72GB memory, 12 CPUs per task, 12 hours. +Duration : 5h 19m 46s +CPU hours : 381.9 (1.4% failed) + +Labeled processes (list below) could submit via slurm array requesting 12 CPUs, 72GB memory with a max queue size of 10. + +- `TRIMMGALORE` +- `STAR_ALIGN_IGENOMES` +- `PICARD_MARKDUPLICATES` +- `QUALIMAP_RNASEQ` + +```bash +nn_seff +Cluster: hpc +Job ID: 3059728 +State: ['OUT_OF_MEMORY'] +Cores: 12 +Tasks: 1 +Nodes: 1 +Job Wall-time: 44.4% 05:20:01 of 12:00:00 time limit +CPU Efficiency: 79.4% 2-02:47:46 of 2-16:00:12 core-walltime +Mem Efficiency: 61.9% 44.55 GB of 72.00 GB +``` From b89ede41c59e7f3055a2dd5ffdbdbc5c6c576f0e Mon Sep 17 00:00:00 2001 From: Jen Reeve Date: Thu, 13 Nov 2025 11:54:42 +1300 Subject: [PATCH 08/12] fixing cylc ref --- docs/Scientific_Computing/Supported_Applications/Cylc.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/Scientific_Computing/Supported_Applications/Cylc.md b/docs/Scientific_Computing/Supported_Applications/Cylc.md index 99eeadff2..fc4e0ed34 100644 --- a/docs/Scientific_Computing/Supported_Applications/Cylc.md +++ b/docs/Scientific_Computing/Supported_Applications/Cylc.md @@ -20,8 +20,7 @@ running a monolithic, parallel executable is that each task will require less resources that the complete problem, it is thus easier for each task to slip into the queue and start running. -See the NeSI [Snakemake](https://snakemake-on-nesi.sschmeier.com/) and [Nextflow](https://nextflow.io/) pages -for other possible choices. +See the [Snakemake](https://snakemake-on-nesi.sschmeier.com/) and [Nextflow](https://nextflow.io/) for other possible choices. In this article, we show how you can create a simple workflow and run it on NeSI's platform. Consult the [Cylc From ec2a1a4372addab3ae9aba8a02857e2dde38ce19 Mon Sep 17 00:00:00 2001 From: Jen Reeve Date: Thu, 13 Nov 2025 11:55:39 +1300 Subject: [PATCH 09/12] cross references --- docs/Scientific_Computing/Supported_Applications/Cylc.md | 2 +- docs/Scientific_Computing/Supported_Applications/Nextflow.md | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/Scientific_Computing/Supported_Applications/Cylc.md b/docs/Scientific_Computing/Supported_Applications/Cylc.md index fc4e0ed34..acf4964d3 100644 --- a/docs/Scientific_Computing/Supported_Applications/Cylc.md +++ b/docs/Scientific_Computing/Supported_Applications/Cylc.md @@ -20,7 +20,7 @@ running a monolithic, parallel executable is that each task will require less resources that the complete problem, it is thus easier for each task to slip into the queue and start running. -See the [Snakemake](https://snakemake-on-nesi.sschmeier.com/) and [Nextflow](https://nextflow.io/) for other possible choices. +See [Snakemake](https://snakemake-on-nesi.sschmeier.com/) and [Nextflow](https://nextflow.io/) for other possible choices. In this article, we show how you can create a simple workflow and run it on NeSI's platform. Consult the [Cylc diff --git a/docs/Scientific_Computing/Supported_Applications/Nextflow.md b/docs/Scientific_Computing/Supported_Applications/Nextflow.md index 2d7206b38..2603cdc80 100644 --- a/docs/Scientific_Computing/Supported_Applications/Nextflow.md +++ b/docs/Scientific_Computing/Supported_Applications/Nextflow.md @@ -5,6 +5,8 @@ {% include "partials/app_header.html" %} [//]: <> (APPS PAGE BOILERPLATE END) +See [Snakemake](https://snakemake-on-nesi.sschmeier.com/) and [Cylc](https://cylc.github.io/) for other possible choices. + ## Configurations Below is an example `nextflow.config` file with some configuration settings that will help when running Nextflow via NeSI. From e7cb3d7b7a1d5399c1de3365b8488ffac928e7ea Mon Sep 17 00:00:00 2001 From: Jen Reeve Date: Thu, 20 Nov 2025 11:29:37 +1300 Subject: [PATCH 10/12] editing configs etc --- .../Supported_Applications/Nextflow.md | 269 ++++++++++++++---- 1 file changed, 207 insertions(+), 62 deletions(-) diff --git a/docs/Scientific_Computing/Supported_Applications/Nextflow.md b/docs/Scientific_Computing/Supported_Applications/Nextflow.md index 2603cdc80..3d4b54207 100644 --- a/docs/Scientific_Computing/Supported_Applications/Nextflow.md +++ b/docs/Scientific_Computing/Supported_Applications/Nextflow.md @@ -7,41 +7,25 @@ See [Snakemake](https://snakemake-on-nesi.sschmeier.com/) and [Cylc](https://cylc.github.io/) for other possible choices. -## Configurations - -Below is an example `nextflow.config` file with some configuration settings that will help when running Nextflow via NeSI. +## Methods for running Nextflow with REANNZ -```json -// enable use of apptainer to run containers -apptainer { - apptainer.pullTimeout = '1h' -} - -// Default settings for all profiles -process { - stageInMode = 'symlink' - cache = 'lenient' -} +There are three suggested methods for running Nextflow pipelines on our system: -// Specific profiles to use in different contexts -profiles { - debug { - cleanup = false - } - local { - executor = 'local' - } -} +1. [Running Nextflow in an interactive Slurm session](#running-nextflow-in-an-interactive-slurm-session) + - This method is best for setting up or debugging pipeline executions as the pipeline will end as soon as the interactive session ends. +2. [Submitting a Nextflow workflow as a batch job](#submitting-a-nextflow-workflow-as-a-batch-job) + - This method will run all sub-processes in the same Slurm job. This is best if your workflow would spawn a large number of short jobs. +3. [Submitting a Nextflow workflow via a head job](#submitting-a-nextflow-workflow-via-a-head-job) + - This method requires submitting a low resource but long running batch job which will control the Nextflow workflow and all processes will be submitted by Nextflow as separate jobs to Slurm. This method is useful for workflows with lots of variation in their computational needs and which comprise mostly long running processes. -cleanup = true -``` +The differences in these methods is largely controlled by configuration/profile settings. The examples below use the REANNZ configuration file provided in the [Configurations section](#configurations). -## Running Nextflow in an interactive Slurm session +### Running Nextflow in an interactive Slurm session Running Nextflow in an interactive session can be especially helpful when setting up or debugging a pipeline. To do so, request an interactive session with appropriate resources for the pipeline: ```bash -srun --account nesi12345 --job-name "InteractiveJob" --cpus-per-task 8 --mem-per-cpu 1500 --time 24:00:00 --pty bash +srun --account nesi12345 --job-name "InteractiveJob" --cpus-per-task 16 --mem-per-cpu 24000 --time 24:00:00 --pty bash ``` Once your interactive session has launched, load the Nextflow module with `module load Nextflow/25.10.1` (or your required version) and proceed. Parallelization of Nextflow processes will occur within this job. @@ -50,61 +34,68 @@ There are several environmental variables that can be helpful to avoid loading c ```bash export NXF_APPTAINER_CACHEDIR=/nesi/nobackup/nesi12345/apptainer_cache -export NXF_PLUGINS_DIR=/nesi/project/nesi12345/user/PROJECT_DIR/.nextflow/plugins +export NXF_PLUGINS_DIR=/nesi/project/nesi12345/.nextflow/plugins ``` You can confirm that Nextflow is loaded and ready using the `nextflow run hello` command to test Nextflow's Hello World pipeline. -## Submitting a Nextflow workflow as a batch job +### Submitting a Nextflow workflow as a batch job The following `sbatch` will submit a Nextflow workflow as a Slurm job. All of the Nextflow processes will run on the same compute node, so request enough resources to run the most intensive process in the workflow and enough time for the entire workflow to complete. ```sl #!/bin/bash -e -#SBATCH --job-name my job +#SBATCH --job-name nextflow-workflow #SBATCH --account nesi12345 -#SBATCH --time 01:00:00 -#SBATCH --mem 2G +#SBATCH --time 12:00:00 +#SBATCH --mem 24G +#SBATCH --cpus-per-task 16 # load Nextflow and set environmental variables module load Nextflow/25.10.0 export NXF_APPTAINER_CACHEDIR=/nesi/nobackup/nesi12345/apptainer_cache export NXF_OFFLINE='true' -export NXF_PLUGINS_DIR=/nesi/project/nesi12345/user/PROJECT_DIR/.nextflow/plugins +export NXF_PLUGINS_DIR=/nesi/project/nesi12345/.nextflow/plugins # run nextflow nextflow run NEXTFLOW_WORKFLOW \ -profile local,apptainer \ - --outdir /nesi/project/nesi12345/user/PROJECT_DIR/NEXTFLOW_WORKFLOW/out \ - -w /nesi/nobackup/nesi12345/user/PROJECT_DIR/NEXTFLOW_WORKFLOW/work \ - -with-trace \ - -with-report \ - -with-dag + --outdir /nesi/project/nesi12345/NEXTFLOW_WORKFLOW/out \ + -w /nesi/nobackup/nesi12345/NEXTFLOW_WORKFLOW/work ``` -## Time/compute intensive processes +### Submitting a Nextflow workflow via a head job -If any of your individual processes regularly take longer than 30 minutes, you can flag them to be submitted as additional Slurm jobs separate from the head job. To do this, give these processes a [label](https://www.nextflow.io/docs/latest/reference/process.html#label) such as `'slurm_array'` and add the following to your `nextflow.config` file: +The following `sbatch` script will request the resources to run a Nextflow head job which will then submit processes to Slurm as separate tasks. Beyond the resources requested for this job, the only difference between this script and the previous one is the change in the `-profile` tag from `local,apptainer` to `slurm,apptainer`. -```json -process { - withLabel: slurm_array { - executor = 'slurm' - queue = 'genoa,milan' - queueSize = 10 - jobName = { "$task.name - $task.hash" } - slurm_array = 100 - } -} -``` +```sl +#!/bin/bash -e + +#SBATCH --job-name nextflow-head +#SBATCH --account nesi12345 +#SBATCH --time 12:00:00 +#SBATCH --mem 4G +#SBATCH --cpus-per-task 4 -This will allow the main Nextflow Slurm job to submit this process to Slurm as separate tasks. `queueSize` limits the number of these additional Slurm jobs that can be queued simultaneously, while `array` tells Nextflow to group up to 100 processes with label and submit them as a job array. You can provide more specific labels and set up the configuration to request different resources for different processes. + +# load Nextflow and set environmental variables +module load Nextflow/25.10.0 +export NXF_APPTAINER_CACHEDIR=/nesi/nobackup/nesi12345/apptainer_cache +export NXF_OFFLINE='true' +export NXF_PLUGINS_DIR=/nesi/project/nesi12345/.nextflow/plugins + +# run nextflow +nextflow run NEXTFLOW_WORKFLOW \ + -profile slurm,apptainer \ + --outdir /nesi/project/nesi12345/NEXTFLOW_WORKFLOW/out \ + -w /nesi/nobackup/nesi12345/NEXTFLOW_WORKFLOW/work +``` !!! warning "Avoid many short jobs" This will put a major burden on the Slurm scheduler for no improvement in your computational speed. Do not use the Nextflow `slurm` executor for jobs which take less than 30 minutes to complete. -## Running with MPI + + +## Configurations + +For reproducibility and clarity, we recommend using the ability to [stack Nextflow configurations](https://nf-co.re/docs/usage/getting_started/configuration#different-config-locations) and use three distinct `.config` files. + +1. Pipeline-level config: This configuration is system and data agnostic, but should be untouched for any runs of the given pipeline. +2. System-level config: This configuration is pipeline agnostic but provides settings for running on a given computer system. We provide a REANNZ-specific config below. +3. Run-level config: This configuration is where changes are made to fine-tune for the specifics of a given run/system/pipeline combination. For clarity, we will refer to this file as `custom.config` + +Below is an example `nextflow.config` file with some configuration settings that will help when running Nextflow via REANNZ systems. + +```json +// REANNZ nf-core configuration profile +// Global default params, used in configs +params { + config_profile_description = 'REANNZ HPC profile provided by nf-core/configs' + config_profile_contact = 'Jennifer Reeve (@jen-reeve)' + config_profile_url = 'https://docs.nesi.org.nz' + max_cpus = 64 + max_memory = 1024.GB +} + +// Default settings for all profiles +process { + stageInMode = 'symlink' + cache = 'lenient' +} + +// Specific profiles to use in different contexts +profiles { + debug { + // This profile will not remove intermediate files from the work directory + cleanup = false + } + local { + // This profile should be used workflows submitted from interactive Slurm sessions or when a workflow will generate many short (<30 minutes) tasks + process.executor = 'local' + } + slurm { + // This profile should be used for workflows which need to submit processes as Slurm jobs + process.executor = 'slurm' + process.array = 100 + } +} + +// Settings for the Slurm executor +executor { + '$slurm' { + queueSize = 500 + submitRateLimit = '20 min' + // 20 per minute + pollInterval = '30 sec' + queueStatInterval = '30 sec' + jobName = { "${task.process}-${task.hash}" } + queue = 'genoa,milan' + } +} + +// Apptainer specific settings +apptainer { + apptainer.pullTimeout = '2h' +} + +cleanup = true +``` + +### Run-level configuration options + +As mentioned above, the pipeline and system specific configuration files should generally be left untouched. This means any adjustments for your workflow will occur in final run level configuration file. + +There are many options you may wish to use to fine-tune your Nextflow runs. For more information, we recommend starting with the [overview on configuration files](https://www.nextflow.io/docs/latest/config.html) and if needed digging into the [configuration reference](https://www.nextflow.io/docs/latest/reference/config.html) available in the Nextflow documentation. + +One option of particular note is the ability to flag certain processes to use a different executor than the main workflow. This can allow you to either flag certain processes to be submitted as separate jobs despite generally [running a workflow in a single Slurm job](#submitting-a-nextflow-workflow-as-a-batch-job) or to flag processes to run in the [head job while most processes are submitted as separate jobs](#submitting-a-nextflow-workflow-via-a-head-job). + +For example, if you submit a workflow as a batch job, but know that several of your individual processes regularly take longer than 30 minutes, you can flag them to be submitted as additional Slurm jobs separate from the head job. To do this, give these processes a [label](https://www.nextflow.io/docs/latest/reference/process.html#label) such as `'slurm_array'` and add the following to your `custom.config` file: + +```json +process { + withLabel: slurm_array { + executor = 'slurm' + } +} ``` -## Improving efficiency +You could additionally provide [details about the specific resources required](https://nf-co.re/docs/usage/getting_started/configuration#tuning-workflow-resources), although this may already be provided by the pipeline level configuration via additional process labels or explicit definitions [using `withName`](https://www.nextflow.io/docs/latest/config.html#process-selectors). Please note that there is an [additional ranking of priority for process configuration settings](https://www.nextflow.io/docs/latest/config.html#selector-priority) beyond that of the layered configuration files. + +## Monitoring and reporting + +Nextflow provides tools that can assist you in making efficient use of the HPC resources. We strongly recommend testing your pipeline with a small subset of data to determine optimal settings before running full datasets. + +The most human-readable, but least configurable option is the execution report which is [an HTML execution report containing CPU and memory utilization information](https://www.nextflow.io/docs/latest/reports.html#execution-report) for each individual process as well as each process type. This information can be used to ensure processes are only getting the resources they need. -Nextflow provides tools that can assist you in making efficient use of the HPC resources. Running a pipeline with the CLI option `-with-report` will produce [a HTML execution report containing CPU and memory utilization information](https://www.nextflow.io/docs/latest/reports.html#execution-report) for each individual process as well as each process type. This information can be used to ensure processes are only getting the resources they need. +For a more configurable option, [the trace file](https://www.nextflow.io/docs/latest/reports.html#trace-file) provides many potential fields of interest which can be requested. A full list of fields is available at the previous documentation link, but several of note for optimization and debugging: + +- `native_id` will provide the job ID for any jobs submitted to Slurm +- `duration` will show the time from *submission* of the process to completion of the process +- `realtime` will show the time from the *start* of the process to completion of the process (job run time) +- `%cpu` will show the percentage of CPU used by the process +- `%mem` will sow the percentage of memory used by the process +- `peak_rss` will show the peak of real memory used +- `workdir` will provide the path to the working directory of the process + +Adding the following to your `custom.config` will produce both an execution report and trace file for each run, named with the timestamp, and put these files in a separate `runInfo` directory rather than within the Nextflow output directory. + +```json +// Name the reports according to when they were run +params.timestamp = new java.util.Date().format('yyyy-MM-dd_HH-mm-ss') + +// Generate report-timestamp.html execution report +report { + enabled = true + overwrite = false + file = "./runInfo/report-${params.timestamp}.html" +} + +// Generate trace-timestamp.txt trace file +trace { + enabled = true + overwrite = false + file = "./runInfo/trace-${params.timestamp}.txt" + fields = 'name,status,exit,duration,realtime,cpus,%cpu,memory,%mem,rss,peak_rss,workdir,native_id' +} +``` ## Community pipelines - nf-core @@ -142,18 +252,22 @@ Nextflow provides tools that can assist you in making efficient use of the HPC r ## Comparison of submission methods -The [nf-core rnaseq pipeline (v3.21.0)](https://nf-co.re/rnaseq/3.21.0) was run using the `test_full` dataset (v3.10) with three methods: a single Slurm job, a Slurm job capable of launching additional jobs, and a Slurm job split 4 ways by MPI. +The [nf-core rnaseq pipeline (v3.21.0)](https://nf-co.re/rnaseq/3.21.0) was run using the `test_full` dataset (v3.10) with three methods: a batch job, a head job, and a batch job with specific processes flagged for submission to Slurm. | Type of run | Number of cores / grid specification | Run time (hrs:mins:secs) | Total CPU | |-----------------------|------------------------------------------|------------------------------|--------------------------------| -| Single Slurm job | 4 cores | 00:05:37 | 16:42.462 | -| Slurm job with additional | max\_nodes=1; cmds\_per\_node=4 | 00:06:13 | 17:16.920 | -| Slurm job with MPI | max\_nodes=3; cmds\_per\_node=4 | 00:09:41 | 42:18.727 | +| [Batch job](#batch-job) | | 00:05:37 | 16:42.462 | +| [Head job](#head-job) | max\_nodes=1; cmds\_per\_node=4 | 00:06:13 | 17:16.920 | +| [Labeled processes](#labeled-processes) | max\_nodes=3; cmds\_per\_node=4 | 00:09:41 | 42:18.727 | + +### Batch job -### Local only +Using the [batch job method](#submitting-a-nextflow-workflow-as-a-batch-job). Requested 1 task, 200GB memory, 16 CPUs per task, 12 hours + Duration : 11h 16m 47s + CPU hours : 319.6 ```bash @@ -169,14 +283,19 @@ CPU Efficiency: 76.6% 5-18:19:39 of 7-12:33:04 core-walltime Mem Efficiency: 53.8% 107.69 GB of 200.00 GB ``` -### MPI +### Head job + +Using the [head job method](#submitting-a-nextflow-workflow-via-a-head-job) -Requested 3 tasks, 200GB memory, 12 CPUs per task, 24 hours. ### Labeled processes +Using the [batch job method](#submitting-a-nextflow-workflow-as-a-batch-job) with specific processes flagged for submission to the Slurm queue as described in the [run-level configuration options section](#run-level-configuration-options) + Requested 1 task, 72GB memory, 12 CPUs per task, 12 hours. + Duration : 5h 19m 46s + CPU hours : 381.9 (1.4% failed) Labeled processes (list below) could submit via slurm array requesting 12 CPUs, 72GB memory with a max queue size of 10. @@ -198,3 +317,29 @@ Job Wall-time: 44.4% 05:20:01 of 12:00:00 time limit CPU Efficiency: 79.4% 2-02:47:46 of 2-16:00:12 core-walltime Mem Efficiency: 61.9% 44.55 GB of 72.00 GB ``` + From a7a797f65939b50b4832cfe016ad896d5cab9519 Mon Sep 17 00:00:00 2001 From: Jen Reeve Date: Thu, 20 Nov 2025 11:30:22 +1300 Subject: [PATCH 11/12] local file issue --- .../Supported_Applications/Nextflow.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/Scientific_Computing/Supported_Applications/Nextflow.md b/docs/Scientific_Computing/Supported_Applications/Nextflow.md index 3d4b54207..9c5ee8171 100644 --- a/docs/Scientific_Computing/Supported_Applications/Nextflow.md +++ b/docs/Scientific_Computing/Supported_Applications/Nextflow.md @@ -256,9 +256,9 @@ The [nf-core rnaseq pipeline (v3.21.0)](https://nf-co.re/rnaseq/3.21.0) was run | Type of run | Number of cores / grid specification | Run time (hrs:mins:secs) | Total CPU | |-----------------------|------------------------------------------|------------------------------|--------------------------------| -| [Batch job](#batch-job) | | 00:05:37 | 16:42.462 | -| [Head job](#head-job) | max\_nodes=1; cmds\_per\_node=4 | 00:06:13 | 17:16.920 | -| [Labeled processes](#labeled-processes) | max\_nodes=3; cmds\_per\_node=4 | 00:09:41 | 42:18.727 | +| [Batch job](#batch-job) | | | | +| [Head job](#head-job) | | | | +| [Labeled processes](#labeled-processes) | | | | ### Batch job @@ -271,7 +271,7 @@ Duration : 11h 16m 47s CPU hours : 319.6 ```bash -nn_seff +> nn_seff Cluster: hpc Job ID: 3034402 State: ['COMPLETED'] From 693b33e05be87295ba32525afcad3d5babc5c6eb Mon Sep 17 00:00:00 2001 From: Jen Reeve Date: Mon, 1 Dec 2025 11:16:33 +1300 Subject: [PATCH 12/12] commenting out the benchmarking to fix later --- docs/Scientific_Computing/Supported_Applications/Nextflow.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/Scientific_Computing/Supported_Applications/Nextflow.md b/docs/Scientific_Computing/Supported_Applications/Nextflow.md index 9c5ee8171..77f29428d 100644 --- a/docs/Scientific_Computing/Supported_Applications/Nextflow.md +++ b/docs/Scientific_Computing/Supported_Applications/Nextflow.md @@ -250,7 +250,7 @@ trace { !!! warning "Nextflow plugins" nf-core pipelines expect to use nf-plugins in their base configuration. If you want to use these plugins, you will need to manually download them and store them in a plugin cache directory that you can specify with the `NXF_PLUGINS_DIR` environmental variable (as in the example `.sl` above) -## Comparison of submission methods +