nf-probe

A simple Nextflow pipeline to validate your AWS infrastructure setup before running real workloads.

What it tests

Test	What it validates	How it fails
Pipeline submission	EC2 head node → AWS Batch communication, IAM instance role	Pipeline won't start at all
S3 access	Read permissions on your S3 bucket (file staging)	CHECK_S3_ACCESS fails with access denied
NVMe disk	Local instance storage is mounted and usable beyond 30 GB	CHECK_NVME_DISK fails at the 35 GB step
Wave containers	Ability to pull containers from community.wave.seqera.io	CHECK_CONTAINER fails to start

Quick start

# From your EC2 head node:
nextflow run main.nf -params-file run_params.json

This repository includes run_params.json as a public template with placeholder values. Copy it to a local params file and fill in your environment-specific values.

Before running, make sure the s3_test_file value points to a real, readable object in your bucket.

Final output files are written to the directory where you launch Nextflow. S3 write-path validation is covered by workDir because task/intermediate files are written there.

Optional overrides

You can still override any value from the params file at runtime:

nextflow run main.nf \
    -params-file run_params.json \
    --runName nf-probe-$(date +%Y%m%d-%H%M%S)

Parameters

Parameter	Default	Description
--s3_test_file	value in run_params.json	Path to any file in S3 to test read permissions
--disk_test_sizes	[5, 10, 20, 35, 50]	List of file sizes in GB to write progressively
--outdir	launch directory	Where to publish the test report (local path)
--aws_region	value in run_params.json	AWS region for AWS Batch
--aws_queue	value in run_params.json	AWS Batch queue name
--jobRole	value in run_params.json	IAM role used by AWS Batch jobs
--workDir	value in run_params.json	Base S3 work directory (workDir/runName) used for S3 write validation
--runName	nf-probe	Suffix used under workDir for this run

Understanding the results

All tests pass ✅

Your AWS infrastructure is correctly configured:

• EC2 can submit jobs to AWS Batch
• S3 read/write permissions are working
• Instances have NVMe local storage mounted
• Wave container registry is accessible

CHECK_NVME_DISK fails at 35 GB ❌

Your Batch compute instances don't have local NVMe storage mounted. The default 30 GB gp3 EBS root volume ran out of space. Fix by:

• Using instance types with local NVMe (e.g., c5d, m5d, r5d families)
• Configuring your Seqera compute environment to use instance types with local storage
• Mounting the NVMe drives in your launch template

CHECK_S3_ACCESS fails ❌

The IAM role attached to your Batch compute instances doesn't have permission to read from the specified S3 bucket. Check your instance role policies.

CHECK_CONTAINER fails ❌

Your instances can't reach community.wave.seqera.io. Check:

• VPC has internet access (NAT gateway or public subnet)
• Security groups allow outbound HTTPS (port 443)
• No proxy or firewall blocking container registry traffic

Customizing disk test sizes

To test different disk capacities, override the default list:

nextflow run main.nf \
    -params-file run_params.json \
    --disk_test_sizes '[10, 50, 100, 200]' \
    --runName nf-probe-large-disk

Requirements

• Nextflow >= 24.04
• AWS Batch compute environment configured
• Docker enabled on compute instances