diff --git a/.github/workflows/awsfulltest.yml b/.github/workflows/awsfulltest.yml index 4190c08..5a8075d 100644 --- a/.github/workflows/awsfulltest.yml +++ b/.github/workflows/awsfulltest.yml @@ -24,8 +24,6 @@ jobs: - name: Launch workflow via Seqera Platform uses: seqeralabs/action-tower-launch@51565b514bff1827cf34620de25d0055759f1fc9 # v2 - # Add full size test data (but still relatively small datasets for few samples) - # on the `test_full.config` test runs with only one set of parameters with: workspace_id: ${{ vars.TOWER_WORKSPACE_ID }} access_token: ${{ secrets.TOWER_ACCESS_TOKEN }} diff --git a/CHANGELOG.md b/CHANGELOG.md index c11b24c..70658fa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,28 +9,9 @@ Initial release of nf-core/datasync, created with the [nf-core](https://nf-co.re ### `Added` -- [[#87](https://github.com/nf-core/datasync/pull/87)] - Add local test profile ([@delfiterradas](https://github.com/delfiterradas), review by [@atrigila](https://github.com/atrigila)). -- [[#79](https://github.com/nf-core/datasync/pull/79)] - Add exit status for `RCLONE` modules in multiqc report ([@delfiterradas](https://github.com/delfiterradas), review by [@atrigila](https://github.com/atrigila)). -- [[#74](https://github.com/nf-core/datasync/pull/74)] - Implement warning and `--download` parameter when working with remote directories and SHA checksums ([@delfiterradas](https://github.com/delfiterradas), review by [@atrigila](https://github.com/atrigila), [@apeltzer](https://github.com/apeltzer) and [@antoniasaracco](https://github.com/antoniasaracco)). -- [[#67](https://github.com/nf-core/datasync/pull/67)] - Copy only files which are successfully validated ([@delfiterradas](https://github.com/delfiterradas), review by [@atrigila](https://github.com/atrigila) and [@antoniasaracco](https://github.com/antoniasaracco)). -- [[#57](https://github.com/nf-core/datasync/pull/57)] - Add nf-tests with edge cases([@atrigila](https://github.com/atrigila), review by [@delfiterradas](https://github.com/delfiterradas)). -- [[#55](https://github.com/nf-core/datasync/pull/55)] - Add subdirectories with sample id to results. Improvements to MultiQC report (width of columns, sections and sub-section headers)colors to multiqc results and display first errors in tables ([@atrigila](https://github.com/atrigila), review by [@delfiterradas](https://github.com/delfiterradas)). -- [[#54](https://github.com/nf-core/datasync/pull/54)] - Add colors to multiqc results and display first errors in tables ([@atrigila](https://github.com/atrigila), review by [@delfiterradas](https://github.com/delfiterradas)). -- [[#31](https://github.com/nf-core/datasync/pull/31)] - Compute checksum for each input file ([@delfiterradas](https://github.com/delfiterradas), review by [@atrigila](https://github.com/atrigila)). -- [[#35](https://github.com/nf-core/datasync/pull/35)] - Compare generated checksum to given checksum in input and update test profiles ([@delfiterradas](https://github.com/delfiterradas), review by [@atrigila](https://github.com/atrigila)). -- [[#46](https://github.com/nf-core/datasync/pull/46)] - Import Rclone module from nf-core([@antoniasaracco](https://github.com/antoniasaracco), review by [@delfiterradas](https://github.com/delfiterradas)). -- [[#41](https://github.com/nf-core/datasync/pull/41)] - Generate MultiQC Report with comparechecksum tables and input samplesheet ([@delfiterradas](https://github.com/delfiterradas), review by [@atrigila](https://github.com/atrigila)). -- [[#49](https://github.com/nf-core/datasync/pull/49)] - Install `RCLONE_CHECK` and `RCLONE_CHECKSUM` modules from nf-core ([@delfiterradas](https://github.com/delfiterradas), review by [@atrigila](https://github.com/atrigila) and [@antoniasaracco](https://github.com/antoniasaracco)). -- [[#59](https://github.com/nf-core/datasync/pull/59)] - Create pipeline documentation ([@antoniasaracco](https://github.com/antoniasaracco), review by [@atrigila](https://github.com/atrigila) and [@delfiterradas](https://github.com/delfiterradas)). - -### `Fixed` - -- [[#84](https://github.com/nf-core/datasync/pull/84)] - Update `RCLONE_` modules, enhance source and destination path handling, add local `create_filter_list` module and other small fixes ([@delfiterradas](https://github.com/delfiterradas), review by [@atrigila](https://github.com/atrigila)). -- [[#83](https://github.com/nf-core/datasync/pull/83)] - Add apptainer version to avoid error in CI test ([@delfiterradas](https://github.com/delfiterradas), review by [@atrigila](https://github.com/atrigila)). -- [[#78](https://github.com/nf-core/datasync/pull/78)] - Fixed linting issues reported by nf-core and Nextflow, and updated `rclone` modules and nf-tests to sort generated report files for deterministic snapshots ([@atrigila](https://github.com/atrigila), review by [@delfiterradas](https://github.com/delfiterradas) and [@antoniasaracco](https://github.com/antoniasaracco)). -- [[#76](https://github.com/nf-core/datasync/pull/76)] - Remove `--one-way` from rclone/checksum confi ([@delfiterradas](https://github.com/delfiterradas), review by [@atrigila](https://github.com/atrigila) and [@antoniasaracco](https://github.com/antoniasaracco)). -- [[#73](https://github.com/nf-core/datasync/pull/73)] - Create tmp `files_to_copy.tx` avoiding error with write permissions in work directory ([@delfiterradas](https://github.com/delfiterradas), review by [@atrigila](https://github.com/atrigila) and [@antoniasaracco](https://github.com/antoniasaracco)). - -### `Dependencies` - -### `Deprecated` +- Samplesheet-driven copying of files and directories between local paths and rclone-supported object storage locations. +- Validation of source data against supplied MD5 and SHA-256 checksum manifests before transfer. +- The ability to copy only files that pass checksum validation, and to download remote files when SHA-256 verification is required. +- Post-transfer comparison of copied data against the source, with detailed rclone status files for each sample. +- A MultiQC report covering the input samplesheet, validation summary, checksum validation, and post-transfer checks. +- A local test profile to explore the pipeline and its outputs. diff --git a/README.md b/README.md index 181e543..96f62ea 100644 --- a/README.md +++ b/README.md @@ -9,14 +9,12 @@ [![GitHub Actions CI Status](https://github.com/nf-core/datasync/actions/workflows/nf-test.yml/badge.svg)](https://github.com/nf-core/datasync/actions/workflows/nf-test.yml) [![GitHub Actions Linting Status](https://github.com/nf-core/datasync/actions/workflows/linting.yml/badge.svg)](https://github.com/nf-core/datasync/actions/workflows/linting.yml)[![AWS CI](https://img.shields.io/badge/CI%20tests-full%20size-FF9900?labelColor=000000&logo=Amazon%20AWS)](https://nf-co.re/datasync/results)[![Cite with Zenodo](http://img.shields.io/badge/DOI-10.5281/zenodo.XXXXXXX-1073c8?labelColor=000000)](https://doi.org/10.5281/zenodo.XXXXXXX) [![nf-test](https://img.shields.io/badge/unit_tests-nf--test-337ab7.svg)](https://www.nf-test.com) - [![Nextflow](https://img.shields.io/badge/version-%E2%89%A525.10.4-green?style=flat&logo=nextflow&logoColor=white&color=%230DC09D&link=https%3A%2F%2Fnextflow.io)](https://www.nextflow.io/) [![nf-core template version](https://img.shields.io/badge/nf--core_template-4.1.0-green?style=flat&logo=nfcore&logoColor=white&color=%2324B064&link=https%3A%2F%2Fnf-co.re)](https://github.com/nf-core/tools/releases/tag/4.1.0) [![run with conda](http://img.shields.io/badge/run%20with-conda-3EB049?labelColor=000000&logo=anaconda)](https://docs.conda.io/en/latest/) [![run with docker](https://img.shields.io/badge/run%20with-docker-0db7ed?labelColor=000000&logo=docker)](https://www.docker.com/) [![run with singularity](https://img.shields.io/badge/run%20with-singularity-1d355c.svg?labelColor=000000)](https://sylabs.io/docs/) [![Launch on Seqera Platform](https://img.shields.io/badge/Launch%20%F0%9F%9A%80-Seqera%20Platform-%234256e7)](https://cloud.seqera.io/launch?pipeline=https://github.com/nf-core/datasync) - [![Get help on Slack](http://img.shields.io/badge/slack-nf--core%20%23datasync-4A154B?labelColor=000000&logo=slack)](https://nfcore.slack.com/channels/datasync)[![Follow on Bluesky](https://img.shields.io/badge/bluesky-%40nf__core-1185fe?labelColor=000000&logo=bluesky)](https://bsky.app/profile/nf-co.re)[![Follow on Mastodon](https://img.shields.io/badge/mastodon-nf__core-6364ff?labelColor=FFFFFF&logo=mastodon)](https://mstdn.science/@nf_core)[![Watch on YouTube](http://img.shields.io/badge/youtube-nf--core-FF0000?labelColor=000000&logo=youtube)](https://www.youtube.com/c/nf-core) ## Introduction @@ -34,7 +32,7 @@ The current tested use case for this pipeline is transfer between S3 buckets. Pass an `rclone` configuration with `--rclone_config` whenever a source or destination URI needs credentials or provider settings. Samplesheet paths use local paths or standard URIs such as `s3://bucket/path`, not rclone's `remote:path` syntax. For non-S3 layouts, design and validate the provider-specific configuration using the upstream [rclone documentation](https://rclone.org/docs/). -![nf-core/datasync metro map](docs/images/datasync-metromap.png) +![nf-core/datasync metro map](docs/images/datasync_nf-metro.svg) ## Quick start diff --git a/assets/samplesheet.csv b/assets/samplesheet.csv index 6d48fd6..9abae2c 100644 --- a/assets/samplesheet.csv +++ b/assets/samplesheet.csv @@ -1,2 +1,2 @@ sample,input,output_path,checksum_md5 -sample,path/to/file,output_path,path/to/file +sample,s3://foo/path/to/files/,output_path/data,path/to/checksum.md5 diff --git a/assets/samplesheet_full.csv b/assets/samplesheet_full.csv deleted file mode 100644 index 9ed843e..0000000 --- a/assets/samplesheet_full.csv +++ /dev/null @@ -1,2 +0,0 @@ -sample,input,checksum_md5,checksum_sha -demultiplex,s3://nf-core-awsmegatests/demultiplex/results-fbec8e442f0599f8b74876e62263af05b9a41d33/,checksum_md5.tsv,checksum_sha.tsv diff --git a/bin/create_test_files.sh b/bin/create_test_files.sh deleted file mode 100644 index ddb0120..0000000 --- a/bin/create_test_files.sh +++ /dev/null @@ -1,8 +0,0 @@ -#!/bin/bash -# This script expects a single folder $1 with several subfolders that then need to be synchronized and checksummed to another location $2 -mkdir -p $1 -mkdir -p $2 -touch $1/DemuxDone #Note this needs to be configured as a pattern to look for in the pipeline configuration, otherwise this trigger won't work. -touch $1/fake_file.fastq.gz -touch $1/fake_file2.fastq.gz -echo "Sync me to another place" > $1/SampleSheet.csv diff --git a/conf/igenomes.config b/conf/igenomes.config deleted file mode 100644 index e69de29..0000000 diff --git a/conf/igenomes_ignored.config b/conf/igenomes_ignored.config deleted file mode 100644 index e69de29..0000000 diff --git a/docs/images/datasync-metromap.png b/docs/images/datasync-metromap.png deleted file mode 100644 index 108bbc0..0000000 Binary files a/docs/images/datasync-metromap.png and /dev/null differ diff --git a/docs/images/datasync-nf-metro.mmd b/docs/images/datasync-nf-metro.mmd new file mode 100644 index 0000000..c4558dd --- /dev/null +++ b/docs/images/datasync-nf-metro.mmd @@ -0,0 +1,33 @@ +%%metro title: nf-core/datasync +%%metro mode: dark +%%metro style: nfcore +%%metro diamond_style: symmetric +%%metro line: main | Pipeline | #4CAF50 + +%%metro file: ref_in | CSV | Input\nSamplesheet | banner +%%metro file: reads_in | Config | Rclone\nConfig | banner +%%metro file: report_out | HTML | Report | banner +%%metro grid: datasync | 0,0 + +graph LR + subgraph datasync [Datasync pipeline] + ref_in[Input] + reads_in[ ] + node1[Rclone\nChecksum] + node2[Create\nFilter List] + node3[Rclone\nCheck] + node4[Rclone\nCopy] + node5[MultiQC] + Rclone\nChecksum[Rclone\nChecksum] + ref_in -->|main| Rclone\nChecksum + reads_in -->|main| Rclone\nChecksum + report_out[Output] + ref_in -->|main| node1 + reads_in -->|main| node1 + node1 -->|main| node2 + node2 -->|main| node4 + node4 -->|main| node3 + node3 -->|main| node5 + node1 -->|main| node4 + node5 -->|main| report_out + end diff --git a/docs/images/datasync_nf-metro.svg b/docs/images/datasync_nf-metro.svg new file mode 100644 index 0000000..62f95c3 --- /dev/null +++ b/docs/images/datasync_nf-metro.svg @@ -0,0 +1,98 @@ + + + + +{"groups":[{"color":"#4CAF50","id":"main","label":"Pipeline"}],"height":419,"match":{"flags":"i","target":"fqProcessName","type":"regex"},"nodes":[{"groups":["main"],"h":10.0,"id":"ref_in","label":"Input","patterns":[],"r":5.0,"region":"datasync","rx":5.0,"w":10.0,"x":80.0,"y":133.6},{"groups":["main"],"h":10.0,"id":"reads_in","label":"reads_in","patterns":[],"r":5.0,"region":"datasync","rx":5.0,"w":10.0,"x":80.0,"y":267.2},{"groups":["main"],"h":10.0,"id":"node1","label":"Rclone\nChecksum","patterns":[],"r":5.0,"region":"datasync","rx":5.0,"w":10.0,"x":176.0,"y":200.4},{"groups":["main"],"h":10.0,"id":"node2","label":"Create\nFilter List","patterns":[],"r":5.0,"region":"datasync","rx":5.0,"w":10.0,"x":236.0,"y":167.0},{"groups":["main"],"h":10.0,"id":"node3","label":"Rclone\nCheck","patterns":[],"r":5.0,"region":"datasync","rx":5.0,"w":10.0,"x":356.0,"y":200.4},{"groups":["main"],"h":10.0,"id":"node4","label":"Rclone\nCopy","patterns":[],"r":5.0,"region":"datasync","rx":5.0,"w":10.0,"x":296.0,"y":200.4},{"groups":["main"],"h":10.0,"id":"node5","label":"MultiQC","patterns":[],"r":5.0,"region":"datasync","rx":5.0,"w":10.0,"x":416.0,"y":200.4},{"groups":["main"],"h":10.0,"id":"report_out","label":"Output","patterns":[],"r":5.0,"region":"datasync","rx":5.0,"w":10.0,"x":476.0,"y":200.4}],"regions":[{"id":"datasync","label":"Datasync pipeline"}],"title":"nf-core/datasync","version":"1.0","width":594} + + +nf-core/datasync + + +1 +Datasync pipeline + + + + + + + + + + + + + + + +CSV +InputSamplesheet + + + + + + + + + +Config +RcloneConfig + + + + + + + + + + + + + + + + + + + + + + + + +HTML +Report + + + +RcloneChecksum + +CreateFilter List + +RcloneCopy + +RcloneCheck + +MultiQC + + +Pipeline +created with nf-metro v2.0.0 + diff --git a/modules/local/create_filter_list/main.nf b/modules/local/create_filter_list/main.nf index b82caed..1da22b7 100644 --- a/modules/local/create_filter_list/main.nf +++ b/modules/local/create_filter_list/main.nf @@ -7,12 +7,7 @@ process CREATE_FILTER_LIST { output: tuple val(meta), path('files_to_copy.txt') - script: - def content = common.join('\n') - - """ - cat > files_to_copy.txt <<'EOF' -${content} -EOF - """ + exec: + def outFile = task.workDir.resolve('files_to_copy.txt') + outFile.text = common.join('\n') + '\n' } diff --git a/nextflow_schema.json b/nextflow_schema.json index 738538e..a4a5f43 100644 --- a/nextflow_schema.json +++ b/nextflow_schema.json @@ -47,24 +47,24 @@ "exists": true, "description": "Path to the rclone config file used for cloud storage authentication.", "help_text": "Provide an rclone config file to support cloud providers such as AWS S3 or Azure Blob Storage. This file is loaded by rclone when copying data.", - "fa_icon": "fas fa-file" + "fa_icon": "fas fa-cog" }, "rclone_dry_run": { "type": "boolean", "description": "Perform a dry run of the rclone copy command.", - "fa_icon": "fas fa-eye", + "fa_icon": "fas fa-forward", "help_text": "If set, the pipeline will not actually copy any files to the destination. Instead, it will print out what would have been copied. This is useful for testing and debugging." }, "copy_matching_only": { "type": "boolean", "description": "Only copy files that matched their provided input checksums.", - "fa_icon": "fas fa-eye", + "fa_icon": "fas fa-copy", "help_text": "If set, the pipeline will only copy files that were correctly validated and will skip any file that did not match their input checksum." }, "download": { "type": "boolean", "description": "Download remote files for sha256 checksum verification in `RCLONE_CHECKSUM`.", - "fa_icon": "fas fa-eye", + "fa_icon": "fas fa-cloud-download-alt", "help_text": "If set, `RCLONE_CHECKSUM` will download remote files for any sample for which a SHA256 file was provided." } } diff --git a/ro-crate-metadata.json b/ro-crate-metadata.json index 8ca15e3..919556e 100644 --- a/ro-crate-metadata.json +++ b/ro-crate-metadata.json @@ -23,7 +23,7 @@ "@type": "Dataset", "creativeWorkStatus": "Stable", "datePublished": "2026-09-08T15:38:46+00:00", - "description": "

\n \n \n \"nf-core/datasync\"\n \n

\n\n[![Open in GitHub Codespaces](https://img.shields.io/badge/Open_In_GitHub_Codespaces-black?labelColor=grey&logo=github)](https://github.com/codespaces/new/nf-core/datasync)\n[![GitHub Actions CI Status](https://github.com/nf-core/datasync/actions/workflows/nf-test.yml/badge.svg)](https://github.com/nf-core/datasync/actions/workflows/nf-test.yml)\n[![GitHub Actions Linting Status](https://github.com/nf-core/datasync/actions/workflows/linting.yml/badge.svg)](https://github.com/nf-core/datasync/actions/workflows/linting.yml)[![AWS CI](https://img.shields.io/badge/CI%20tests-full%20size-FF9900?labelColor=000000&logo=Amazon%20AWS)](https://nf-co.re/datasync/results)[![Cite with Zenodo](http://img.shields.io/badge/DOI-10.5281/zenodo.XXXXXXX-1073c8?labelColor=000000)](https://doi.org/10.5281/zenodo.XXXXXXX)\n[![nf-test](https://img.shields.io/badge/unit_tests-nf--test-337ab7.svg)](https://www.nf-test.com)\n\n[![Nextflow](https://img.shields.io/badge/version-%E2%89%A525.10.4-green?style=flat&logo=nextflow&logoColor=white&color=%230DC09D&link=https%3A%2F%2Fnextflow.io)](https://www.nextflow.io/)\n[![nf-core template version](https://img.shields.io/badge/nf--core_template-4.1.0-green?style=flat&logo=nfcore&logoColor=white&color=%2324B064&link=https%3A%2F%2Fnf-co.re)](https://github.com/nf-core/tools/releases/tag/4.1.0)\n[![run with conda](http://img.shields.io/badge/run%20with-conda-3EB049?labelColor=000000&logo=anaconda)](https://docs.conda.io/en/latest/)\n[![run with docker](https://img.shields.io/badge/run%20with-docker-0db7ed?labelColor=000000&logo=docker)](https://www.docker.com/)\n[![run with singularity](https://img.shields.io/badge/run%20with-singularity-1d355c.svg?labelColor=000000)](https://sylabs.io/docs/)\n[![Launch on Seqera Platform](https://img.shields.io/badge/Launch%20%F0%9F%9A%80-Seqera%20Platform-%234256e7)](https://cloud.seqera.io/launch?pipeline=https://github.com/nf-core/datasync)\n\n[![Get help on Slack](http://img.shields.io/badge/slack-nf--core%20%23datasync-4A154B?labelColor=000000&logo=slack)](https://nfcore.slack.com/channels/datasync)[![Follow on Bluesky](https://img.shields.io/badge/bluesky-%40nf__core-1185fe?labelColor=000000&logo=bluesky)](https://bsky.app/profile/nf-co.re)[![Follow on Mastodon](https://img.shields.io/badge/mastodon-nf__core-6364ff?labelColor=FFFFFF&logo=mastodon)](https://mstdn.science/@nf_core)[![Watch on YouTube](http://img.shields.io/badge/youtube-nf--core-FF0000?labelColor=000000&logo=youtube)](https://www.youtube.com/c/nf-core)\n\n## Introduction\n\n**nf-core/datasync** is a Nextflow pipeline for copying files and directories between storage locations and documenting their integrity. For every row in an input samplesheet, the pipeline:\n\n1. validates the source against a supplied MD5 and/or SHA-256 checksum manifest using [`rclone checksum`](https://rclone.org/commands/rclone_checksum/);\n2. copies the source to the requested destination with [`rclone copy`](https://rclone.org/);\n3. compares the copied data with the source using [`rclone check`](https://rclone.org/commands/rclone_check/); and\n4. produces detailed `rclone` status files and a consolidated MultiQC report.\n\nSources and destinations may be local paths or object-storage URIs such as Amazon S3, S3-compatible storage, or Azure Blob Storage. HTTP(S) URLs are not currently supported for samplesheet `input` or `output_path` values.\n\nThe current tested use case for this pipeline is transfer between S3 buckets.\n\nPass an `rclone` configuration with `--rclone_config` whenever a source or destination URI needs credentials or provider settings. Samplesheet paths use local paths or standard URIs such as `s3://bucket/path`, not rclone's `remote:path` syntax. For non-S3 layouts, design and validate the provider-specific configuration using the upstream [rclone documentation](https://rclone.org/docs/).\n\n![nf-core/datasync metro map](docs/images/datasync-metromap.png)\n\n## Quick start\n\n> [!NOTE]\n> If you are new to Nextflow and nf-core, see the [nf-core environment setup guide](https://nf-co.re/docs/get_started/environment_setup/overview). Nextflow 25.10.4 or later is required.\n\nTo explore the pipeline outputs before preparing your own data, run the bundled `test` profile with a container profile:\n\n```bash\nnextflow run nf-core/datasync \\\n -profile test,docker \\\n --outdir results\n```\n\nThe `test` profile supplies a small samplesheet and `rclone` configuration automatically. It also enables `--rclone_dry_run`, so no files are actually transferred. This makes it useful for exploring the `rclone/` output folders and `multiqc/multiqc_report.html`; remember that post-copy comparison reports describe whatever is already present at the destination because the dry run does not write transfer data.\n\nTo run the pipeline on your own data, create a samplesheet containing one transfer per row:\n\n```csv\nsample,input,output_path,checksum_md5,checksum_sha\nrun_001,/data/run_001,s3://archive/runs,/data/manifests/run_001_md5.tsv\nreference,/data/reference.fa,/data/references,,/data/manifests/reference_sha256.tsv\n```\n\nThen launch the pipeline using:\n\n```bash\nnextflow run nf-core/datasync \\\n -r \\\n -profile docker \\\n --input samplesheet.csv \\\n --outdir results \\\n --rclone_config /path/to/rclone.conf\n```\n\n`--rclone_config` is optional only when every source and destination is accessible without a configured rclone remote. See the [`rclone` configuration section](docs/usage.md#configuring-rclone-remotes) for the tested S3-to-S3 use case and guidance on adapting rclone configuration files for other providers. To preview copy operations without transferring data, add `--rclone_dry_run`; note that subsequent comparison reports will then describe the unchanged destination.\n\nSee the [usage documentation](docs/usage.md) for samplesheet rules, destination semantics, remote configuration, and reproducible execution. The complete generated parameter reference is available on the [nf-core pipeline page](https://nf-co.re/datasync/parameters).\n\n## Pipeline output\n\nResults are written below `--outdir`. See the [output documentation](docs/output.md) for file names and status-code interpretation.\n\n## Credits\n\nnf-core/datasync was originally written by Alexander Peltzer.\n\nWe thank the following people for their extensive assistance in the development of this pipeline:\n\n- Julian Schwab\n- Gregor Sturm\n- Antonia Saracco\n- Delfina Terradas\n- Anabella Trigila\n\n## Contributions and Support\n\nIf you would like to contribute to this pipeline, please see the [contributing guidelines](docs/CONTRIBUTING.md).\n\nFor further information or help, don't hesitate to get in touch on the [Slack `#datasync` channel](https://nfcore.slack.com/channels/datasync) (you can join with [this invite](https://nf-co.re/join/slack)).\n\n## Citations\n\n\n\n\nAn extensive list of references for the tools used by the pipeline can be found in the [`CITATIONS.md`](CITATIONS.md) file.\n\nYou can cite the `nf-core` publication as follows:\n\n> **The nf-core framework for community-curated bioinformatics pipelines.**\n>\n> Philip Ewels, Alexander Peltzer, Sven Fillinger, Harshil Patel, Johannes Alneberg, Andreas Wilm, Maxime Ulysse Garcia, Paolo Di Tommaso & Sven Nahnsen.\n>\n> _Nat Biotechnol._ 2020 Feb 13. doi: [10.1038/s41587-020-0439-x](https://dx.doi.org/10.1038/s41587-020-0439-x).\n", + "description": "

\n \n \n \"nf-core/datasync\"\n \n

\n\n[![Open in GitHub Codespaces](https://img.shields.io/badge/Open_In_GitHub_Codespaces-black?labelColor=grey&logo=github)](https://github.com/codespaces/new/nf-core/datasync)\n[![GitHub Actions CI Status](https://github.com/nf-core/datasync/actions/workflows/nf-test.yml/badge.svg)](https://github.com/nf-core/datasync/actions/workflows/nf-test.yml)\n[![GitHub Actions Linting Status](https://github.com/nf-core/datasync/actions/workflows/linting.yml/badge.svg)](https://github.com/nf-core/datasync/actions/workflows/linting.yml)[![AWS CI](https://img.shields.io/badge/CI%20tests-full%20size-FF9900?labelColor=000000&logo=Amazon%20AWS)](https://nf-co.re/datasync/results)[![Cite with Zenodo](http://img.shields.io/badge/DOI-10.5281/zenodo.XXXXXXX-1073c8?labelColor=000000)](https://doi.org/10.5281/zenodo.XXXXXXX)\n[![nf-test](https://img.shields.io/badge/unit_tests-nf--test-337ab7.svg)](https://www.nf-test.com)\n\n[![Nextflow](https://img.shields.io/badge/version-%E2%89%A525.10.4-green?style=flat&logo=nextflow&logoColor=white&color=%230DC09D&link=https%3A%2F%2Fnextflow.io)](https://www.nextflow.io/)\n[![nf-core template version](https://img.shields.io/badge/nf--core_template-4.1.0-green?style=flat&logo=nfcore&logoColor=white&color=%2324B064&link=https%3A%2F%2Fnf-co.re)](https://github.com/nf-core/tools/releases/tag/4.1.0)\n[![run with conda](http://img.shields.io/badge/run%20with-conda-3EB049?labelColor=000000&logo=anaconda)](https://docs.conda.io/en/latest/)\n[![run with docker](https://img.shields.io/badge/run%20with-docker-0db7ed?labelColor=000000&logo=docker)](https://www.docker.com/)\n[![run with singularity](https://img.shields.io/badge/run%20with-singularity-1d355c.svg?labelColor=000000)](https://sylabs.io/docs/)\n[![Launch on Seqera Platform](https://img.shields.io/badge/Launch%20%F0%9F%9A%80-Seqera%20Platform-%234256e7)](https://cloud.seqera.io/launch?pipeline=https://github.com/nf-core/datasync)\n\n[![Get help on Slack](http://img.shields.io/badge/slack-nf--core%20%23datasync-4A154B?labelColor=000000&logo=slack)](https://nfcore.slack.com/channels/datasync)[![Follow on Bluesky](https://img.shields.io/badge/bluesky-%40nf__core-1185fe?labelColor=000000&logo=bluesky)](https://bsky.app/profile/nf-co.re)[![Follow on Mastodon](https://img.shields.io/badge/mastodon-nf__core-6364ff?labelColor=FFFFFF&logo=mastodon)](https://mstdn.science/@nf_core)[![Watch on YouTube](http://img.shields.io/badge/youtube-nf--core-FF0000?labelColor=000000&logo=youtube)](https://www.youtube.com/c/nf-core)\n\n## Introduction\n\n**nf-core/datasync** is a Nextflow pipeline for copying files and directories between storage locations and documenting their integrity. For every row in an input samplesheet, the pipeline:\n\n1. validates the source against a supplied MD5 and/or SHA-256 checksum manifest using [`rclone checksum`](https://rclone.org/commands/rclone_checksum/);\n2. copies the source to the requested destination with [`rclone copy`](https://rclone.org/);\n3. compares the copied data with the source using [`rclone check`](https://rclone.org/commands/rclone_check/); and\n4. produces detailed `rclone` status files and a consolidated MultiQC report.\n\nSources and destinations may be local paths or object-storage URIs such as Amazon S3, S3-compatible storage, or Azure Blob Storage. HTTP(S) URLs are not currently supported for samplesheet `input` or `output_path` values.\n\nThe current tested use case for this pipeline is transfer between S3 buckets.\n\nPass an `rclone` configuration with `--rclone_config` whenever a source or destination URI needs credentials or provider settings. Samplesheet paths use local paths or standard URIs such as `s3://bucket/path`, not rclone's `remote:path` syntax. For non-S3 layouts, design and validate the provider-specific configuration using the upstream [rclone documentation](https://rclone.org/docs/).\n\n![nf-core/datasync metro map](docs/images/datasync_nf-metro.svg)\n\n## Quick start\n\n> [!NOTE]\n> If you are new to Nextflow and nf-core, see the [nf-core environment setup guide](https://nf-co.re/docs/get_started/environment_setup/overview). Nextflow 25.10.4 or later is required.\n\nTo explore the pipeline outputs before preparing your own data, run the bundled `test` profile with a container profile:\n\n```bash\nnextflow run nf-core/datasync \\\n -profile test,docker \\\n --outdir results\n```\n\nThe `test` profile supplies a small samplesheet and `rclone` configuration automatically. It also enables `--rclone_dry_run`, so no files are actually transferred. This makes it useful for exploring the `rclone/` output folders and `multiqc/multiqc_report.html`; remember that post-copy comparison reports describe whatever is already present at the destination because the dry run does not write transfer data.\n\nTo run the pipeline on your own data, create a samplesheet containing one transfer per row:\n\n```csv\nsample,input,output_path,checksum_md5,checksum_sha\nrun_001,/data/run_001,s3://archive/runs,/data/manifests/run_001_md5.tsv\nreference,/data/reference.fa,/data/references,,/data/manifests/reference_sha256.tsv\n```\n\nThen launch the pipeline using:\n\n```bash\nnextflow run nf-core/datasync \\\n -r \\\n -profile docker \\\n --input samplesheet.csv \\\n --outdir results \\\n --rclone_config /path/to/rclone.conf\n```\n\n`--rclone_config` is optional only when every source and destination is accessible without a configured rclone remote. See the [`rclone` configuration section](docs/usage.md#configuring-rclone-remotes) for the tested S3-to-S3 use case and guidance on adapting rclone configuration files for other providers. To preview copy operations without transferring data, add `--rclone_dry_run`; note that subsequent comparison reports will then describe the unchanged destination.\n\nSee the [usage documentation](docs/usage.md) for samplesheet rules, destination semantics, remote configuration, and reproducible execution. The complete generated parameter reference is available on the [nf-core pipeline page](https://nf-co.re/datasync/parameters).\n\n## Pipeline output\n\nResults are written below `--outdir`. See the [output documentation](docs/output.md) for file names and status-code interpretation.\n\n## Credits\n\nnf-core/datasync was originally written by Alexander Peltzer.\n\nWe thank the following people for their extensive assistance in the development of this pipeline:\n\n- Julian Schwab\n- Gregor Sturm\n- Antonia Saracco\n- Delfina Terradas\n- Anabella Trigila\n\n## Contributions and Support\n\nIf you would like to contribute to this pipeline, please see the [contributing guidelines](docs/CONTRIBUTING.md).\n\nFor further information or help, don't hesitate to get in touch on the [Slack `#datasync` channel](https://nfcore.slack.com/channels/datasync) (you can join with [this invite](https://nf-co.re/join/slack)).\n\n## Citations\n\n\n\n\nAn extensive list of references for the tools used by the pipeline can be found in the [`CITATIONS.md`](CITATIONS.md) file.\n\nYou can cite the `nf-core` publication as follows:\n\n> **The nf-core framework for community-curated bioinformatics pipelines.**\n>\n> Philip Ewels, Alexander Peltzer, Sven Fillinger, Harshil Patel, Johannes Alneberg, Andreas Wilm, Maxime Ulysse Garcia, Paolo Di Tommaso & Sven Nahnsen.\n>\n> _Nat Biotechnol._ 2020 Feb 13. doi: [10.1038/s41587-020-0439-x](https://dx.doi.org/10.1038/s41587-020-0439-x).\n", "hasPart": [ { "@id": "main.nf" diff --git a/subworkflows/local/utils_nfcore_datasync_pipeline/main.nf b/subworkflows/local/utils_nfcore_datasync_pipeline/main.nf index dca6fc9..2f42376 100644 --- a/subworkflows/local/utils_nfcore_datasync_pipeline/main.nf +++ b/subworkflows/local/utils_nfcore_datasync_pipeline/main.nf @@ -11,6 +11,7 @@ include { UTILS_NFSCHEMA_PLUGIN } from '../../nf-core/utils_nfschema_plugin' include { paramsSummaryMap } from 'plugin/nf-schema' include { samplesheetToList } from 'plugin/nf-schema' +include { paramsHelp } from 'plugin/nf-schema' include { completionEmail } from '../../nf-core/utils_nfcore_pipeline' include { completionSummary } from '../../nf-core/utils_nfcore_pipeline' include { UTILS_NFCORE_PIPELINE } from '../../nf-core/utils_nfcore_pipeline'