Network resiliency #197

muffato · 2025-09-10T14:43:15Z

Closes #196

A mixed bag pull-request. My first motivation was to put in a fix for #196. The script bin/generate_config.py already had some code to cope with network errors (wait and retry) but it was only 1 API call. I've now put this into a function and made all API calls use it.

Then, since we need to a release, I've started the process of bumping the version number up, updating the changelog, etc.

@DLBPointon : do you have a Pokemon name to suggest ? :)

PR checklist

github-actions · 2025-09-10T14:44:23Z

`nf-core pipelines lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit dce9497

+| ✅ 187 tests passed       |+
#| ❔  28 tests were ignored |#
!| ❗  19 tests had warnings |!

❗ Test warnings:

nextflow_config - Config manifest.version should end in dev: 0.9.0
local_component_structure - windowstats_input.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
local_component_structure - nohit_list.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
local_component_structure - create_bed.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
local_component_structure - compressblobdir.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
local_component_structure - restructurebuscodir.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
local_component_structure - jsonify_taxdump.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
local_component_structure - generate_config.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
local_component_structure - coverage_stats.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
local_component_structure - minimap_alignment.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
local_component_structure - busco_diamond_blastp.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
local_component_structure - run_blastn.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
local_component_structure - run_blastx.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
local_component_structure - collate_stats.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
local_component_structure - prepare_genome.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
local_component_structure - blobtools.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
local_component_structure - input_check.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
local_component_structure - view.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
local_component_structure - finalise_blobdir.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure

❔ Tests ignored:

files_exist - File is ignored: CODE_OF_CONDUCT.md
files_exist - File is ignored: assets/nf-core-blobtoolkit_logo_light.png
files_exist - File is ignored: docs/images/nf-core-blobtoolkit_logo_light.png
files_exist - File is ignored: docs/images/nf-core-blobtoolkit_logo_dark.png
files_exist - File is ignored: .github/ISSUE_TEMPLATE/config.yml
files_exist - File is ignored: .github/workflows/awstest.yml
files_exist - File is ignored: .github/workflows/awsfulltest.yml
files_exist - File is ignored: conf/igenomes.config
nextflow_config - Config variable ignored: manifest.name
nextflow_config - Config variable ignored: manifest.homePage
files_unchanged - File ignored due to lint config: CODE_OF_CONDUCT.md
files_unchanged - File ignored due to lint config: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_unchanged - File ignored due to lint config: .github/CONTRIBUTING.md
files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/bug_report.yml
files_unchanged - File does not exist: .github/ISSUE_TEMPLATE/config.yml
files_unchanged - File ignored due to lint config: .github/PULL_REQUEST_TEMPLATE.md
files_unchanged - File ignored due to lint config: .github/workflows/branch.yml
files_unchanged - File ignored due to lint config: .github/workflows/linting_comment.yml
files_unchanged - File ignored due to lint config: .github/workflows/linting.yml
files_unchanged - File ignored due to lint config: assets/email_template.html
files_unchanged - File ignored due to lint config: assets/email_template.txt
files_unchanged - File ignored due to lint config: assets/sendmail_template.txt
files_unchanged - File ignored due to lint config: assets/nf-core-blobtoolkit_logo_light.png
files_unchanged - File ignored due to lint config: docs/images/nf-core-blobtoolkit_logo_light.png
files_unchanged - File ignored due to lint config: docs/images/nf-core-blobtoolkit_logo_dark.png
actions_awstest - 'awstest.yml' workflow not found: /home/runner/work/blobtoolkit/blobtoolkit/.github/workflows/awstest.yml
template_strings - template_strings
merge_markers - merge_markers

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: conf/modules.config
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: conf/igenomes_ignored.config
files_exist - File found: modules.json
files_exist - File found: ro-crate-metadata.json
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: docs/images/nf-core-blobtoolkit_logo.png
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/NfcoreTemplate.groovy
files_exist - File not found check: lib/Utils.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: lib/WorkflowMain.groovy
files_exist - File not found check: lib/WorkflowBlobtoolkit.groovy
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: pipeline_template.yml
files_exist - File not found check: Singularity
files_exist - File not found check: lib/nfcore_external_java_deps.jar
files_exist - File not found check: .travis.yml
nextflow_config - Found nf-schema plugin
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: validation.help.enabled
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable found: validation.help.beforeText
nextflow_config - Config variable found: validation.help.afterText
nextflow_config - Config variable found: validation.help.command
nextflow_config - Config variable found: validation.summary.beforeText
nextflow_config - Config variable found: validation.summary.afterText
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config variable (correctly) not found: params.max_cpus
nextflow_config - Config variable (correctly) not found: params.max_memory
nextflow_config - Config variable (correctly) not found: params.max_time
nextflow_config - Config variable (correctly) not found: params.validationFailUnrecognisedParams
nextflow_config - Config variable (correctly) not found: params.validationLenientMode
nextflow_config - Config variable (correctly) not found: params.validationSchemaIgnoreParams
nextflow_config - Config variable (correctly) not found: params.validationShowHiddenParams
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
nextflow_config - nextflow.config contains configuration profile test
nextflow_config - Config default value correct: params.image_format= png
nextflow_config - Config default value correct: params.outdir= results
nextflow_config - Config default value correct: params.custom_config_version= master
nextflow_config - Config default value correct: params.custom_config_base= https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Config default value correct: params.publish_dir_mode= copy
nextflow_config - Config default value correct: params.max_multiqc_email_size= 25.MB
nextflow_config - Config default value correct: params.validate_params= true
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - docs/README.md matches the template
files_unchanged - .gitignore matches the template
files_unchanged - .prettierignore matches the template
actions_ci - '.github/workflows/ci.yml' is triggered on expected events
actions_ci - '.github/workflows/ci.yml' checks minimum NF version
readme - README Nextflow minimum version badge matched config. Badge: 24.04.2, Config: 24.04.2
readme - README Zenodo placeholder was replaced with DOI.
pipeline_todos - No TODO strings found
plugin_includes - No wrong validation plugin imports have been found
pipeline_name_conventions - Name adheres to nf-core convention
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
schema_params - Schema matched params returned from nextflow config
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: linting_comment.yml
actions_schema_validation - Workflow validation passed: sanger_test_full.yml
actions_schema_validation - Workflow validation passed: template_version_comment.yml
actions_schema_validation - Workflow validation passed: download_pipeline.yml
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: branch.yml
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: sanger_test.yml
actions_schema_validation - Workflow validation passed: fix-linting.yml
actions_schema_validation - Workflow validation passed: ci.yml
modules_json - Only installed modules found in modules.json
multiqc_config - assets/multiqc_config.yml found and not ignored.
multiqc_config - assets/multiqc_config.yml contains report_section_order
multiqc_config - assets/multiqc_config.yml contains export_plots
multiqc_config - assets/multiqc_config.yml contains report_comment
multiqc_config - assets/multiqc_config.yml follows the ordering scheme of the minimally required plugins.
multiqc_config - assets/multiqc_config.yml contains 'export_plots: true'.
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'
base_config - conf/base.config found and not ignored.
base_config - WINDOWSTATS_INPUT found in conf/base.config and Nextflow scripts.
base_config - BLOBTOOLKIT_WINDOWSTATS found in conf/base.config and Nextflow scripts.
base_config - FASTAWINDOWS found in conf/base.config and Nextflow scripts.
base_config - BUSCO_BUSCO found in conf/base.config and Nextflow scripts.
base_config - BLAST_BLASTN found in conf/base.config and Nextflow scripts.
base_config - NOHIT_LIST found in conf/base.config and Nextflow scripts.
modules_config - conf/modules.config found and not ignored.
modules_config - WINDOWMASKER_MKCOUNTS found in conf/modules.config and Nextflow scripts.
modules_config - WINDOWMASKER_USTAT found in conf/modules.config and Nextflow scripts.
modules_config - MINIMAP2_HIC found in conf/modules.config and Nextflow scripts.
modules_config - MINIMAP2_ILMN found in conf/modules.config and Nextflow scripts.
modules_config - MINIMAP2_CCS found in conf/modules.config and Nextflow scripts.
modules_config - MINIMAP2_CLR found in conf/modules.config and Nextflow scripts.
modules_config - MINIMAP2_ONT found in conf/modules.config and Nextflow scripts.
modules_config - MINIMAP2_ found in conf/modules.config and Nextflow scripts.
modules_config - SAMTOOLS_VIEW found in conf/modules.config and Nextflow scripts.
modules_config - SAMTOOLS_INDEX found in conf/modules.config and Nextflow scripts.
modules_config - PIGZ_COMPRESS found in conf/modules.config and Nextflow scripts.
modules_config - BLOBTK_DEPTH found in conf/modules.config and Nextflow scripts.
modules_config - BUSCO_BUSCO found in conf/modules.config and Nextflow scripts.
modules_config - RESTRUCTUREBUSCODIR found in conf/modules.config and Nextflow scripts.
modules_config - DIAMOND_BLASTP found in conf/modules.config and Nextflow scripts.
modules_config - DIAMOND_BLASTX found in conf/modules.config and Nextflow scripts.
modules_config - BLOBTK_DEPTH found in conf/modules.config and Nextflow scripts.
modules_config - BLOBTOOLKIT_WINDOWSTATS found in conf/modules.config and Nextflow scripts.
modules_config - BLOBTOOLKIT_CREATEBLOBDIR found in conf/modules.config and Nextflow scripts.
modules_config - BLOBTOOLKIT_UPDATEBLOBDIR found in conf/modules.config and Nextflow scripts.
modules_config - BLOBTOOLKIT_CHUNK found in conf/modules.config and Nextflow scripts.
modules_config - BLOBTOOLKIT_UNCHUNK found in conf/modules.config and Nextflow scripts.
modules_config - NOHIT_LIST found in conf/modules.config and Nextflow scripts.
modules_config - BLAST_BLASTN found in conf/modules.config and Nextflow scripts.
modules_config - COMPRESSBLOBDIR found in conf/modules.config and Nextflow scripts.
modules_config - BLOBTK_IMAGES found in conf/modules.config and Nextflow scripts.
modules_config - JSONIFY_TAXDUMP found in conf/modules.config and Nextflow scripts.
modules_config - MULTIQC found in conf/modules.config and Nextflow scripts.
nfcore_yml - Repository type in .nf-core.yml is valid: pipeline
nfcore_yml - nf-core version in .nf-core.yml is set to the latest version: 3.2.1

Run details

nf-core/tools version 3.2.1
Run at 2025-09-10 15:54:01

muffato · 2025-09-10T14:45:00Z

(the linting is complaining about the changelog, but it's a line that will change anyway once we have the release name)

DLBPointon · 2025-09-10T15:23:02Z

Oo, one of my favourites is Scyther!

DLBPointon

LGTM

Beyond the name change, everything else looks good.

sanger-tolsoft · 2025-09-10T15:53:45Z

Warning

Newer version of the nf-core template is available.

Your pipeline is using an old version of the nf-core template: 3.2.1.
Please update your pipeline to the latest version.

For more documentation on how to update your pipeline, please see the nf-core documentation and Synchronisation documentation.

muffato added 5 commits September 10, 2025 15:35

Edited for consistency

238d198

More time to allow for larger genomes

e805a38

Version bump

ecad033

Rolled out the session-retry mechanism to all API calls

b001c94

Changelog update

10840b1

muffato requested a review from DLBPointon September 10, 2025 14:43

muffato self-assigned this Sep 10, 2025

muffato marked this pull request as ready for review September 10, 2025 14:45

DLBPointon approved these changes Sep 10, 2025

View reviewed changes

Release name !

dce9497

muffato merged commit 97931d6 into dev Sep 10, 2025
5 checks passed

muffato deleted the network_resiliency branch September 10, 2025 16:38

muffato linked an issue Sep 10, 2025 that may be closed by this pull request

test is not working as expected. Fails on generate config #196

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Network resiliency #197

Network resiliency #197

Uh oh!

muffato commented Sep 10, 2025

Uh oh!

github-actions bot commented Sep 10, 2025 •

edited

Loading

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

Uh oh!

muffato commented Sep 10, 2025

Uh oh!

DLBPointon commented Sep 10, 2025

Uh oh!

DLBPointon left a comment

Uh oh!

sanger-tolsoft commented Sep 10, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Network resiliency #197

Network resiliency #197

Uh oh!

Conversation

muffato commented Sep 10, 2025

PR checklist

Uh oh!

github-actions bot commented Sep 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

nf-core pipelines lint overall result: Passed ✅ ⚠️

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

Uh oh!

muffato commented Sep 10, 2025

Uh oh!

DLBPointon commented Sep 10, 2025

Uh oh!

DLBPointon left a comment

Choose a reason for hiding this comment

Uh oh!

sanger-tolsoft commented Sep 10, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

github-actions bot commented Sep 10, 2025 •

edited

Loading

`nf-core pipelines lint` overall result: Passed ✅ ⚠️