Fix some general pipeline problems #104

nictru · 2024-03-18T10:51:48Z

This PR tackles multiple things (sorry for that):

Fix caching problems with nextflow fastq split functionality by utilizing seqkit_split
Update modules & subworkflows
Unify structure of ch_gtf and ch_fasta channels
Fix a problem with the tmp directory that only occurs with apptainer as containerization engine

github-actions · 2024-03-18T10:53:28Z

`nf-core lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit e691f5c

+| ✅ 192 tests passed       |+
!| ❗   1 tests had warnings |!

❗ Test warnings:

readme - README contains the placeholder zenodo.XXXXXXX. This should be replaced with the zenodo doi (after the first release).

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: assets/nf-core-circrna_logo_light.png
files_exist - File found: conf/modules.config
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/images/nf-core-circrna_logo_light.png
files_exist - File found: docs/images/nf-core-circrna_logo_dark.png
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: conf/igenomes.config
files_exist - File found: .github/workflows/awstest.yml
files_exist - File found: .github/workflows/awsfulltest.yml
files_exist - File found: modules.json
files_exist - File found: pyproject.toml
files_exist - File not found check: Singularity
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: pipeline_template.yml
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: docs/images/nf-core-circrna_logo.png
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: lib/Utils.groovy
files_exist - File not found check: lib/WorkflowMain.groovy
files_exist - File not found check: lib/NfcoreTemplate.groovy
files_exist - File not found check: lib/WorkflowCircrna.groovy
files_exist - File not found check: lib/nfcore_external_java_deps.jar
files_exist - File not found check: .travis.yml
nextflow_config - Config variable found: manifest.name
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: manifest.homePage
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: params.validationShowHiddenParams
nextflow_config - Config variable found: params.validationSchemaIgnoreParams
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config manifest.name began with nf-core/
nextflow_config - Config variable manifest.homePage began with https://github.com/nf-core/
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config manifest.version ends in dev: dev
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
nextflow_config - nextflow.config contains configuration profile test
nextflow_config - Config default value correct: params.tool= circexplorer2
nextflow_config - Config default value correct: params.module= circrna_discovery
nextflow_config - Config default value correct: params.bsj_reads= 0
nextflow_config - Config default value correct: params.tool_filter= 0
nextflow_config - Config default value correct: params.duplicates_fun= mean
nextflow_config - Config default value correct: params.save_intermediates= false
nextflow_config - Config default value correct: params.exon_boundary= 200
nextflow_config - Config default value correct: params.sjdboverhang= 100
nextflow_config - Config default value correct: params.chimJunctionOverhangMin= 10
nextflow_config - Config default value correct: params.alignSJDBoverhangMin= 10
nextflow_config - Config default value correct: params.chimSegmentMin= 10
nextflow_config - Config default value correct: params.seglen= 25
nextflow_config - Config default value correct: params.min_intron= 20
nextflow_config - Config default value correct: params.max_intron= 1000000
nextflow_config - Config default value correct: params.min_map_len= 40
nextflow_config - Config default value correct: params.min_fusion_distance= 200
nextflow_config - Config default value correct: params.save_unaligned= false
nextflow_config - Config default value correct: params.save_reference= true
nextflow_config - Config default value correct: params.hisat2_build_memory= 200.GB
nextflow_config - Config default value correct: params.skip_trimming= false
nextflow_config - Config default value correct: params.save_trimmed= false
nextflow_config - Config default value correct: params.skip_fastqc= false
nextflow_config - Config default value correct: params.min_trimmed_reads= 10000
nextflow_config - Config default value correct: params.custom_config_version= master
nextflow_config - Config default value correct: params.custom_config_base= https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Config default value correct: params.max_cpus= 50
nextflow_config - Config default value correct: params.max_memory= 300.GB
nextflow_config - Config default value correct: params.max_time= 240.h
nextflow_config - Config default value correct: params.publish_dir_mode= copy
nextflow_config - Config default value correct: params.max_multiqc_email_size= 25.MB
nextflow_config - Config default value correct: params.validate_params= true
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - CODE_OF_CONDUCT.md matches the template
files_unchanged - LICENSE matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/CONTRIBUTING.md matches the template
files_unchanged - .github/ISSUE_TEMPLATE/bug_report.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/config.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/PULL_REQUEST_TEMPLATE.md matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - .github/workflows/linting.yml matches the template
files_unchanged - assets/email_template.html matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - assets/sendmail_template.txt matches the template
files_unchanged - assets/nf-core-circrna_logo_light.png matches the template
files_unchanged - docs/images/nf-core-circrna_logo_light.png matches the template
files_unchanged - docs/images/nf-core-circrna_logo_dark.png matches the template
files_unchanged - docs/README.md matches the template
files_unchanged - .gitignore matches the template
files_unchanged - .prettierignore matches the template
files_unchanged - pyproject.toml matches the template
actions_ci - '.github/workflows/ci.yml' is triggered on expected events
actions_ci - '.github/workflows/ci.yml' checks minimum NF version
actions_awstest - '.github/workflows/awstest.yml' is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml does not use -profile test
readme - README Nextflow minimum version badge matched config. Badge: 23.04.0, Config: 23.04.0
pipeline_todos - No TODO strings found
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (310 files)
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
schema_params - Schema matched params returned from nextflow config
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: awstest.yml
actions_schema_validation - Workflow validation passed: fix-linting.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: branch.yml
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: release-announcements.yml
actions_schema_validation - Workflow validation passed: awsfulltest.yml
actions_schema_validation - Workflow validation passed: download_pipeline.yml
actions_schema_validation - Workflow validation passed: linting.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
multiqc_config - 'assets/multiqc_config.yml' contains report_section_order
multiqc_config - 'assets/multiqc_config.yml' contains export_plots
multiqc_config - 'assets/multiqc_config.yml' contains report_comment
multiqc_config - 'assets/multiqc_config.yml' follows the ordering scheme of the minimally required plugins.
multiqc_config - 'assets/multiqc_config.yml' contains a matching 'report_comment'.
multiqc_config - 'assets/multiqc_config.yml' contains 'export_plots: true'.
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'

Run details

nf-core/tools version 2.13.1
Run at 2024-03-19 16:13:18

conf/test.config

FriederikeHanssen

looks really good, just maybe check on that chromomse thing

modules/local/mapsplice/align/main.nf

modules/local/seqkit/split/main.nf

MariekeVromman

looks good as far as I can tell

modules.json

subworkflows/local/circrna_discovery.nf

nictru added 6 commits March 18, 2024 08:58

Improve channel structure

5a68987

Implement seqkit_split

0a782be

Update seqkit version

0964206

Update modules

c4a3e8e

Fix some common problems in CIRIquant and HISAT2_ALIGN

72dd514

Fix miranda channel structure

6fc9ae8

nictru added 7 commits March 18, 2024 11:54

Fix linting

4e50b0b

Add some meta inputs

919f5fa

Prettier

50a6ca2

Unify species_id parameter

30552fa

Simplify handling of tool indices

b4929ee

Add defaults for previously undefined parameters

f741c75

Linting

c2089ca

FriederikeHanssen reviewed Mar 19, 2024

View reviewed changes

conf/test.config Outdated Show resolved Hide resolved

Fix indent

e691f5c

FriederikeHanssen approved these changes Mar 19, 2024

View reviewed changes

modules/local/mapsplice/align/main.nf Show resolved Hide resolved

modules/local/seqkit/split/main.nf Show resolved Hide resolved

nictru mentioned this pull request Mar 19, 2024

Implement common quantification #106

Merged

MariekeVromman self-requested a review March 21, 2024 09:57

MariekeVromman approved these changes Mar 21, 2024

View reviewed changes

modules.json Show resolved Hide resolved

subworkflows/local/circrna_discovery.nf Show resolved Hide resolved

nictru merged commit d950657 into dev Mar 21, 2024
4 checks passed

MariekeVromman mentioned this pull request Apr 8, 2024

CIRIquant test fails since commit 72dd514, PR #104 #108

Closed

nictru deleted the improve-genome-preparation branch May 30, 2024 15:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix some general pipeline problems #104

Fix some general pipeline problems #104

nictru commented Mar 18, 2024

github-actions bot commented Mar 18, 2024 •

edited

❗ Test warnings:

✅ Tests passed:

Run details

FriederikeHanssen left a comment

MariekeVromman left a comment

Fix some general pipeline problems #104

Fix some general pipeline problems #104

Conversation

nictru commented Mar 18, 2024

github-actions bot commented Mar 18, 2024 • edited

nf-core lint overall result: Passed ✅ ⚠️

❗ Test warnings:

✅ Tests passed:

Run details

FriederikeHanssen left a comment

Choose a reason for hiding this comment

MariekeVromman left a comment

Choose a reason for hiding this comment

github-actions bot commented Mar 18, 2024 •

edited

`nf-core lint` overall result: Passed ✅ ⚠️