Simplify circrna_discovery subworkflow #122

nictru · 2024-05-24T14:34:44Z

The following changes were performed:

Introduce multiple new sub-workflows, which increase the readability of the formerly huge circrna_discovery workflow
Replace some simple format unification processes with GAWK calls using the according nf-core module
Perform disabling of discovery processes using nextflow if-statements instead of ext.when directives
Remove checks for activation of the circrna_discovery module as this module is mandatory anyway and executing the pipeline without it does not make sense
- This is the first step towards deprecating the redundant modules parameter. The modules that can be deactivated are differential expression and miRNA prediction. Both only make sense if their required input files are provided (miRNA FASTA for miRNA prediction and phenotype table for differential expression). In future versions, we should be able to execute all modules for which the necessary data was provided.

github-actions · 2024-05-24T14:36:41Z

`nf-core lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit f351b96

+| ✅ 226 tests passed       |+
!| ❗   1 tests had warnings |!

❗ Test warnings:

readme - README contains the placeholder zenodo.XXXXXXX. This should be replaced with the zenodo doi (after the first release).

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: assets/nf-core-circrna_logo_light.png
files_exist - File found: conf/modules.config
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/images/nf-core-circrna_logo_light.png
files_exist - File found: docs/images/nf-core-circrna_logo_dark.png
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: conf/igenomes.config
files_exist - File found: .github/workflows/awstest.yml
files_exist - File found: .github/workflows/awsfulltest.yml
files_exist - File found: modules.json
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: docs/images/nf-core-circrna_logo.png
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/NfcoreTemplate.groovy
files_exist - File not found check: lib/Utils.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: lib/WorkflowMain.groovy
files_exist - File not found check: lib/WorkflowCircrna.groovy
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: pipeline_template.yml
files_exist - File not found check: Singularity
files_exist - File not found check: lib/nfcore_external_java_deps.jar
files_exist - File not found check: .travis.yml
nextflow_config - Config variable found: manifest.name
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: manifest.homePage
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: params.validationShowHiddenParams
nextflow_config - Config variable found: params.validationSchemaIgnoreParams
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config manifest.name began with nf-core/
nextflow_config - Config variable manifest.homePage began with https://github.com/nf-core/
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config manifest.version ends in dev: dev
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
nextflow_config - nextflow.config contains configuration profile test
nextflow_config - Config default value correct: params.tool= circexplorer2
nextflow_config - Config default value correct: params.module= circrna_discovery
nextflow_config - Config default value correct: params.bsj_reads= 0
nextflow_config - Config default value correct: params.tool_filter= 0
nextflow_config - Config default value correct: params.duplicates_fun= mean
nextflow_config - Config default value correct: params.save_intermediates= false
nextflow_config - Config default value correct: params.exon_boundary= 200
nextflow_config - Config default value correct: params.sjdboverhang= 100
nextflow_config - Config default value correct: params.chimJunctionOverhangMin= 10
nextflow_config - Config default value correct: params.alignSJDBoverhangMin= 10
nextflow_config - Config default value correct: params.limitSjdbInsertNsj= 1000000
nextflow_config - Config default value correct: params.chimSegmentMin= 10
nextflow_config - Config default value correct: params.seglen= 25
nextflow_config - Config default value correct: params.min_intron= 20
nextflow_config - Config default value correct: params.max_intron= 1000000
nextflow_config - Config default value correct: params.min_map_len= 40
nextflow_config - Config default value correct: params.min_fusion_distance= 200
nextflow_config - Config default value correct: params.bootstrap_samples= 30
nextflow_config - Config default value correct: params.save_unaligned= false
nextflow_config - Config default value correct: params.save_reference= true
nextflow_config - Config default value correct: params.hisat2_build_memory= 200.GB
nextflow_config - Config default value correct: params.skip_trimming= false
nextflow_config - Config default value correct: params.save_trimmed= false
nextflow_config - Config default value correct: params.skip_fastqc= false
nextflow_config - Config default value correct: params.min_trimmed_reads= 10000
nextflow_config - Config default value correct: params.custom_config_version= master
nextflow_config - Config default value correct: params.custom_config_base= https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Config default value correct: params.max_cpus= 50
nextflow_config - Config default value correct: params.max_memory= 300.GB
nextflow_config - Config default value correct: params.max_time= 240.h
nextflow_config - Config default value correct: params.publish_dir_mode= copy
nextflow_config - Config default value correct: params.max_multiqc_email_size= 25.MB
nextflow_config - Config default value correct: params.validate_params= true
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - CODE_OF_CONDUCT.md matches the template
files_unchanged - LICENSE matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/CONTRIBUTING.md matches the template
files_unchanged - .github/ISSUE_TEMPLATE/bug_report.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/config.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/PULL_REQUEST_TEMPLATE.md matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - .github/workflows/linting.yml matches the template
files_unchanged - assets/email_template.html matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - assets/sendmail_template.txt matches the template
files_unchanged - assets/nf-core-circrna_logo_light.png matches the template
files_unchanged - docs/images/nf-core-circrna_logo_light.png matches the template
files_unchanged - docs/images/nf-core-circrna_logo_dark.png matches the template
files_unchanged - docs/README.md matches the template
files_unchanged - .gitignore matches the template
files_unchanged - .prettierignore matches the template
actions_ci - '.github/workflows/ci.yml' is triggered on expected events
actions_ci - '.github/workflows/ci.yml' checks minimum NF version
actions_awstest - '.github/workflows/awstest.yml' is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml does not use -profile test
readme - README Nextflow minimum version badge matched config. Badge: 23.04.0, Config: 23.04.0
pipeline_todos - No TODO strings found
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (372 files)
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
schema_params - Schema matched params returned from nextflow config
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: branch.yml
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: awsfulltest.yml
actions_schema_validation - Workflow validation passed: fix-linting.yml
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: download_pipeline.yml
actions_schema_validation - Workflow validation passed: release-announcements.yml
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: awstest.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
multiqc_config - assets/multiqc_config.yml found and not ignored.
multiqc_config - assets/multiqc_config.yml contains report_section_order
multiqc_config - assets/multiqc_config.yml contains export_plots
multiqc_config - assets/multiqc_config.yml contains report_comment
multiqc_config - assets/multiqc_config.yml follows the ordering scheme of the minimally required plugins.
multiqc_config - assets/multiqc_config.yml contains a matching 'report_comment'.
multiqc_config - assets/multiqc_config.yml contains 'export_plots: true'.
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'
base_config - conf/base.config found and not ignored.
modules_config - conf/modules.config found and not ignored.
modules_config - CUSTOM_DUMPSOFTWAREVERSIONS found in conf/modules.config and Nextflow scripts.
modules_config - CLEAN_FASTA found in conf/modules.config and Nextflow scripts.
modules_config - SEQKIT_SPLIT found in conf/modules.config and Nextflow scripts.
modules_config - BOWTIE_BUILD found in conf/modules.config and Nextflow scripts.
modules_config - BOWTIE2_BUILD found in conf/modules.config and Nextflow scripts.
modules_config - BWA_INDEX found in conf/modules.config and Nextflow scripts.
modules_config - HISAT2_EXTRACTSPLICESITES found in conf/modules.config and Nextflow scripts.
modules_config - HISAT2_BUILD found in conf/modules.config and Nextflow scripts.
modules_config - STAR_GENOMEGENERATE found in conf/modules.config and Nextflow scripts.
modules_config - FILTER_BSJS found in conf/modules.config and Nextflow scripts.
modules_config - ANNOTATION found in conf/modules.config and Nextflow scripts.
modules_config - UPSET_SAMPLES found in conf/modules.config and Nextflow scripts.
modules_config - UPSET_ALL found in conf/modules.config and Nextflow scripts.
modules_config - ADD_BACKSPLICE found in conf/modules.config and Nextflow scripts.
modules_config - COMBINE_TRANSCRIPTOME_GTFS found in conf/modules.config and Nextflow scripts.
modules_config - EXCLUDE_OVERLONG_TRANSCRIPTS found in conf/modules.config and Nextflow scripts.
modules_config - MARK_CIRCULAR found in conf/modules.config and Nextflow scripts.
modules_config - TARGETSCAN_DATABASE found in conf/modules.config and Nextflow scripts.
modules_config - TARGETSCAN found in conf/modules.config and Nextflow scripts.
modules_config - MIRANDA found in conf/modules.config and Nextflow scripts.
modules_config - MIRNA_TARGETS found in conf/modules.config and Nextflow scripts.
modules_config - HISAT2_ALIGN found in conf/modules.config and Nextflow scripts.
modules_config - SAMTOOLS_SORT found in conf/modules.config and Nextflow scripts.
modules_config - STRINGTIE_STRINGTIE found in conf/modules.config and Nextflow scripts.
modules_config - STRINGTIE_PREPDE found in conf/modules.config and Nextflow scripts.
modules_config - DESEQ2_DIFFERENTIAL_EXPRESSION found in conf/modules.config and Nextflow scripts.
modules_config - PARENT_GENE found in conf/modules.config and Nextflow scripts.
modules_config - PREPARE_CLR_TEST found in conf/modules.config and Nextflow scripts.
modules_config - CIRCTEST found in conf/modules.config and Nextflow scripts.
nfcore_yml - Repository type in .nf-core.yml is valid: pipeline
nfcore_yml - nf-core version in .nf-core.yml is set to the latest version: 2.14.1

Run details

nf-core/tools version 2.14.1
Run at 2024-05-27 13:11:16

nictru · 2024-05-24T14:36:51Z

@nf-core-bot fix linting

maxulysse · 2024-05-27T16:00:58Z

what does DCC stands for?

nictru · 2024-05-27T16:04:30Z

It is a package name: https://github.com/dieterich-lab/DCC

maxulysse

LGTM, just a bit confused with how you use SEGEMHL now, you don't generate the index anymore?

nictru · 2024-05-27T16:07:06Z

subworkflows/local/discovery/segemehl.nf

+    main:
+    ch_versions = Channel.empty()
+
+    index = index ?: INDEX( fasta ).index


Since the segemehl index is only used in the segemehl subworkflow I thought it might be more understandable putting it right here

nictru added 24 commits May 24, 2024 08:21

Simplify STAR and SEGEMEHL discovery

1bb983b

Simplify CIRCEXPLORER2 workflow

aa4597e

Simplify CIRCRNA_FINDER

14b0f12

Simplify FIND_CIRC

5032443

Simplify CIRIQUANT

bb73664

Simplify DCC

3491bed

Simplify MAPSPLICE

6c9eefc

Restructure discovery tool calls

291c14e

Restructure discovery downstream workflows

5fa979b

Simplify annotation workflow

a746548

Change the way discovery tools are disabled

ca5abfa

Switch segemehl to bed output

27b5ab7

Implement central BSJ filtering

d5d0f26

Implement circexplorer2 unification

749c436

Unify find_circ

bc09e30

Unify ciriquant

e02cb54

Unify DCC

de9645c

Unify mapsplice

0dd9a4f

Unify circrna_finder

4fa56f3

Rename RUN to MAIN in all occurrences

01b7899

Add circrna_finder unify version capture

a6a82f3

Remove tool-specific filtering artifacts

aca81ce

Remove technical debt with former "results" channel

fca0475

Remove unnecessary checks if circrna_discovery is enabled

e6e3157

Fix code style problems

f351b96

maxulysse reviewed May 27, 2024

View reviewed changes

nictru commented May 27, 2024

View reviewed changes

maxulysse approved these changes May 28, 2024

View reviewed changes

nictru merged commit 5ef923d into dev May 28, 2024
4 checks passed

nictru deleted the discovery-splitup branch May 28, 2024 09:49

nictru mentioned this pull request May 30, 2024

Segemehl update #123

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Simplify circrna_discovery subworkflow #122

Simplify circrna_discovery subworkflow #122

nictru commented May 24, 2024

github-actions bot commented May 24, 2024 •

edited

Loading

❗ Test warnings:

✅ Tests passed:

Run details

nictru commented May 24, 2024

maxulysse commented May 27, 2024

nictru commented May 27, 2024

maxulysse left a comment

nictru May 27, 2024

nictru May 27, 2024

Simplify circrna_discovery subworkflow #122

Simplify circrna_discovery subworkflow #122

Conversation

nictru commented May 24, 2024

github-actions bot commented May 24, 2024 • edited Loading

nf-core lint overall result: Passed ✅ ⚠️

❗ Test warnings:

✅ Tests passed:

Run details

nictru commented May 24, 2024

maxulysse commented May 27, 2024

nictru commented May 27, 2024

maxulysse left a comment

Choose a reason for hiding this comment

nictru May 27, 2024

Choose a reason for hiding this comment

nictru May 27, 2024

Choose a reason for hiding this comment

github-actions bot commented May 24, 2024 •

edited

Loading

`nf-core lint` overall result: Passed ✅ ⚠️