Impove annotation performance #95

nictru · 2024-02-01T11:59:05Z

This PR increases the performance of the annotation by using bedtools and aggregating hits using pandas.

github-actions · 2024-02-01T12:06:14Z

`nf-core lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit 2de5b60

+| ✅ 192 tests passed       |+
!| ❗   1 tests had warnings |!

❗ Test warnings:

readme - README contains the placeholder zenodo.XXXXXXX. This should be replaced with the zenodo doi (after the first release).

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: assets/nf-core-circrna_logo_light.png
files_exist - File found: conf/modules.config
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/images/nf-core-circrna_logo_light.png
files_exist - File found: docs/images/nf-core-circrna_logo_dark.png
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: conf/igenomes.config
files_exist - File found: .github/workflows/awstest.yml
files_exist - File found: .github/workflows/awsfulltest.yml
files_exist - File found: modules.json
files_exist - File found: pyproject.toml
files_exist - File not found check: Singularity
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: pipeline_template.yml
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: docs/images/nf-core-circrna_logo.png
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: lib/Utils.groovy
files_exist - File not found check: lib/WorkflowMain.groovy
files_exist - File not found check: lib/NfcoreTemplate.groovy
files_exist - File not found check: lib/WorkflowCircrna.groovy
files_exist - File not found check: lib/nfcore_external_java_deps.jar
files_exist - File not found check: .travis.yml
nextflow_config - Config variable found: manifest.name
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: manifest.homePage
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: params.validationShowHiddenParams
nextflow_config - Config variable found: params.validationSchemaIgnoreParams
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config manifest.name began with nf-core/
nextflow_config - Config variable manifest.homePage began with https://github.com/nf-core/
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config manifest.version ends in dev: dev
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
nextflow_config - nextflow.config contains configuration profile test
nextflow_config - Config default value correct: params.tool= circexplorer2
nextflow_config - Config default value correct: params.module= circrna_discovery
nextflow_config - Config default value correct: params.bsj_reads= 0
nextflow_config - Config default value correct: params.tool_filter= 0
nextflow_config - Config default value correct: params.duplicates_fun= mean
nextflow_config - Config default value correct: params.save_intermediates= false
nextflow_config - Config default value correct: params.exon_boundary= 200
nextflow_config - Config default value correct: params.sjdboverhang= 100
nextflow_config - Config default value correct: params.chimJunctionOverhangMin= 10
nextflow_config - Config default value correct: params.alignSJDBoverhangMin= 10
nextflow_config - Config default value correct: params.chimSegmentMin= 10
nextflow_config - Config default value correct: params.seglen= 25
nextflow_config - Config default value correct: params.min_intron= 20
nextflow_config - Config default value correct: params.max_intron= 1000000
nextflow_config - Config default value correct: params.min_map_len= 40
nextflow_config - Config default value correct: params.min_fusion_distance= 200
nextflow_config - Config default value correct: params.save_unaligned= false
nextflow_config - Config default value correct: params.save_reference= true
nextflow_config - Config default value correct: params.hisat2_build_memory= 200.GB
nextflow_config - Config default value correct: params.skip_trimming= false
nextflow_config - Config default value correct: params.save_trimmed= false
nextflow_config - Config default value correct: params.skip_fastqc= false
nextflow_config - Config default value correct: params.min_trimmed_reads= 10000
nextflow_config - Config default value correct: params.custom_config_version= master
nextflow_config - Config default value correct: params.custom_config_base= https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Config default value correct: params.max_cpus= 50
nextflow_config - Config default value correct: params.max_memory= 300.GB
nextflow_config - Config default value correct: params.max_time= 240.h
nextflow_config - Config default value correct: params.publish_dir_mode= copy
nextflow_config - Config default value correct: params.max_multiqc_email_size= 25.MB
nextflow_config - Config default value correct: params.validate_params= true
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - CODE_OF_CONDUCT.md matches the template
files_unchanged - LICENSE matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/CONTRIBUTING.md matches the template
files_unchanged - .github/ISSUE_TEMPLATE/bug_report.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/config.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/PULL_REQUEST_TEMPLATE.md matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - .github/workflows/linting.yml matches the template
files_unchanged - assets/email_template.html matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - assets/sendmail_template.txt matches the template
files_unchanged - assets/nf-core-circrna_logo_light.png matches the template
files_unchanged - docs/images/nf-core-circrna_logo_light.png matches the template
files_unchanged - docs/images/nf-core-circrna_logo_dark.png matches the template
files_unchanged - docs/README.md matches the template
files_unchanged - .gitignore matches the template
files_unchanged - .prettierignore matches the template
files_unchanged - pyproject.toml matches the template
actions_ci - '.github/workflows/ci.yml' is triggered on expected events
actions_ci - '.github/workflows/ci.yml' checks minimum NF version
actions_awstest - '.github/workflows/awstest.yml' is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml does not use -profile test
readme - README Nextflow minimum version badge matched config. Badge: 23.04.0, Config: 23.04.0
pipeline_todos - No TODO strings found
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (300 files)
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
schema_params - Schema matched params returned from nextflow config
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: branch.yml
actions_schema_validation - Workflow validation passed: awstest.yml
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: fix-linting.yml
actions_schema_validation - Workflow validation passed: awsfulltest.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
actions_schema_validation - Workflow validation passed: download_pipeline.yml
actions_schema_validation - Workflow validation passed: release-announcements.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
multiqc_config - 'assets/multiqc_config.yml' contains report_section_order
multiqc_config - 'assets/multiqc_config.yml' contains export_plots
multiqc_config - 'assets/multiqc_config.yml' contains report_comment
multiqc_config - 'assets/multiqc_config.yml' follows the ordering scheme of the minimally required plugins.
multiqc_config - 'assets/multiqc_config.yml' contains a matching 'report_comment'.
multiqc_config - 'assets/multiqc_config.yml' contains 'export_plots: true'.
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'

Run details

nf-core/tools version 2.13.1
Run at 2024-03-01 12:19:57

MariekeVromman

Looks good to me, thanks for the work!

modules/local/annotation/full_annotation/main.nf

Co-authored-by: Maxime U Garcia <max.u.garcia@gmail.com>

MariekeVromman self-requested a review February 2, 2024 08:55

nictru mentioned this pull request Feb 2, 2024

Annotation parallelization #84

Closed

MariekeVromman approved these changes Feb 5, 2024

View reviewed changes

MariekeVromman mentioned this pull request Feb 6, 2024

Implement improved annotation of hits also in the DIFFERENTIAL_EXPRESSION:PARENT_GENE process #97

Open

maxulysse reviewed Feb 14, 2024

View reviewed changes

modules/local/annotation/full_annotation/main.nf Outdated Show resolved Hide resolved

maxulysse reviewed Feb 14, 2024

View reviewed changes

modules/local/annotation/full_annotation/main.nf Outdated Show resolved Hide resolved

nictru and others added 6 commits March 1, 2024 12:29

Implement pandas-based annotation

93c2881

Implement handling of intergenic circRNAs

cfee6c7

Fix linting problems

3e5f72a

Implement handling of "gene" rows in annotation

740d17a

Remove quay.io prefix from container definition

686e40f

Co-authored-by: Maxime U Garcia <max.u.garcia@gmail.com>

Implement capture of versions in annotation

49f47ca

nictru force-pushed the parallel-annotation branch from 6d98bff to 49f47ca Compare March 1, 2024 11:29

nictru added 3 commits March 1, 2024 12:31

Fix linting

fb6a1d9

Fix problem with annotation.py script calling

cf46f11

Fix problem with escape characters in dedicated python script

2de5b60

maxulysse approved these changes Mar 1, 2024

View reviewed changes

nictru merged commit d7fe40f into dev Mar 1, 2024
4 checks passed

nictru deleted the parallel-annotation branch March 1, 2024 13:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Impove annotation performance #95

Impove annotation performance #95

nictru commented Feb 1, 2024

github-actions bot commented Feb 1, 2024 •

edited

❗ Test warnings:

✅ Tests passed:

Run details

MariekeVromman left a comment

Impove annotation performance #95

Impove annotation performance #95

Conversation

nictru commented Feb 1, 2024

github-actions bot commented Feb 1, 2024 • edited

nf-core lint overall result: Passed ✅ ⚠️

❗ Test warnings:

✅ Tests passed:

Run details

MariekeVromman left a comment

Choose a reason for hiding this comment

github-actions bot commented Feb 1, 2024 •

edited

`nf-core lint` overall result: Passed ✅ ⚠️