Md cram #735

FriederikeHanssen · 2022-09-07T11:25:55Z

PR checklist

maxulysse · 2022-09-07T11:35:42Z

Do we still need the BAM_TO_CRAM subworkflow?

FriederikeHanssen · 2022-09-07T11:37:32Z

Do we still need the BAM_TO_CRAM subworkflow?

Yes for the cases where we don't do duplicate marking. It would be simplified though by importing cram_qc in bam_to_cram

maxulysse · 2022-09-07T11:38:30Z

Do we still need the BAM_TO_CRAM subworkflow?

Yes for the cases where we don't do duplicate marking. It would be simplified though by importing cram_qc in bam_to_cram

Good enough for me

FriederikeHanssen · 2022-09-07T11:43:07Z

TODO:

Markduplicates bam output on save_output_bam
Mosdepth has issues
test in CRAM/BAm works when step markduplicates

github-actions · 2022-09-07T11:51:17Z

`nf-core lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit 90d252c

+| ✅ 154 tests passed       |+
#| ❔   4 tests were ignored |#
!| ❗   1 tests had warnings |!

❗ Test warnings:

pipeline_todos - TODO string in methods_description_template.yml: #Update the HTML below to your prefered methods description, e.g. add publication citation for this pipeline

❔ Tests ignored:

files_unchanged - File ignored due to lint config: assets/nf-core-sarek_logo_light.png
files_unchanged - File ignored due to lint config: docs/images/nf-core-sarek_logo_light.png
files_unchanged - File ignored due to lint config: docs/images/nf-core-sarek_logo_dark.png
files_unchanged - File ignored due to lint config: lib/NfcoreTemplate.groovy

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: assets/nf-core-sarek_logo_light.png
files_exist - File found: conf/modules.config
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/images/nf-core-sarek_logo_light.png
files_exist - File found: docs/images/nf-core-sarek_logo_dark.png
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: lib/nfcore_external_java_deps.jar
files_exist - File found: lib/NfcoreSchema.groovy
files_exist - File found: lib/NfcoreTemplate.groovy
files_exist - File found: lib/Utils.groovy
files_exist - File found: lib/WorkflowMain.groovy
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: conf/igenomes.config
files_exist - File found: .github/workflows/awstest.yml
files_exist - File found: .github/workflows/awsfulltest.yml
files_exist - File found: lib/WorkflowSarek.groovy
files_exist - File found: modules.json
files_exist - File found: pyproject.toml
files_exist - File not found check: Singularity
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: docs/images/nf-core-sarek_logo.png
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: .travis.yml
nextflow_config - Config variable found: manifest.name
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: manifest.homePage
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: params.show_hidden_params
nextflow_config - Config variable found: params.schema_ignore_params
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.version
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config manifest.name began with nf-core/
nextflow_config - Config variable manifest.homePage began with https://github.com/nf-core/
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config manifest.version ends in dev: '3.1dev'
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - CODE_OF_CONDUCT.md matches the template
files_unchanged - LICENSE matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/CONTRIBUTING.md matches the template
files_unchanged - .github/ISSUE_TEMPLATE/bug_report.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/config.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/PULL_REQUEST_TEMPLATE.md matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - .github/workflows/linting.yml matches the template
files_unchanged - assets/email_template.html matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - assets/sendmail_template.txt matches the template
files_unchanged - docs/README.md matches the template
files_unchanged - lib/nfcore_external_java_deps.jar matches the template
files_unchanged - lib/NfcoreSchema.groovy matches the template
files_unchanged - .gitignore matches the template
files_unchanged - .prettierignore matches the template
files_unchanged - pyproject.toml matches the template
actions_ci - '.github/workflows/ci.yml' is triggered on expected events
actions_ci - '.github/workflows/ci.yml' checks minimum NF version
actions_awstest - '.github/workflows/awstest.yml' is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml does not use -profile test
readme - README Nextflow minimum version badge matched config. Badge: 21.10.3, Config: 21.10.3
readme - README Nextflow minimum version in Quick Start section matched config. README: 21.10.3, Config: 21.10.3
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (409 files)
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
schema_params - Schema matched params returned from nextflow config
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: awsfulltest_germline.yml
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: fix-linting.yml
actions_schema_validation - Workflow validation passed: awstest.yml
actions_schema_validation - Workflow validation passed: awsfulltest.yml
actions_schema_validation - Workflow validation passed: branch.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
multiqc_config - 'assets/multiqc_config.yml' follows the ordering scheme of the minimally required plugins.
multiqc_config - 'assets/multiqc_config.yml' contains a matching 'report_comment'.
multiqc_config - 'assets/multiqc_config.yml' contains 'export_plots: true'.
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'

Run details

nf-core/tools version 2.6
Run at 2022-10-11 16:51:05

maxulysse

<3

maxulysse · 2022-09-07T15:28:19Z

still a couple of tests to fix 😄

CHANGELOG.md

maxulysse

Love how it's now simplified

FriederikeHanssen · 2022-09-08T12:38:39Z

Love how it's now simplified

therfore I broke now everything 🙈

Co-authored-by: Maxime U. Garcia <maxime.garcia@scilifelab.se>

maxulysse · 2022-09-12T08:22:07Z

nextflow.config

@@ -43,7 +43,7 @@ params {
    // Preprocessing
    aligner            = 'bwa-mem'  // Default is bwa-mem, bwa-mem2 and dragmap can be used too
    use_gatk_spark     = null       // GATK Spark implementation of their tools in local mode not used by default
-    save_bam_mapped    = false      // Mapped BAMs not saved
+    save_mapped    = false      // Mapped BAMs not saved


Suggested change

save_mapped = false // Mapped BAMs not saved

save_mapped = false // Mapped files not saved

maxulysse · 2022-09-12T08:22:21Z

nextflow.config

@@ -43,7 +43,7 @@ params {
    // Preprocessing
    aligner            = 'bwa-mem'  // Default is bwa-mem, bwa-mem2 and dragmap can be used too
    use_gatk_spark     = null       // GATK Spark implementation of their tools in local mode not used by default
-    save_bam_mapped    = false      // Mapped BAMs not saved
+    save_mapped    = false      // Mapped BAMs not saved
    save_output_as_bam = false      //Output files from preprocessing are saved as bam and not as cram files


Suggested change

save_output_as_bam = false //Output files from preprocessing are saved as bam and not as cram files

save_output_as_bam = false // Output files from preprocessing are saved as bam and not as cram files

nextflow_schema.json

tests/test_default.yml

FriederikeHanssen added 2 commits September 7, 2022 13:18

update markduplicates module

2500878

use it in subworkflows

fff88f5

FriederikeHanssen requested a review from maxulysse as a code owner September 7, 2022 11:25

FriederikeHanssen added 2 commits September 7, 2022 13:27

update changelog

2e3431f

merge upstream/dev

d97c228

maxulysse approved these changes Sep 7, 2022

View reviewed changes

prefix must use output file name

708dd6f

maxulysse approved these changes Sep 7, 2022

View reviewed changes

FriederikeHanssen added 10 commits September 7, 2022 22:33

use samtools index, because the md index is not working

5d3fcd9

fix bam output

ec57d1e

check estimatelib can run on cram

5f10df0

merge upstream/dev

07c7ba9

remove merge conflict markers

e8fab9e

remove merge conflict markers

8476c78

fix metrics path

e53164c

fix output names

5eae260

update usage docs

afc794a

update output docs

aa05776

maxulysse reviewed Sep 8, 2022

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

maxulysse approved these changes Sep 8, 2022

View reviewed changes

FriederikeHanssen and others added 3 commits September 8, 2022 14:38

Update CHANGELOG.md

dbdf861

Co-authored-by: Maxime U. Garcia <maxime.garcia@scilifelab.se>

revert module chagnes

b28a3b3

Merge remote-tracking branch 'origin/md_cram' into md_cram

90eed33

maxulysse reviewed Sep 12, 2022

View reviewed changes

nextflow_schema.json Outdated Show resolved Hide resolved

FriederikeHanssen and others added 10 commits September 12, 2022 20:41

fix channel output

33f5843

publish converted cram files to a converted subdir

176e20c

fix file name

655f4b9

address review comments

4651c36

fix md5sums

25e810f

workaround weird meta map thing

8469ec8

checksum fixes

72dcf0b

add more md5sums

011be69

Merge branch 'dev' into md_cram

9caa37c

remove merge conflict markers

63749bf

maxulysse reviewed Sep 23, 2022

View reviewed changes

tests/test_default.yml Outdated Show resolved Hide resolved

maxulysse added this to the 3.1 milestone Sep 26, 2022

FriederikeHanssen added 11 commits October 10, 2022 10:40

Merge upstream/dev

48c70ba

fix paths

8f7605f

remove merge markers

447da0b

fix module imports

2ad3704

add process import back in that I removed when merging

04212c8

fix skip_md tests

9be3627

fix saveoutputbam tests

2e5e484

fix haplotypecaller checksums

b1325d7

some variabel md5sums

e8df4b5

some more checksum checks

2600ba7

Add tests for saveoutputbam + saveoutputmapped & docs

4843179

maxulysse mentioned this pull request Oct 11, 2022

Update all subworkflows name #784

Merged

9 tasks

FriederikeHanssen added 2 commits October 11, 2022 18:10

Merge remote-tracking branch 'upstream/dev' into md_cram

1b6e14d

fix edmunds linting

90d252c

maxulysse merged commit 727cf9c into nf-core:dev Oct 12, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Md cram #735

Md cram #735

FriederikeHanssen commented Sep 7, 2022

maxulysse commented Sep 7, 2022

FriederikeHanssen commented Sep 7, 2022

maxulysse commented Sep 7, 2022

FriederikeHanssen commented Sep 7, 2022 •

edited

github-actions bot commented Sep 7, 2022 •

edited

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

maxulysse left a comment

maxulysse commented Sep 7, 2022

maxulysse left a comment

FriederikeHanssen commented Sep 8, 2022

maxulysse Sep 12, 2022 •

edited

maxulysse Sep 12, 2022

	save_mapped = false // Mapped BAMs not saved
	save_mapped = false // Mapped files not saved

	save_output_as_bam = false //Output files from preprocessing are saved as bam and not as cram files
	save_output_as_bam = false // Output files from preprocessing are saved as bam and not as cram files

Md cram #735

Md cram #735

Conversation

FriederikeHanssen commented Sep 7, 2022

PR checklist

maxulysse commented Sep 7, 2022

FriederikeHanssen commented Sep 7, 2022

maxulysse commented Sep 7, 2022

FriederikeHanssen commented Sep 7, 2022 • edited

github-actions bot commented Sep 7, 2022 • edited

nf-core lint overall result: Passed ✅ ⚠️

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

maxulysse left a comment

Choose a reason for hiding this comment

maxulysse commented Sep 7, 2022

maxulysse left a comment

Choose a reason for hiding this comment

FriederikeHanssen commented Sep 8, 2022

maxulysse Sep 12, 2022 • edited

Choose a reason for hiding this comment

maxulysse Sep 12, 2022

Choose a reason for hiding this comment

FriederikeHanssen commented Sep 7, 2022 •

edited

github-actions bot commented Sep 7, 2022 •

edited

`nf-core lint` overall result: Passed ✅ ⚠️

maxulysse Sep 12, 2022 •

edited