Sfitz input vcfs #274

sorelfitzgibbon · 2024-02-28T00:58:50Z

Description

This allows the user to input a yaml with 2-4 VCF files for intersection. Some restructuring was involved.
I will update the README in a separate PR.

Testing Results

VCF input

Tumor/Normal Paired Sample:
- sample: DTB-003
- input YAML: /hot/user/jieunoh/su2c/snv_intersect/input_yaml/DTB-003-T.yaml
- config: /hot/user/jieunoh/su2c/snv_intersect/snv_intersect.config
- output: /hot/user/jieunoh/su2c/output/intersected_call_ssnv_vcfs/call-sSNV-8.0.0/DTB-003-T/Intersect-BCFtools-1.17/output

NFtest BAM input

a_mini-all-tools-std-input
output: /hot/software/pipeline/pipeline-call-sSNV/Nextflow/development/unreleased/sfitz-input-vcfs/a_mini-all-tools-std-input
log: /hot/software/pipeline/pipeline-call-sSNV/Nextflow/development/unreleased/sfitz-input-vcfs/log-nftest-20240515T190437Z.log

NFtest VCF input

a_mini-all-tools-vcf-input
output: /hot/software/pipeline/pipeline-call-sSNV/Nextflow/development/unreleased/sfitz-input-vcfs/a_mini-all-tools-vcf-input
-log: /hot/software/pipeline/pipeline-call-sSNV/Nextflow/development/unreleased/sfitz-input-vcfs/log-nftest-20240515T175646Z.log

For some reason the VCF input tests are yielding many of these warnings which seem to resolve themselves:
WARN: Failed to publish file:..

Checklist

I have read the code review guidelines and the code review best practice on GitHub check-list.
I have reviewed the Nextflow pipeline standards.
The name of the branch is meaningful and well formatted following the standards, using [AD_username (or 5 letters of AD if AD is too long)]-[brief_description_of_branch].
I have set up or verified the branch protection rule following the github standards before opening this pull request.
I have added my name to the contributors listings in the manifest block in the nextflow.config as part of this pull request; I am listed already, or do not wish to be listed. (This acknowledgement is optional.)
I have added the changes included in this pull request to the CHANGELOG.md under the next release version or unreleased, and updated the date.
I have updated the version number in the metadata.yaml and manifest block of the nextflow.config file following semver, or the version number has already been updated. (Leave it unchecked if you are unsure about new version number and discuss it with the infrastructure team in this PR.)
I have tested the pipeline on at least one A-mini sample.

…sfitz-input-vcfs

sorelfitzgibbon · 2024-03-21T21:02:17Z

config/methods.config

+    check_valid_algorithms = {
+        valid_algorithms = params.single_NT_paired ? ['somaticsniper', 'strelka2', 'mutect2', 'muse'] : ['mutect2']
+        for (algo in params.algorithm) {
+            if (!(algo in valid_algorithms)) {
+                throw new Exception("ERROR: params.algorithm ${params.algorithm} contains an invalid value. Valid algorithms for given inputs: ${valid_algorithms}")
+            }
+        }
+    }
+


moved from below

config/methods.config

main.nf

github-actions · 2024-05-13T23:52:32Z

Bleep bloop, I am a robot.

Alas, some of the Nextflow configuration tests failed!

test/configtest-F16.json

@ ["params","input_type"]
+ "bam"
@ ["params","keep_input_prefix"]
+ false

test/configtest-F32.json

@ ["params","input_type"]
+ "bam"
@ ["params","keep_input_prefix"]
+ false

If the above changes are surprising, stop and determine what happened.

If the above changes are expected, there are two ways to fix this:

Automatically: Post a comment starting with "/fix-tests" (without the quotes) and I will update the tests for you (you must review my work afterwards).
Manually: Follow these steps on Confluence.

sorelfitzgibbon · 2024-05-14T17:54:15Z

/fix-tests

github-actions · 2024-05-14T17:55:41Z

Bleep bloop, I am a robot.

I have updated all of the failing tests for you with 17159d7. You must review my work before merging this pull request!

github-actions · 2024-05-14T19:44:00Z

Bleep bloop, I am a robot.

Alas, some of the Nextflow configuration tests failed!

test/configtest-F16.json

@ ["params","proc_resource_params","call_sIndel_Manta"]
- {"cpus":"6","memory":"6 GB","retry_strategy":{"memory":{"operand":"3 GB","strategy":"add"}}}
@ ["params","proc_resource_params","call_sSNV_MuSE"]
- {"cpus":"6","memory":"24 GB","retry_strategy":{"memory":{"operand":"8 GB","strategy":"add"}}}
@ ["params","proc_resource_params","call_sSNV_Mutect2"]
- {"cpus":"1","memory":"3 GB","retry_strategy":{"memory":{"operand":"2 GB","strategy":"add"}}}
@ ["params","proc_resource_params","call_sSNV_SomaticSniper"]
- {"cpus":"1","memory":"1 GB","retry_strategy":{"memory":{"operand":"3 GB","strategy":"add"}}}
@ ["params","proc_resource_params","call_sSNV_Strelka2"]
- {"cpus":"6","ext":{"retry_codes":[]},"memory":"2 GB","retry_strategy":{"memory":{"operand":"12 GB","strategy":"add"}}}
@ ["params","proc_resource_params","convert_BAM2Pileup_SAMtools"]
- {"cpus":"1","memory":"1 GB","retry_strategy":{"memory":{"operand":"3 GB","strategy":"add"}}}
@ ["params","proc_resource_params","create_IndelCandidate_SAMtools"]
- {"cpus":"1","memory":"1 GB","retry_strategy":{"memory":{"operand":"3 GB","strategy":"add"}}}
@ ["params","proc_resource_params","run_LearnReadOrientationModel_GATK"]
- {"cpus":"1","memory":"8 GB","retry_strategy":{"memory":{"operand":"2","strategy":"exponential"}}}
@ ["params","proc_resource_params","run_sump_MuSE"]
- {"cpus":"8","memory":"24 GB","retry_strategy":{"memory":{"operand":"8 GB","strategy":"add"}}}
@ ["process","withName:call_sIndel_Manta"]
- {"cpus":"6","memory":{"1":"6 GB","2":"9 GB","3":"12 GB","closure":"retry_updater(6 GB, add, 3 GB, $task.attempt, memory)"}}
@ ["process","withName:call_sSNV_MuSE"]
- {"cpus":"6","memory":{"1":"24 GB","2":"31 GB","3":"31 GB","closure":"retry_updater(24 GB, add, 8 GB, $task.attempt, memory)"}}
@ ["process","withName:call_sSNV_Mutect2"]
- {"cpus":"1","memory":{"1":"3 GB","2":"5 GB","3":"7 GB","closure":"retry_updater(3 GB, add, 2 GB, $task.attempt, memory)"}}
@ ["process","withName:call_sSNV_SomaticSniper"]
- {"cpus":"1","memory":{"1":"1 GB","2":"4 GB","3":"7 GB","closure":"retry_updater(1 GB, add, 3 GB, $task.attempt, memory)"}}
@ ["process","withName:call_sSNV_Strelka2"]
- {"cpus":"6","ext":{"retry_codes":[]},"memory":{"1":"2 GB","2":"14 GB","3":"26 GB","closure":"retry_updater(2 GB, add, 12 GB, $task.attempt, memory)"}}
@ ["process","withName:convert_BAM2Pileup_SAMtools"]
- {"cpus":"1","memory":{"1":"1 GB","2":"4 GB","3":"7 GB","closure":"retry_updater(1 GB, add, 3 GB, $task.attempt, memory)"}}
@ ["process","withName:create_IndelCandidate_SAMtools"]
- {"cpus":"1","memory":{"1":"1 GB","2":"4 GB","3":"7 GB","closure":"retry_updater(1 GB, add, 3 GB, $task.attempt, memory)"}}
@ ["process","withName:run_LearnReadOrientationModel_GATK"]
- {"cpus":"1","memory":{"1":"8 GB","2":"16 GB","3":"31 GB","closure":"retry_updater(8 GB, exponential, 2, $task.attempt, memory)"}}
@ ["process","withName:run_sump_MuSE"]
- {"cpus":"8","memory":{"1":"24 GB","2":"31 GB","3":"31 GB","closure":"retry_updater(24 GB, add, 8 GB, $task.attempt, memory)"}}

test/configtest-F32.json

@ ["params","proc_resource_params","call_sSNV_Strelka2","ext"]
- {"retry_codes":[]}
@ ["process","withName:call_sSNV_Strelka2","ext"]
- {"retry_codes":[]}

If the above changes are surprising, stop and determine what happened.

If the above changes are expected, there are two ways to fix this:

Automatically: Post a comment starting with "/fix-tests" (without the quotes) and I will update the tests for you (you must review my work afterwards).
Manually: Follow these steps on Confluence.

sorelfitzgibbon · 2024-05-14T19:59:33Z

/fix-tests

github-actions · 2024-05-14T20:01:03Z

Bleep bloop, I am a robot.

I have updated all of the failing tests for you with 7808bfd. You must review my work before merging this pull request!

sorelfitzgibbon · 2024-05-14T20:17:27Z

@yashpatel6 this PR is ready for review

sorelfitzgibbon · 2024-05-15T19:32:03Z

@yashpatel6 , new main branch merged in and retested. Ready again for review.

yashpatel6

Added some comments:

config/F16.config

config/custom_schema_types.config

config/mutect-only.config

config/default.config

config/mutect-only.config

input/call-sSNV-template-VCF.yaml

module/mutect2-processes.nf

main.nf

module/vcf-input.nf

…l-sSNV into sfitz-input-vcfs

github-actions · 2024-05-29T20:18:05Z

Bleep bloop, I am a robot.

Alas, some of the Nextflow configuration tests failed!

test/configtest-F16.json

@ ["params","log_output_dir"]
- "/tmp/outputs/call-sSNV-8.0.0/0192847/log-call-sSNV-8.0.0-19970704T165655Z"
+ "/tmp/outputs/call-sSNV-8.1.0/0192847/log-call-sSNV-8.1.0-19970704T165655Z"
@ ["params","save_intermediate_files"]
- true
+ false

test/configtest-F32.json

@ ["params","log_output_dir"]
- "/tmp/outputs/call-sSNV-8.0.0/0192847/log-call-sSNV-8.0.0-19970704T165655Z"
+ "/tmp/outputs/call-sSNV-8.1.0/0192847/log-call-sSNV-8.1.0-19970704T165655Z"
@ ["params","save_intermediate_files"]
- true
+ false

If the above changes are surprising, stop and determine what happened.

If the above changes are expected, there are two ways to fix this:

Automatically: Post a comment starting with "/fix-tests" (without the quotes) and I will update the tests for you (you must review my work afterwards).
Manually: Follow these steps on Confluence.

nwiltsie · 2024-05-29T20:19:20Z

@sorelfitzgibbon hang on, let me fix up the tests with the latest version-ignoring changes!

nwiltsie · 2024-05-29T20:26:12Z

@sorelfitzgibbon okay, I fixed up the version numbering stuff problems with a1a92d6, and that shouldn't be a problem anymore (see uclahs-cds/tool-Nextflow-action#40).

The tests are still failing because params.save_intermediate_files changed - I didn't touch that part.

sorelfitzgibbon · 2024-05-29T21:09:44Z

@sorelfitzgibbon okay, I fixed up the version numbering stuff problems with a1a92d6, and that shouldn't be a problem anymore (see uclahs-cds/tool-Nextflow-action#40).

The tests are still failing because params.save_intermediate_files changed - I didn't touch that part.

Thanks @nwiltsie !

sorelfitzgibbon · 2024-05-29T21:11:25Z

/fix-tests

github-actions · 2024-05-29T21:12:56Z

Bleep bloop, I am a robot.

I have updated all of the failing tests for you with 5480446. You must review my work before merging this pull request!

sorelfitzgibbon · 2024-05-29T21:59:28Z

@yashpatel6 I believe I have addressed all of your concerns.

sorelfitzgibbon · 2024-05-29T23:14:50Z

Retested after merging in submodule updates:
vcf: /hot/software/pipeline/pipeline-call-sSNV/Nextflow/development/unreleased/sfitz-input-vcfs/log-nftest-20240529T230920Z.log
std: /hot/software/pipeline/pipeline-call-sSNV/Nextflow/development/unreleased/sfitz-input-vcfs/log-nftest-20240529T231143Z.log

yashpatel6

A couple of re-organizing comments but otherwise looks good! Once the given parts are moved/removed and the test cases succeed, we can merge

yashpatel6 · 2024-05-30T18:28:49Z

config/custom_schema_types.config

+    /**
+    * Check if proper VCF entry list
+    */
+    check_vcf_list = { Map options, String name, Map properties ->
+        custom_schema_types.check_if_list(options[name], name)
+        for (item in options[name]) {
+            custom_schema_types.check_if_namespace(item, name)
+            properties.elements.each { key, val ->
+                schema.validate_parameter(item, key, val)
+            }
+        }
+    }
+
+    /**
+    * Check that at least one kind of input is given and only one of BAM or VCF is given
+    */
+    check_input_presence_and_exclusivity = { Map options ->
+        def bam_given = options.containsKey('bam')
+        def vcf_given = options.containsKey('vcf')
+
+        if (!bam_given && !vcf_given) {
+            throw new Exception("At least one input type (BAM or VCF) must be provided.")
+        }
+
+        if (bam_given && vcf_given) {
+            throw new Exception("Only one input type (either BAM or VCF) should be provided.")
+        }
+    }
+


I think these two functions aren't being used anywhere?

Yes, check_input_presence_and_exclusivity was accidentally left behind.

I guess check_vcf_list can't be used and isn't necessary because of the way the VCFs are input. I removed the function and the VCFEntryList that referred to it.

yashpatel6 · 2024-05-30T18:29:40Z

config/custom_schema_types.config

@@ -134,6 +198,7 @@ custom_schema_types {
    types = [
        'InputNamespace': custom_schema_types.check_input_namespace,
        'BAMEntryList': custom_schema_types.check_bam_list,
+        'VCFEntryList': custom_schema_types.check_vcf_list,


VCFEntryList is also not being used in the schema I think

yashpatel6 · 2024-05-30T18:38:36Z

module/process-vcfs.nf

+rename_id_ch = Channel.value(['orig_id': params.input_tumor_id,'id': params.tumor_id, 'sample_type': 'tumor' ])
+    .mix(Channel.value(['orig_id': params.input_normal_id, 'id': params.normal_id, 'sample_type': 'normal' ]))
+    .mix(Channel.value(['orig_id': 'TUMOR', 'id': params.tumor_id, 'sample_type': 'tumor' ]))
+    .mix(Channel.value(['orig_id': 'NORMAL', 'id': params.normal_id, 'sample_type': 'normal' ]))
+    .collect()


I suggest moving this into the workflow below as the channel generated is used in the workflow and having the statements outside the workflow could have unexpected channels being created when imports happen

sorelfitzgibbon · 2024-05-30T20:38:13Z

Changes made and successfully re-tested: /hot/software/pipeline/pipeline-call-sSNV/Nextflow/development/unreleased/sfitz-input-vcfs. Will push changes and merge.

sorelfitzgibbon added 4 commits February 13, 2024 21:25

vcf input yaml

fbfb984

overlooked nftest.yml actual paths including tool versions

4eb7988

working but need to add pipeval for input vcfs and tweak log structure

5967e75

add test vcf yaml

47bba4d

sorelfitzgibbon requested review from tyamaguchi-ucla, yashpatel6, maotian06 and a team as code owners February 28, 2024 00:58

sorelfitzgibbon added 2 commits March 11, 2024 15:06

update VCF template yaml

f17d4f6

Merge branch 'main' of github.com:uclahs-cds/pipeline-call-sSNV into …

92752b5

…sfitz-input-vcfs

sorelfitzgibbon commented Mar 21, 2024

View reviewed changes

sorelfitzgibbon added 2 commits March 21, 2024 14:53

update changelog

00a2d82

merge from origin main

13542e1

Autofix Nextflow configuration regression tests

17159d7

sorelfitzgibbon added 4 commits May 14, 2024 12:27

avoid unused process warnings

a4f96cc

add input vcf to nftest

91865cf

update changelog

defe784

Merge branch 'sfitz-input-vcfs' with --no-ff

dc80c74

avoid unused process warnings, all F config

0d22d0f

Autofix Nextflow configuration regression tests

7808bfd

sorelfitzgibbon added 3 commits May 15, 2024 10:50

merge in from main

cebbe15

fix intersection script resources

1ab2631

define input_type before used in F configs

59d0137

yashpatel6 self-assigned this May 22, 2024

yashpatel6 reviewed May 22, 2024

View reviewed changes

sorelfitzgibbon added 8 commits May 29, 2024 13:00

change process name

31e287a

remove unused file

fba06b3

fix input template

a74120d

turn off intermediate output

0511693

consolidate include statements

65cbf52

add input checks

a024846

update configtests

4486115

Merge branch 'sfitz-input-vcfs' of github.com:uclahs-cds/pipeline-cal…

6e6af05

…l-sSNV into sfitz-input-vcfs

Mask version numbers in tests

a1a92d6

Autofix Nextflow configuration regression tests

5480446

sorelfitzgibbon mentioned this pull request May 29, 2024

Create separate workflows for each input type #296

Open

Merge remote-tracking branch 'origin' into sfitz-input-vcfs

56a29fb

yashpatel6 approved these changes May 30, 2024

View reviewed changes

sorelfitzgibbon added 3 commits May 30, 2024 13:40

remove unused functions

11d4c29

move channel creation into workflow

bc570f0

fix changelog

f61eb0f

sorelfitzgibbon merged commit e27b4e7 into main May 30, 2024
7 checks passed

sorelfitzgibbon deleted the sfitz-input-vcfs branch May 30, 2024 20:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sfitz input vcfs #274

Sfitz input vcfs #274

sorelfitzgibbon commented Feb 28, 2024 •

edited

Loading

sorelfitzgibbon Mar 21, 2024

github-actions bot commented May 13, 2024

sorelfitzgibbon commented May 14, 2024

github-actions bot commented May 14, 2024

github-actions bot commented May 14, 2024

sorelfitzgibbon commented May 14, 2024

github-actions bot commented May 14, 2024

sorelfitzgibbon commented May 14, 2024

sorelfitzgibbon commented May 15, 2024

yashpatel6 left a comment

github-actions bot commented May 29, 2024

nwiltsie commented May 29, 2024

nwiltsie commented May 29, 2024

sorelfitzgibbon commented May 29, 2024

sorelfitzgibbon commented May 29, 2024

github-actions bot commented May 29, 2024

sorelfitzgibbon commented May 29, 2024

sorelfitzgibbon commented May 29, 2024

yashpatel6 left a comment

yashpatel6 May 30, 2024

sorelfitzgibbon May 30, 2024

yashpatel6 May 30, 2024

yashpatel6 May 30, 2024

sorelfitzgibbon commented May 30, 2024

Sfitz input vcfs #274

Sfitz input vcfs #274

Conversation

sorelfitzgibbon commented Feb 28, 2024 • edited Loading

Description

Testing Results

VCF input

NFtest BAM input

NFtest VCF input

Checklist

sorelfitzgibbon Mar 21, 2024

Choose a reason for hiding this comment

github-actions bot commented May 13, 2024

test/configtest-F16.json

test/configtest-F32.json

sorelfitzgibbon commented May 14, 2024

github-actions bot commented May 14, 2024

github-actions bot commented May 14, 2024

test/configtest-F16.json

test/configtest-F32.json

sorelfitzgibbon commented May 14, 2024

github-actions bot commented May 14, 2024

sorelfitzgibbon commented May 14, 2024

sorelfitzgibbon commented May 15, 2024

yashpatel6 left a comment

Choose a reason for hiding this comment

github-actions bot commented May 29, 2024

test/configtest-F16.json

test/configtest-F32.json

nwiltsie commented May 29, 2024

nwiltsie commented May 29, 2024

sorelfitzgibbon commented May 29, 2024

sorelfitzgibbon commented May 29, 2024

github-actions bot commented May 29, 2024

sorelfitzgibbon commented May 29, 2024

sorelfitzgibbon commented May 29, 2024

yashpatel6 left a comment

Choose a reason for hiding this comment

yashpatel6 May 30, 2024

Choose a reason for hiding this comment

sorelfitzgibbon May 30, 2024

Choose a reason for hiding this comment

yashpatel6 May 30, 2024

Choose a reason for hiding this comment

yashpatel6 May 30, 2024

Choose a reason for hiding this comment

sorelfitzgibbon commented May 30, 2024

sorelfitzgibbon commented Feb 28, 2024 •

edited

Loading