Standalone variant calling steps + joint discovery

[vdauwera]

Adding WDLs for various component steps of the variant calling pipeline

    - HaplotypeCaller GVCF scatter-gathered across intervals
    - GenotypeGVCFs scatter-gathered across intervals
    - VQSR (no scatter)
    - Joint discovery (GGVCFs through VQSR, no genotype refinement)

[vdauwera]

• HaplotypeCaller test 462f4d14-6bff-4d65-a86f-84b92eddcc72
• GenotypeGVCFs test dd61cfb1-adef-401b-9e28-928047b7a139
• VQSR test bf51923c-448d-4af3-ac20-c14d3becce66
• Joint discovery test ed60e03b-7ae1-425a-a4c9-9e7fe284342b

[vdauwera]

Updating all VQSR steps to use calling intervals for potential efficiency. Shouldn't make a difference but just in case -- it can't hurt.

• VQSR test 167d9c70-1759-48f9-bd79-93508b9303c7
• Joint discovery test 185683e8-1c07-4de0-b292-8c0c646dbd27

[vdauwera]

Done

[vdauwera]

Could I get a documentation review from @sooheelee or @knoblett? Would like to know if it's clear what these do and what they require.

[sooheelee]

Ok, will review the changes.

Too bad we couldn't use the WDL scripts I wrote for the same processes.

[sooheelee]

I'm making comments on the changed files because this is the view that github gives me. This works for new documentation but for changes to documentation, this review approach is not ideal given only changes will be visible. So we should think of a better way to review.

• I've only reviewed the hashtagged comments.
• If a comment applies to another section, I omit it so as to only make a particular point once.
• No comments on formatting.

[sooheelee]

Here, are programs tools?

[vdauwera]

correct

[sooheelee]

I'm going to start using programs instead of tools too then.

[vdauwera]

Actually my "correct" comment was an oversimplification. I should have clarified that I use these terms to distinguish two levels: programs are GATK, Picard, BWA, Cromwell, and tools, such as HaplotypeCaller, are what we invoke in programs that are toolkits, like GATK or Picard. So they're not interchangeable.

[sooheelee]

Change to "unzip block-compressed VCFs with .gz extension as GenotypeGVCFs currently does not handle compressed files."

[vdauwera]

good point, thanks

[vdauwera]

done

[sooheelee]

Why can't we name the gunzipped output to ${unzipped_basename}.g.vcf instead to start with? Then we can avoid the renaming.

[vdauwera]

Mainly because this should be a very temporary situation, so when it's fixed we can just change this bit. In fact I should check whether it's still even the case.

[sooheelee]

I suggest stating only "Unzip GVCFs". Also, perhaps extended explanations, e.g. that GenotypeGVCFs does not handle block compressed files, should all be kept either to the task or to the workflow but not in both.

[vdauwera]

Fair point -- this should probably be only at workflow level

[vdauwera]

done

[sooheelee]

Instead of "execution", use "workflow".

[vdauwera]

I'd rather continue to distinguish the workflow (the thing that gets executed) from its execution.

[sooheelee]

Ok Oliver.

[vdauwera]

Hey :)

[sooheelee]

A link to our documentation explaining GVCF mode may be helpful here.

[vdauwera]

Eh, that's kind of a slippery slope... By the same logic we could link every single tool call to docs. I don't have the stomach to take that on right now (though for the future it would be nice).

[sooheelee]

Ok

[sooheelee]

Below you say WGS and exome files. However, here you only mention WGS.

[vdauwera]

good catch

[vdauwera]

done

[sooheelee]

This section needs more thought.

[vdauwera]

You mean you want to give it more thought, or I should give more details?

[vdauwera]

punting for later -- we can always come back and refine this

[sooheelee]

Incorporate the information in the parentheses into the main body of the text.

[vdauwera]

So make it a separate sentence?

[vdauwera]

done

[sooheelee]

Why do we want to launch tasks that will finish sooner first?

[vdauwera]

The two models are built in parallel, but then the recals are applied in series. With this wiring, we can already start applying Indel recalibration to the VCF while SNP model is still being built. By the time it finishes, the indel-recalibrated file is available to serve as input to apply the SNP recal. If we did it the other way around, we would have to wait until SNP recal file is available despite indel recal file being there already, then apply SNP, then apply Indel. Longer wall clock time for complete workflow execution. Switching the two apply calls bypasses this problem.

[vdauwera]

Added a note to explain this

[sooheelee]

Good to know.

[vdauwera]

Thanks for the review, @sooheelee. I'm going to call this good. We can do some additional refinement (eg add doc links) in a future pass over all our workflows.

[vdauwera]

Btw I'm pretty sure I used your WDLs as basis to develop most of these, @sooheelee. It's been a while since I started this so I don't remember precisely but I try to avoid reinventing wheels.