Update Consensus Genomes workflow to handle Oxford Nanopore data

[kislyuk]

This builds on @katrinakalantar's prototyping branch, https://github.com/chanzuckerberg/idseq-workflows/tree/kkalantar-prototype-ont-cg.

Notes from sync with @katrinakalantar:

• primer_schemes/nCoV-2019.reference.fasta must be used as reference fasta when in ONT mode for consistency with ARTIC internal tooling.
• Quast input fastqs correct value is fastqs = RemoveHost.host_removed_fastqs.
• Unclear why flatten() is being used, check if Tiago knows
• Let's add an input check in the WDL itself or in the validate task to make sure only 1 fastq is passed in ONT mode
• In RemoveHost, let's simplify the InsufficientReads conditional to just look at whether the first filtered fastq is empty
• In RunMinion, need to check if we are emitting the correct sorted bam files
• Need to decide whether it's OK to run artic minion --no-longshot
• FIXME: make explicit conditional on technology not on # of fastqs (@kislyuk)
• In computeStats, remove depths = np.array([0]*pysam.AlignmentFile("~{cleaned_bam}", "rb").lengths[0]) - this branch is only reached on empty input, which should have thrown an error earlier
• Make a TODO to upgrade to latest ARTIC package version when release of variant filtering fix occurs (Release 1.3.0 artic-network/fieldbioinformatics#70)
• Emit minion log, sample_name.minion.log.txt, from RunMinion task
• Use only "~{sample}.pass.vcf" downstream of RunMinion. "~{sample}.merged.vcf" contains unfiltered variants
• Run test samples to calibrate "Normalize" parameter to medaka. Expecting to bump from 200 to 1000 to reduce false positive variants.

[kislyuk]

The diff of the WDL from @katrinakalantar's original prototype can be seen at https://github.com/chanzuckerberg/idseq-workflows/compare/kkalantar-prototype-ont-cg..akislyuk-cg-updates (scroll to consensus-genome/run.wdl, click "Load diff").

[kislyuk]

[Moved notes to PR title]

[katrinakalantar]

These are a bunch of comments - please let me know if I can add any additional context / answer questions on these. Some comments include requested changes.

One outstanding big-picture question (also mentioned in a comment) is around how we want to handle the logic to selectively run VADR for sars-cov-2 given that this workflow will also be used for generic-CG.

[katrinakalantar]

Curious on your thoughts about the importance of adding this functionality / validation in the workflow for v0. Currently this function does not do anything. Particularly curious if we want to add error-handing to ensure that the correct technology type is selected.

[kislyuk]

I think we should keep this task as a placeholder and I absolutely agree we need that logic in here. I'll try to add it myself but may have to offload that task to someone else given the urgency of shipping v0.

[katrinakalantar]

see suggestion above that we might remove this optional parameter and be more explicit about not supporting multiple input .fastq files for ONT (at least for v0)

[katrinakalantar]

this is so great to have this validation in place! also really helpful to see how the structure can be used for validation of pipeline output metrics.

[katrinakalantar]

This tipped me off to the fact that we should be replacing *.merged.vcf to *.merged.pass.vcf in the RunMinion step. The number of SNPs should drop substantially after that change.

[katrinakalantar]

These changes look like they address all of the items discussed above!

[katrinakalantar]

One thing to call out explicitly is that this branch will break idseq-workflows for general viral consensus genomes until a subsequent change is made to make the VADR step conditional on reference_genome == sars-cov-2.

[kislyuk]

Thanks, yes, I left a comment to that effect. I will be implementing the relevant logic for general CG, so I'll take care of that change.