Add inputs and expected outputs for testing the vector short read alignment pipeline

[alimanfoo]

This PR brings in two tab-delimited files to provide links to test inputs and expected outputs for the vector short read alignment pipeline. The files are:

• pipelines/short-read-alignment-vector/fixtures/ag1000g-phase1-minimal/lanelets.tsv - this has six lanelets for two samples, including URLs for data on ENA as fastq
• pipelines/short-read-alignment-vector/fixtures/ag1000g-phase1-minimal/expected_outputs.tsv - this has links to two analysis-read bam files for the same two samples, produced previously using the previous vr-pipe implementation of the pipeline

Resolves #6.

[kbergin]

That's great, thank you Alistair! Just to confirm - would these bams be the output after ValidateSamFile in the spec? I ask because then I imagine we would feel pretty confident using them to run samtools stat and gatk callableloci with the exact same commands in the spec to get variants and metrics to compare to. Does that seem reasonable to you @gbggrant ?

[gbggrant]

Looks good - thanks for pointing to the FASTQs.

[gbggrant]

Yes, that sounds good @kbergin

[alimanfoo]

That's great, thank you Alistair! Just to confirm - would these bams be the output after ValidateSamFile in the spec?

That's right.

I ask because then I imagine we would feel pretty confident using them to run samtools stat and gatk callableloci with the exact same commands in the spec to get variants and metrics to compare to.

Yes that might be a light way to check if the outputs are similar or not.

[tnguyensanger]

I left a comment on the original issue, but I'll also leave a comment on this pull request as well:

There are some possible gotchas when comparing realigned bams from cromwell/WDL pipelines with legacy realigned bams:

The mosquito legacy pipelines would use the bedtools bam2fastq v1.1.0 binary to convert bam to fastq.
https://github.com/malariagen/legacy_pipelines/blob/master/vector-ops/pipelines/vo_agam/vo_agam_setups.pl

https://github.com/wtsi-team112/vr-pipe/blob/master/modules/VRPipe/Steps/bam_to_fastq.pm

They would also shuffle the alignments in the bams before converting to fastq and realigning.

https://github.com/wtsi-team112/vr-pipe/blob/master/modules/VRPipe/Steps/bam_shuffle_by_name.pm

The original reasons are lost to the departed bioinformaticists, but presumably it was done to remove any bias during remapping while calculating insert size. In older aligners, the earlier paired reads in the fastq would dictate the insert size. See https://gatkforums.broadinstitute.org/gatk/discussion/2908/howto-revert-a-bam-file-to-fastq-format and http://seqanswers.com/forums/archive/index.php/t-38985.html.

I am not sure if bam reshuffling is still necessary, but it is something to watch out for when we are comparing realigned bams from the test pipelines with legacy realigned bams.

[kbergin]

Thanks for highlighting this @tnguyensanger !