[ADAM-1170] Deprecate SAMFileHeaderWritable

[heuermh]

Fixes #1170

[coveralls]

Coverage decreased (-0.04%) to 78.842% when pulling 003aa6c on heuermh:prepare-for-convert into 38f916f on bigdatagenomics:master.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/3006/
Test PASSed.

[akmorrow13]

Some local benchmarks

Ran bam/sam conversions to adam format via adam-submit.
Local performance seems pretty comparable.

800MB bam file

OLD:

real	2m13.281s
user	11m40.431s
sys	0m10.263s

NEW:

real	2m5.253s
user	11m16.020s
sys	0m10.250s

3.6GB SAM file

OLD

real	2m7.400s
user	11m56.519s
sys	0m15.019s

NEW

real	2m5.641s
user	12m35.352s
sys	0m14.541s

[heuermh]

What, did that last run terminate early or something?

[akmorrow13]

Fixed

[heuermh]

Thanks, that looks better. I only expect to see differences (if any) across multi-node clusters, because this pull request changes how the headers are serialized.

Unfortunately, this #2157 prevents running on 2.2.x on the bdg cluster, and we don't have Spark 2.4.x available there yet.

[heuermh]

Here are some results running on (rather undersized) m3.xlarge instances in AWS EMR for 5.7G BAM file on HDFS

git HEAD

$ time ./bin/adam-submit \
  --master yarn \
  --deploy-mode cluster \
  --driver-memory 8g \
  --executor-memory 8g \
  --conf spark.driver.cores=6 \
  --conf spark.executor.cores=6 \
  --conf spark.executor.memoryOverhead=2048 \
  -- \
  transformAlignments \
  NA12878.chrom22.CGI.teramap.CEU.high_coverage.20120417.bam \
  NA12878.chrom22.CGI.teramap.CEU.high_coverage.20120417.alignments.adam

(3 nodes + driver)

real	12m56.854s
user	0m17.041s
sys	0m2.501s

(4 nodes + driver)

real	9m32.446s
user	0m16.184s
sys	0m2.372s

#2156 heuermh:prepare-for-convert branch

$ time ./bin/adam-submit \
  --master yarn \
  --deploy-mode cluster \
  --driver-memory 8g \
  --executor-memory 8g \
  --conf spark.driver.cores=6 \
  --conf spark.executor.cores=6 \
  --conf spark.executor.memoryOverhead=2048 \
  -- \
  transformAlignments \
  NA12878.chrom22.CGI.teramap.CEU.high_coverage.20120417.bam \
  NA12878.chrom22.CGI.teramap.CEU.high_coverage.20120417.2156.alignments.adam

(3 nodes + driver)

real	13m29.480s
user	0m16.762s
sys	0m2.591s

(4 nodes + driver)

real	9m36.646s
user	0m16.260s
sys	0m2.218s

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/3019/
Test PASSed.