Update dependency versions, including htsjdk to 2.16.1 and guava to 27.0-jre

[heuermh]

• htsjdk 2.13.2 → 2.16.1 required changes to BlockCompressedOutputStream ctr calls
• Guice 4.1.0 → 4.2.1 required override annotation to configure method
• Guava > 16.0 used to throw runtime exceptions due to classpath conflicts; updating to 27.0-jre seems to work fine now, at least with Hadoop 2.7.5 and Spark 2.3.2
• Hadoop default 2.7.3 → 2.7.5

[coveralls]

Coverage decreased (-0.1%) to 79.142% when pulling 88f4c95 on heuermh:update-dependencies into ee9d73f on bigdatagenomics:master.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2839/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > git rev-parse origin/pr/2072/merge^{commit} # timeout=10 > git branch -a -v --no-abbrev --contains 7d9333f # timeout=10Checking out Revision 7d9333f (origin/pr/2072/merge) > git config core.sparsecheckout # timeout=10 > git checkout -f 7d9333fc33889a3b354095e80c21653cc9ddd9f9First time build. Skipping changelog.Triggering ADAM-prb ? 2.7.3,2.11,2.2.2,ubuntuTriggering ADAM-prb ? 2.6.2,2.11,2.2.2,ubuntuADAM-prb ? 2.7.3,2.11,2.2.2,ubuntu completed with result SUCCESSADAM-prb ? 2.6.2,2.11,2.2.2,ubuntu completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

[heuermh]

Jenkins build fails only with Hadoop 2.6.x

...
E Py4JJavaError: An error occurred while calling o4213.count.
E : java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.mapred.FileInputFormat
E at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:312)
E at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:199)

This is one of the runtime exceptions due to conflicting guava versions mentioned above.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2841/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > git rev-parse origin/pr/2072/merge^{commit} # timeout=10 > git branch -a -v --no-abbrev --contains 6a80284 # timeout=10Checking out Revision 6a80284 (origin/pr/2072/merge) > git config core.sparsecheckout # timeout=10 > git checkout -f 6a80284ab1c57999a678eb7efbfe788dede12537First time build. Skipping changelog.Triggering ADAM-prb ? 2.7.3,2.11,2.2.2,ubuntuTriggering ADAM-prb ? 2.6.2,2.11,2.2.2,ubuntuADAM-prb ? 2.7.3,2.11,2.2.2,ubuntu completed with result FAILUREADAM-prb ? 2.6.2,2.11,2.2.2,ubuntu completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2842/
Test PASSed.

[akmorrow13]

LGTM after a rebase

[heuermh]

Per #2073 I think we'll drop Hadoop 2.6 support instead. I'll rebase without the additional commits.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2844/
Test PASSed.

[heuermh]

This works for me:
Spark 2.2.2 in Jenkins
Spark 2.3.2 standalone
CDH Spark 2.2.0 with --packages org.apache.parquet:parquet-avro:1.8.3
CDH Spark 2.2.0 with --packages org.apache.parquet:parquet-avro:1.8.2

[akmorrow13]

@heuermh just to clarify from above:

CDH Spark 2.2.0 with --packages org.apache.parquet:parquet-avro:1.8.3
CDH Spark 2.2.0 with --packages org.apache.parquet:parquet-avro:1.8.2

Is it 1.8.2 or 1.8.3 for CDH Spark 2.2.0?

[heuermh]

Is it 1.8.2 or 1.8.3 for CDH Spark 2.2.0?

They both work, as far as I can tell. I ran through a bunch of Parquet-related operations in adam-shell and didn't run into any problems.

[akmorrow13]

@heuermh can you rebase this on master?

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2847/
Test PASSed.

[akmorrow13]

Thanks @heuermh!