New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update dependency versions, including htsjdk to 2.16.1 and guava to 27.0-jre #2072

Merged
merged 1 commit into from Nov 5, 2018

Conversation

Projects
None yet
4 participants
@heuermh
Copy link
Member

heuermh commented Oct 26, 2018

  • htsjdk 2.13.2 → 2.16.1 required changes to BlockCompressedOutputStream ctr calls
  • Guice 4.1.0 → 4.2.1 required override annotation to configure method
  • Guava > 16.0 used to throw runtime exceptions due to classpath conflicts; updating to 27.0-jre seems to work fine now, at least with Hadoop 2.7.5 and Spark 2.3.2
  • Hadoop default 2.7.3 → 2.7.5

@heuermh heuermh added this to the 0.24.1 milestone Oct 26, 2018

@heuermh heuermh requested review from akmorrow13 and fnothaft Oct 26, 2018

@coveralls

This comment has been minimized.

Copy link

coveralls commented Oct 26, 2018

Coverage Status

Coverage decreased (-0.1%) to 79.142% when pulling 88f4c95 on heuermh:update-dependencies into ee9d73f on bigdatagenomics:master.

@AmplabJenkins

This comment has been minimized.

Copy link

AmplabJenkins commented Oct 26, 2018

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2839/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > git rev-parse origin/pr/2072/merge^{commit} # timeout=10 > git branch -a -v --no-abbrev --contains 7d9333f # timeout=10Checking out Revision 7d9333f (origin/pr/2072/merge) > git config core.sparsecheckout # timeout=10 > git checkout -f 7d9333fc33889a3b354095e80c21653cc9ddd9f9First time build. Skipping changelog.Triggering ADAM-prb ? 2.7.3,2.11,2.2.2,ubuntuTriggering ADAM-prb ? 2.6.2,2.11,2.2.2,ubuntuADAM-prb ? 2.7.3,2.11,2.2.2,ubuntu completed with result SUCCESSADAM-prb ? 2.6.2,2.11,2.2.2,ubuntu completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@heuermh

This comment has been minimized.

Copy link
Member

heuermh commented Oct 26, 2018

Jenkins build fails only with Hadoop 2.6.x

...
E Py4JJavaError: An error occurred while calling o4213.count.
E : java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.mapred.FileInputFormat
E at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:312)
E at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:199)

This is one of the runtime exceptions due to conflicting guava versions mentioned above.

@AmplabJenkins

This comment has been minimized.

Copy link

AmplabJenkins commented Oct 26, 2018

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2841/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > git rev-parse origin/pr/2072/merge^{commit} # timeout=10 > git branch -a -v --no-abbrev --contains 6a80284 # timeout=10Checking out Revision 6a80284 (origin/pr/2072/merge) > git config core.sparsecheckout # timeout=10 > git checkout -f 6a80284ab1c57999a678eb7efbfe788dede12537First time build. Skipping changelog.Triggering ADAM-prb ? 2.7.3,2.11,2.2.2,ubuntuTriggering ADAM-prb ? 2.6.2,2.11,2.2.2,ubuntuADAM-prb ? 2.7.3,2.11,2.2.2,ubuntu completed with result FAILUREADAM-prb ? 2.6.2,2.11,2.2.2,ubuntu completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@AmplabJenkins

This comment has been minimized.

Copy link

AmplabJenkins commented Oct 27, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2842/
Test PASSed.

@akmorrow13
Copy link
Contributor

akmorrow13 left a comment

LGTM after a rebase

@heuermh

This comment has been minimized.

Copy link
Member

heuermh commented Oct 28, 2018

Per #2073 I think we'll drop Hadoop 2.6 support instead. I'll rebase without the additional commits.

@heuermh heuermh force-pushed the heuermh:update-dependencies branch from 78de437 to 563485f Oct 28, 2018

@AmplabJenkins

This comment has been minimized.

Copy link

AmplabJenkins commented Oct 28, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2844/
Test PASSed.

@heuermh

This comment has been minimized.

Copy link
Member

heuermh commented Oct 28, 2018

This works for me:
Spark 2.2.2 in Jenkins
Spark 2.3.2 standalone
CDH Spark 2.2.0 with --packages org.apache.parquet:parquet-avro:1.8.3
CDH Spark 2.2.0 with --packages org.apache.parquet:parquet-avro:1.8.2

@akmorrow13

This comment has been minimized.

Copy link
Contributor

akmorrow13 commented Oct 28, 2018

@heuermh just to clarify from above:

CDH Spark 2.2.0 with --packages org.apache.parquet:parquet-avro:1.8.3
CDH Spark 2.2.0 with --packages org.apache.parquet:parquet-avro:1.8.2

Is it 1.8.2 or 1.8.3 for CDH Spark 2.2.0?

@heuermh

This comment has been minimized.

Copy link
Member

heuermh commented Oct 28, 2018

Is it 1.8.2 or 1.8.3 for CDH Spark 2.2.0?

They both work, as far as I can tell. I ran through a bunch of Parquet-related operations in adam-shell and didn't run into any problems.

@akmorrow13

This comment has been minimized.

Copy link
Contributor

akmorrow13 commented Nov 4, 2018

@heuermh can you rebase this on master?

@heuermh heuermh force-pushed the heuermh:update-dependencies branch from 563485f to 88f4c95 Nov 4, 2018

@AmplabJenkins

This comment has been minimized.

Copy link

AmplabJenkins commented Nov 5, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2847/
Test PASSed.

@akmorrow13 akmorrow13 merged commit f3ec463 into bigdatagenomics:master Nov 5, 2018

1 check passed

default Merged build finished.
Details
@akmorrow13

This comment has been minimized.

Copy link
Contributor

akmorrow13 commented Nov 5, 2018

Thanks @heuermh!

@heuermh heuermh deleted the heuermh:update-dependencies branch Nov 5, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment