Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ADAM-1597] Move to Scala 2.11 and Spark 2.x. #1861

Merged
merged 9 commits into from Jan 9, 2018

Conversation

Projects
3 participants
@heuermh
Copy link
Member

heuermh commented Jan 5, 2018

Fixes #1597.

  • Move poms to Scala 2.11 and Spark 2.x.
  • Remove move_to_scala_2.10.sh and move_to_spark_1.sh scripts.
  • Update jenkins-test not to call move scripts unnecessarily.
  • Update release.sh to only build and release Scala 2.11 and Spark 2.x.
  • Update Scala dependency to the latest 2.11.x.
  • Update Spark dependency to the latest 2.2.x.
  • Update Jenkins build matrix configuration and jenkins-test to match Scala and Spark dependency versions.
  • Update README as necessary.
  • Update documentation as necessary.

I'd like to leave as much of the pom re-writing in place as possible so that when Spark goes to version 3 or starts releasing for Scala 2.12 we'll be able to handle it smoothly.

@fnothaft

This comment has been minimized.

Copy link
Member

fnothaft commented Jan 5, 2018

Thoughts on dropping spark2 from the artifact names? My preference is to nix it, but I can see the argument of keeping them in favor of consistency.

@AmplabJenkins

This comment has been minimized.

Copy link

AmplabJenkins commented Jan 5, 2018

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2543/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1861/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains 296c26c # timeout=10Checking out Revision 296c26c (origin/pr/1861/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 296c26c32f1c9915e66cf98c8c2163f03c249c30First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.2,2.11,1.6.3,centosTriggering ADAM-prb ? 2.7.3,2.10,1.6.3,centosTriggering ADAM-prb ? 2.7.3,2.10,2.2.0,centosTriggering ADAM-prb ? 2.7.3,2.11,1.6.3,centosTriggering ADAM-prb ? 2.6.2,2.10,2.2.0,centosTriggering ADAM-prb ? 2.6.2,2.10,1.6.3,centosTriggering ADAM-prb ? 2.6.2,2.11,2.2.0,centosTriggering ADAM-prb ? 2.7.3,2.11,2.2.0,centosADAM-prb ? 2.6.2,2.11,1.6.3,centos completed with result FAILUREADAM-prb ? 2.7.3,2.10,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.10,2.2.0,centos completed with result FAILUREADAM-prb ? 2.7.3,2.11,1.6.3,centos completed with result FAILUREADAM-prb ? 2.6.2,2.10,2.2.0,centos completed with result FAILUREADAM-prb ? 2.6.2,2.10,1.6.3,centos completed with result FAILUREADAM-prb ? 2.6.2,2.11,2.2.0,centos completed with result FAILUREADAM-prb ? 2.7.3,2.11,2.2.0,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@heuermh

This comment has been minimized.

Copy link
Member Author

heuermh commented Jan 5, 2018

I was thinking of leaving spark2 in there for when spark3 comes along.

@AmplabJenkins

This comment has been minimized.

Copy link

AmplabJenkins commented Jan 5, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2544/
Test PASSed.

@fnothaft

This comment has been minimized.

Copy link
Member

fnothaft commented Jan 5, 2018

I was thinking of leaving spark2 in there for when spark3 comes along.

SGTM!

@AmplabJenkins

This comment has been minimized.

Copy link

AmplabJenkins commented Jan 5, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2545/
Test PASSed.

@AmplabJenkins

This comment has been minimized.

Copy link

AmplabJenkins commented Jan 5, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2546/
Test PASSed.

@fnothaft

This comment has been minimized.

Copy link
Member

fnothaft commented Jan 5, 2018

@heuermh I've bumped the Jenkins build version for you.

@AmplabJenkins

This comment has been minimized.

Copy link

AmplabJenkins commented Jan 5, 2018

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2547/

Build result: FAILURE

[...truncated 7 lines...] > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1861/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains c0ac0b4 # timeout=10Checking out Revision c0ac0b4 (origin/pr/1861/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f c0ac0b4b1b3aca89c62d2b43e75e1e1f523db678First time build. Skipping changelog.Triggering ADAM-prb ? 2.7.3,2.11,2.2.1,centosTriggering ADAM-prb ? 2.6.2,2.10,2.2.1,centosTriggering ADAM-prb ? 2.7.3,2.10,2.2.1,centosTriggering ADAM-prb ? 2.6.2,2.11,2.2.1,centosADAM-prb ? 2.7.3,2.11,2.2.1,centos completed with result FAILUREADAM-prb ? 2.6.2,2.10,2.2.1,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.10,2.2.1,centos completed with result SUCCESSADAM-prb ? 2.6.2,2.11,2.2.1,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@heuermh

This comment has been minimized.

Copy link
Member Author

heuermh commented Jan 5, 2018

Looks like the cloudfront URLs we were using to bypass the Apache mirror script aren't working for Spark 2.2.1.

$ wget http://d3kbcqa49mib13.cloudfront.net/spark-2.2.1-bin-hadoop2.7.tgz
--2018-01-05 15:12:25--  http://d3kbcqa49mib13.cloudfront.net/spark-2.2.1-bin-hadoop2.7.tgz
Resolving d3kbcqa49mib13.cloudfront.net (d3kbcqa49mib13.cloudfront.net)... 52.222.218.79, 52.222.218.77, 52.222.218.190, ...
Connecting to d3kbcqa49mib13.cloudfront.net (d3kbcqa49mib13.cloudfront.net)|52.222.218.79|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2018-01-05 15:12:25 ERROR 404: Not Found.

$ wget http://d3kbcqa49mib13.cloudfront.net/spark-2.2.1-bin-hadoop2.6.tgz
--2018-01-05 15:12:29--  http://d3kbcqa49mib13.cloudfront.net/spark-2.2.1-bin-hadoop2.6.tgz
Resolving d3kbcqa49mib13.cloudfront.net (d3kbcqa49mib13.cloudfront.net)... 52.222.218.79, 52.222.218.77, 52.222.218.190, ...
Connecting to d3kbcqa49mib13.cloudfront.net (d3kbcqa49mib13.cloudfront.net)|52.222.218.79|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2018-01-05 15:12:29 ERROR 404: Not Found.
@fnothaft

This comment has been minimized.

Copy link
Member

fnothaft commented Jan 5, 2018

Indeed; Spark 2.2.1 is not in the cloudfront bucket. IIRC, they stopped listing the direct download link due to Apache frowning on the process.

@AmplabJenkins

This comment has been minimized.

Copy link

AmplabJenkins commented Jan 6, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2548/
Test PASSed.

@heuermh

This comment has been minimized.

Copy link
Member Author

heuermh commented Jan 6, 2018

I'm not sure what to do with cgcloud.rst. The Input and Output data on HDFS and S3 and later sections are worthwhile, perhaps we modify cgcloud.rst into a generic running on AWS page? I'd like to add EMR doc, but this PR shouldn't be tied to that.

@heuermh heuermh added this to the 0.24.0 milestone Jan 6, 2018

@AmplabJenkins

This comment has been minimized.

Copy link

AmplabJenkins commented Jan 6, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2549/
Test PASSed.

@AmplabJenkins

This comment has been minimized.

Copy link

AmplabJenkins commented Jan 9, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2553/
Test PASSed.

@fnothaft fnothaft merged commit 90572b5 into bigdatagenomics:master Jan 9, 2018

1 of 2 checks passed

Codacy/PR Quality Review Hang in there, Codacy is reviewing your Pull request.
Details
default Merged build finished.
Details
@fnothaft

This comment has been minimized.

Copy link
Member

fnothaft commented Jan 9, 2018

Merged! Thanks @heuermh!

@heuermh heuermh added this to Completed in Release 0.24.0 Jan 9, 2018

@heuermh heuermh deleted the heuermh:drop-spark-1x branch Jan 9, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.