Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ADAM-1517] Bump Parquet version in a manner compatible with Spark 2.2.x #1722

Merged
merged 1 commit into from Sep 14, 2017

Conversation

@fnothaft
Copy link
Member

@fnothaft fnothaft commented Sep 13, 2017

Resolves #1517.

@fnothaft fnothaft force-pushed the fnothaft:issues/1517-parquet-bump branch from 9ba3dff to 20fc5ce Sep 13, 2017
@AmplabJenkins
Copy link

@AmplabJenkins AmplabJenkins commented Sep 13, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2367/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1722/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains 3becedc7a249d33c152edb4720fdf091ffdd8eb5 # timeout=10Checking out Revision 3becedc7a249d33c152edb4720fdf091ffdd8eb5 (origin/pr/1722/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 3becedc7a249d33c152edb4720fdf091ffdd8eb5First time build. Skipping changelog.Triggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,2.2.0,centosTriggering ADAM-prb ? 2.3.0,2.10,2.2.0,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.10,2.2.0,centosTriggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,2.2.0,centosADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,2.2.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,2.2.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,2.2.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,2.2.0,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@fnothaft
Copy link
Member Author

@fnothaft fnothaft commented Sep 13, 2017

Whoopsed and forgot to touch up the changes to the move_to_spark1.sh script. Should be good to go now.

@AmplabJenkins
Copy link

@AmplabJenkins AmplabJenkins commented Sep 13, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2368/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1722/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains cca0667 # timeout=10Checking out Revision cca0667 (origin/pr/1722/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f cca066788f7e7f8a8576a71da266aadc1f56562aFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,2.2.0,centosTriggering ADAM-prb ? 2.3.0,2.10,2.2.0,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.10,2.2.0,centosTriggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,2.2.0,centosADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,2.2.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,2.2.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,2.2.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,2.2.0,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@fnothaft
Copy link
Member Author

@fnothaft fnothaft commented Sep 13, 2017

Jenkins, test this please.

@AmplabJenkins
Copy link

@AmplabJenkins AmplabJenkins commented Sep 13, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2369/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1722/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains cca0667 # timeout=10Checking out Revision cca0667 (origin/pr/1722/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f cca0667 > /home/jenkins/git2/bin/git rev-list cca0667 # timeout=10Triggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,2.2.0,centosTriggering ADAM-prb ? 2.3.0,2.10,2.2.0,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.10,2.2.0,centosTriggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,2.2.0,centosADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,2.2.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,2.2.0,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,2.2.0,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,2.2.0,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@fnothaft
Copy link
Member Author

@fnothaft fnothaft commented Sep 13, 2017

Spark 2.2.x drops Hadoop 2.3, which is why that build failed. Otherwise, this looks good on my end.

@fnothaft
Copy link
Member Author

@fnothaft fnothaft commented Sep 14, 2017

Jenkins, test this please.

@AmplabJenkins
Copy link

@AmplabJenkins AmplabJenkins commented Sep 14, 2017

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2370/
Test PASSed.

@heuermh
Copy link
Member

@heuermh heuermh commented Sep 14, 2017

Spark 2.2.x drops Hadoop 2.3, which is why that build failed.

Should we take this opportunity (before the 0.23.0 release) to move 2.6.0 to 2.7.3 in the Jenkins configuration matrix? And bump Spark 1.6.1 to Spark 1.6.3?

Then is there any reason to add Spark 2.1.x? I don't know of any.

pom.xml Outdated
@@ -515,7 +515,13 @@
<dependency>
<groupId>org.apache.parquet</groupId>
<artifactId>parquet-avro</artifactId>
<version>${parquet.version}</version>
<version>1.8.1</version>

This comment has been minimized.

@heuermh

heuermh Sep 14, 2017
Member

Could you add a comment here about intentional pinning to version 1.8.1?

This comment has been minimized.

@fnothaft

fnothaft Sep 14, 2017
Author Member

+1

@heuermh heuermh dismissed their stale review Sep 14, 2017

Should be request changes

@heuermh
Copy link
Member

@heuermh heuermh commented Sep 14, 2017

heuermh dismissed their review 2 minutes from now

Sorry for the noise, that didn't do what I thought it would do.

@heuermh heuermh added this to the 0.23.0 milestone Sep 14, 2017
@fnothaft fnothaft force-pushed the fnothaft:issues/1517-parquet-bump branch from 20fc5ce to c6c5573 Sep 14, 2017
@fnothaft
Copy link
Member Author

@fnothaft fnothaft commented Sep 14, 2017

Rebased and added the comment.

Should we take this opportunity (before the 0.23.0 release) to move 2.6.0 to 2.7.3 in the Jenkins configuration matrix? And bump Spark 1.6.1 to Spark 1.6.3?

+1 and done! Instead of moving 2.6.0, I dropped 2.3.0, added 2.7.3, and bumped 2.6.0 to 2.6.2. I believe there are still some commercial Hadoop distros pushing 2.6.x.

Then is there any reason to add Spark 2.1.x? I don't know of any.

Agreed.

@fnothaft
Copy link
Member Author

@fnothaft fnothaft commented Sep 14, 2017

From a sequencing perspective, let's merge this PR before any other PRs, since I've bumped the jenkins config.

@AmplabJenkins
Copy link

@AmplabJenkins AmplabJenkins commented Sep 14, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2371/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1722/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains 37a5f084d89d099dae506ea111795eac1fd93132 # timeout=10Checking out Revision 37a5f084d89d099dae506ea111795eac1fd93132 (origin/pr/1722/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 37a5f084d89d099dae506ea111795eac1fd93132 > /home/jenkins/git2/bin/git rev-list cca0667 # timeout=10Triggering ADAM-prb ? 2.6.2,2.11,1.6.3,centosTriggering ADAM-prb ? 2.7.3,2.11,1.6.3,centosTriggering ADAM-prb ? 2.7.3,2.11,2.2.0,centosTriggering ADAM-prb ? 2.7.3,2.10,1.6.3,centosTriggering ADAM-prb ? 2.6.2,2.10,1.6.3,centosTriggering ADAM-prb ? 2.6.2,2.10,2.2.0,centosTriggering ADAM-prb ? 2.7.3,2.10,2.2.0,centosTriggering ADAM-prb ? 2.6.2,2.11,2.2.0,centosADAM-prb ? 2.6.2,2.11,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.11,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.11,2.2.0,centos completed with result FAILUREADAM-prb ? 2.7.3,2.10,1.6.3,centos completed with result FAILUREADAM-prb ? 2.6.2,2.10,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.6.2,2.10,2.2.0,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.10,2.2.0,centos completed with result SUCCESSADAM-prb ? 2.6.2,2.11,2.2.0,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@fnothaft fnothaft force-pushed the fnothaft:issues/1517-parquet-bump branch from c6c5573 to 99100ca Sep 14, 2017
@AmplabJenkins
Copy link

@AmplabJenkins AmplabJenkins commented Sep 14, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2374/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1722/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains d39178aaf75e4b103a344d41c5ab4c471e1ab686 # timeout=10Checking out Revision d39178aaf75e4b103a344d41c5ab4c471e1ab686 (origin/pr/1722/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f d39178aaf75e4b103a344d41c5ab4c471e1ab686 > /home/jenkins/git2/bin/git rev-list 37a5f084d89d099dae506ea111795eac1fd93132 # timeout=10Triggering ADAM-prb ? 2.6.2,2.11,1.6.3,centosTriggering ADAM-prb ? 2.7.3,2.11,1.6.3,centosTriggering ADAM-prb ? 2.7.3,2.11,2.2.0,centosTriggering ADAM-prb ? 2.7.3,2.10,1.6.3,centosTriggering ADAM-prb ? 2.6.2,2.10,1.6.3,centosTriggering ADAM-prb ? 2.6.2,2.10,2.2.0,centosTriggering ADAM-prb ? 2.7.3,2.10,2.2.0,centosTriggering ADAM-prb ? 2.6.2,2.11,2.2.0,centosADAM-prb ? 2.6.2,2.11,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.11,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.11,2.2.0,centos completed with result FAILUREADAM-prb ? 2.7.3,2.10,1.6.3,centos completed with result FAILUREADAM-prb ? 2.6.2,2.10,1.6.3,centos completed with result FAILUREADAM-prb ? 2.6.2,2.10,2.2.0,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.10,2.2.0,centos completed with result SUCCESSADAM-prb ? 2.6.2,2.11,2.2.0,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@fnothaft
Copy link
Member Author

@fnothaft fnothaft commented Sep 14, 2017

Looks like an S3 outage...

@fnothaft
Copy link
Member Author

@fnothaft fnothaft commented Sep 14, 2017

Jenkins, test this please.

@AmplabJenkins
Copy link

@AmplabJenkins AmplabJenkins commented Sep 14, 2017

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2375/
Test PASSed.

@heuermh heuermh merged commit 325ee5d into bigdatagenomics:master Sep 14, 2017
2 checks passed
2 checks passed
codacy/pr Good work! A positive pull request.
Details
default Merged build finished.
Details
@heuermh
Copy link
Member

@heuermh heuermh commented Sep 14, 2017

Thank you, @fnothaft

@heuermh heuermh added this to Completed in Release 0.23.0 Jan 4, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants
You can’t perform that action at this time.