[ADAM-1517] Bump Parquet version in a manner compatible with Spark 2.2.x #1722

fnothaft · 2017-09-13T22:08:59Z

Resolves #1517.

AmplabJenkins · 2017-09-13T22:33:48Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2367/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1722/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains 3becedc7a249d33c152edb4720fdf091ffdd8eb5 # timeout=10Checking out Revision 3becedc7a249d33c152edb4720fdf091ffdd8eb5 (origin/pr/1722/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 3becedc7a249d33c152edb4720fdf091ffdd8eb5First time build. Skipping changelog.Triggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,2.2.0,centosTriggering ADAM-prb ? 2.3.0,2.10,2.2.0,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.10,2.2.0,centosTriggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,2.2.0,centosADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,2.2.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,2.2.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,2.2.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,2.2.0,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

fnothaft · 2017-09-13T22:47:07Z

Whoopsed and forgot to touch up the changes to the move_to_spark1.sh script. Should be good to go now.

AmplabJenkins · 2017-09-13T22:55:40Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2368/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1722/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains cca0667 # timeout=10Checking out Revision cca0667 (origin/pr/1722/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f cca066788f7e7f8a8576a71da266aadc1f56562aFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,2.2.0,centosTriggering ADAM-prb ? 2.3.0,2.10,2.2.0,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.10,2.2.0,centosTriggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,2.2.0,centosADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,2.2.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,2.2.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,2.2.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,2.2.0,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

fnothaft · 2017-09-13T23:15:40Z

Jenkins, test this please.

AmplabJenkins · 2017-09-13T23:40:01Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2369/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1722/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains cca0667 # timeout=10Checking out Revision cca0667 (origin/pr/1722/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f cca0667 > /home/jenkins/git2/bin/git rev-list cca0667 # timeout=10Triggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,2.2.0,centosTriggering ADAM-prb ? 2.3.0,2.10,2.2.0,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.10,2.2.0,centosTriggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,2.2.0,centosADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,2.2.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,2.2.0,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,2.2.0,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,2.2.0,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

fnothaft · 2017-09-13T23:43:21Z

Spark 2.2.x drops Hadoop 2.3, which is why that build failed. Otherwise, this looks good on my end.

fnothaft · 2017-09-14T06:58:32Z

Jenkins, test this please.

AmplabJenkins · 2017-09-14T07:25:26Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2370/
Test PASSed.

heuermh · 2017-09-14T15:22:29Z

Spark 2.2.x drops Hadoop 2.3, which is why that build failed.

Should we take this opportunity (before the 0.23.0 release) to move 2.6.0 to 2.7.3 in the Jenkins configuration matrix? And bump Spark 1.6.1 to Spark 1.6.3?

Then is there any reason to add Spark 2.1.x? I don't know of any.

heuermh · 2017-09-14T15:25:02Z

pom.xml

@@ -515,7 +515,13 @@
      <dependency>
        <groupId>org.apache.parquet</groupId>
        <artifactId>parquet-avro</artifactId>
-        <version>${parquet.version}</version>
+        <version>1.8.1</version>


Could you add a comment here about intentional pinning to version 1.8.1?

Should be request changes

heuermh · 2017-09-14T15:29:01Z

heuermh dismissed their review 2 minutes from now

Sorry for the noise, that didn't do what I thought it would do.

fnothaft · 2017-09-14T16:16:29Z

Rebased and added the comment.

Should we take this opportunity (before the 0.23.0 release) to move 2.6.0 to 2.7.3 in the Jenkins configuration matrix? And bump Spark 1.6.1 to Spark 1.6.3?

+1 and done! Instead of moving 2.6.0, I dropped 2.3.0, added 2.7.3, and bumped 2.6.0 to 2.6.2. I believe there are still some commercial Hadoop distros pushing 2.6.x.

Then is there any reason to add Spark 2.1.x? I don't know of any.

Agreed.

fnothaft · 2017-09-14T16:19:24Z

From a sequencing perspective, let's merge this PR before any other PRs, since I've bumped the jenkins config.

AmplabJenkins · 2017-09-14T16:41:26Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2371/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1722/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains 37a5f084d89d099dae506ea111795eac1fd93132 # timeout=10Checking out Revision 37a5f084d89d099dae506ea111795eac1fd93132 (origin/pr/1722/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 37a5f084d89d099dae506ea111795eac1fd93132 > /home/jenkins/git2/bin/git rev-list cca0667 # timeout=10Triggering ADAM-prb ? 2.6.2,2.11,1.6.3,centosTriggering ADAM-prb ? 2.7.3,2.11,1.6.3,centosTriggering ADAM-prb ? 2.7.3,2.11,2.2.0,centosTriggering ADAM-prb ? 2.7.3,2.10,1.6.3,centosTriggering ADAM-prb ? 2.6.2,2.10,1.6.3,centosTriggering ADAM-prb ? 2.6.2,2.10,2.2.0,centosTriggering ADAM-prb ? 2.7.3,2.10,2.2.0,centosTriggering ADAM-prb ? 2.6.2,2.11,2.2.0,centosADAM-prb ? 2.6.2,2.11,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.11,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.11,2.2.0,centos completed with result FAILUREADAM-prb ? 2.7.3,2.10,1.6.3,centos completed with result FAILUREADAM-prb ? 2.6.2,2.10,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.6.2,2.10,2.2.0,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.10,2.2.0,centos completed with result SUCCESSADAM-prb ? 2.6.2,2.11,2.2.0,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

…2.x. Resolves bigdatagenomics#1517.

AmplabJenkins · 2017-09-14T19:13:36Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2374/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1722/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains d39178aaf75e4b103a344d41c5ab4c471e1ab686 # timeout=10Checking out Revision d39178aaf75e4b103a344d41c5ab4c471e1ab686 (origin/pr/1722/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f d39178aaf75e4b103a344d41c5ab4c471e1ab686 > /home/jenkins/git2/bin/git rev-list 37a5f084d89d099dae506ea111795eac1fd93132 # timeout=10Triggering ADAM-prb ? 2.6.2,2.11,1.6.3,centosTriggering ADAM-prb ? 2.7.3,2.11,1.6.3,centosTriggering ADAM-prb ? 2.7.3,2.11,2.2.0,centosTriggering ADAM-prb ? 2.7.3,2.10,1.6.3,centosTriggering ADAM-prb ? 2.6.2,2.10,1.6.3,centosTriggering ADAM-prb ? 2.6.2,2.10,2.2.0,centosTriggering ADAM-prb ? 2.7.3,2.10,2.2.0,centosTriggering ADAM-prb ? 2.6.2,2.11,2.2.0,centosADAM-prb ? 2.6.2,2.11,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.11,1.6.3,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.11,2.2.0,centos completed with result FAILUREADAM-prb ? 2.7.3,2.10,1.6.3,centos completed with result FAILUREADAM-prb ? 2.6.2,2.10,1.6.3,centos completed with result FAILUREADAM-prb ? 2.6.2,2.10,2.2.0,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.10,2.2.0,centos completed with result SUCCESSADAM-prb ? 2.6.2,2.11,2.2.0,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

fnothaft · 2017-09-14T19:17:53Z

Looks like an S3 outage...

fnothaft · 2017-09-14T20:03:08Z

Jenkins, test this please.

AmplabJenkins · 2017-09-14T20:30:25Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2375/
Test PASSed.

heuermh · 2017-09-14T21:14:16Z

Thank you, @fnothaft

fnothaft force-pushed the issues/1517-parquet-bump branch from 9ba3dff to 20fc5ce Compare September 13, 2017 22:30

heuermh previously approved these changes Sep 14, 2017

View reviewed changes

heuermh mentioned this pull request Sep 14, 2017

[ADAM-1517] Move to Parquet 1.8.2 in preparation for moving to Spark 2.2.0 #1518

Closed

heuermh added this to the 0.23.0 milestone Sep 14, 2017

fnothaft force-pushed the issues/1517-parquet-bump branch from 20fc5ce to c6c5573 Compare September 14, 2017 16:13

[ADAM-1517] Bump Parquet version in a manner compatible with Spark 2.…

99100ca

…2.x. Resolves bigdatagenomics#1517.

fnothaft force-pushed the issues/1517-parquet-bump branch from c6c5573 to 99100ca Compare September 14, 2017 18:48

fnothaft mentioned this pull request Sep 14, 2017

add hive style partitioning for contigName #1620

Closed

heuermh approved these changes Sep 14, 2017

View reviewed changes

heuermh merged commit 325ee5d into bigdatagenomics:master Sep 14, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ADAM-1517] Bump Parquet version in a manner compatible with Spark 2.2.x #1722

[ADAM-1517] Bump Parquet version in a manner compatible with Spark 2.2.x #1722

fnothaft commented Sep 13, 2017

AmplabJenkins commented Sep 13, 2017

fnothaft commented Sep 13, 2017

AmplabJenkins commented Sep 13, 2017

fnothaft commented Sep 13, 2017

AmplabJenkins commented Sep 13, 2017

fnothaft commented Sep 13, 2017

fnothaft commented Sep 14, 2017

AmplabJenkins commented Sep 14, 2017

heuermh commented Sep 14, 2017 •

edited

Loading

heuermh Sep 14, 2017

fnothaft Sep 14, 2017

heuermh commented Sep 14, 2017

fnothaft commented Sep 14, 2017

fnothaft commented Sep 14, 2017

AmplabJenkins commented Sep 14, 2017

AmplabJenkins commented Sep 14, 2017

fnothaft commented Sep 14, 2017

fnothaft commented Sep 14, 2017

AmplabJenkins commented Sep 14, 2017

heuermh commented Sep 14, 2017

[ADAM-1517] Bump Parquet version in a manner compatible with Spark 2.2.x #1722

[ADAM-1517] Bump Parquet version in a manner compatible with Spark 2.2.x #1722

Conversation

fnothaft commented Sep 13, 2017

AmplabJenkins commented Sep 13, 2017

Build result: FAILURE

fnothaft commented Sep 13, 2017

AmplabJenkins commented Sep 13, 2017

Build result: FAILURE

fnothaft commented Sep 13, 2017

AmplabJenkins commented Sep 13, 2017

Build result: FAILURE

fnothaft commented Sep 13, 2017

fnothaft commented Sep 14, 2017

AmplabJenkins commented Sep 14, 2017

heuermh commented Sep 14, 2017 • edited Loading

heuermh Sep 14, 2017

Choose a reason for hiding this comment

fnothaft Sep 14, 2017

Choose a reason for hiding this comment

heuermh commented Sep 14, 2017

fnothaft commented Sep 14, 2017

fnothaft commented Sep 14, 2017

AmplabJenkins commented Sep 14, 2017

Build result: FAILURE

AmplabJenkins commented Sep 14, 2017

Build result: FAILURE

fnothaft commented Sep 14, 2017

fnothaft commented Sep 14, 2017

AmplabJenkins commented Sep 14, 2017

heuermh commented Sep 14, 2017

heuermh commented Sep 14, 2017 •

edited

Loading