[SPARK-6566][SQL]: Related changes for newer parquet version #5889

saucam · 2015-05-04T15:13:31Z

This brings in major improvement in that footers are not read on the driver. This also cleans up the code in parquetTableOperations, where we had to override getSplits to eliminate multiple listStatus calls.

cc @liancheng

are there any other changes we need for this ?

AmplabJenkins · 2015-05-04T15:17:10Z

Merged build triggered.

AmplabJenkins · 2015-05-04T15:17:17Z

Merged build started.

SparkQA · 2015-05-04T15:18:53Z

Test build #31760 has started for PR 5889 at commit 3e3cbf9.

SparkQA · 2015-05-04T17:37:53Z

Test build #31760 has finished for PR 5889 at commit 3e3cbf9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-04T17:37:58Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-04T17:37:59Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31760/
Test PASSed.

liancheng · 2015-05-05T01:19:22Z

Thanks for doing this! This has been on my todo list for a long time :) We're currently terribly busy working on features that are expected to be delivered in 1.4 release. Will come back to this once I finish my work at hand.

liancheng · 2015-05-05T01:21:16Z

(I'm so glad that we can finally remove the messy getSplit hacky stuff!)

AmplabJenkins · 2015-05-07T08:07:12Z

Merged build triggered.

AmplabJenkins · 2015-05-07T08:07:22Z

Merged build started.

SparkQA · 2015-05-07T08:08:57Z

Test build #32089 has started for PR 5889 at commit 7e8db22.

SparkQA · 2015-05-07T10:37:06Z

Test build #32089 has finished for PR 5889 at commit 7e8db22.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class SetInFilter[T <: Comparable[T]](

AmplabJenkins · 2015-05-07T10:37:12Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-07T10:37:12Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32089/
Test PASSed.

AmplabJenkins · 2015-05-27T07:02:13Z

Merged build triggered.

AmplabJenkins · 2015-05-27T07:02:21Z

Merged build started.

SparkQA · 2015-05-27T07:04:25Z

Test build #33570 has started for PR 5889 at commit 695f6d9.

SparkQA · 2015-05-27T09:25:38Z

Test build #33570 has finished for PR 5889 at commit 695f6d9.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class SetInFilter[T <: Comparable[T]](

AmplabJenkins · 2015-05-27T09:25:44Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-27T09:25:44Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33570/
Test PASSed.

…n inset into parquet

AmplabJenkins · 2015-06-05T05:22:12Z

Merged build triggered.

AmplabJenkins · 2015-06-05T05:22:19Z

Merged build started.

AmplabJenkins · 2015-06-05T05:37:22Z

Merged build finished. Test FAILed.

saucam · 2015-06-05T05:47:38Z

cc @liancheng

I have rebased.
can we retest this ? How to determine what is failing ?

liancheng · 2015-06-05T16:27:01Z

ok to test

AmplabJenkins · 2015-06-05T16:27:12Z

Merged build triggered.

AmplabJenkins · 2015-06-05T16:27:19Z

Merged build started.

SparkQA · 2015-06-05T16:28:06Z

Test build #34285 has started for PR 5889 at commit c9aa042.

liancheng · 2015-06-05T16:28:13Z

@saucam It seemed to be Jenkins issue rather than the problem of you PR.

SparkQA · 2015-06-05T16:29:45Z

Test build #34285 has finished for PR 5889 at commit c9aa042.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class SetInFilter[T <: Comparable[T]](

AmplabJenkins · 2015-06-05T16:29:45Z

Merged build finished. Test FAILed.

liancheng · 2015-06-05T16:34:02Z

sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetFilters.scala

+      if (value == null) {
+        return false
+      }
+      return hSet.contains(value)


Generally we try to avoid using return in Scala code. (You may find a detailed guide here.) For this method, we'd prefer:

value != null && hSet.contains(value)

BTW, why the name hSet? Maybe valueSet instead?

Oh, just saw the hset field in InSet, still prefer valueSet here though...

JoshRosen · 2015-06-05T16:36:02Z

May want to update the title of this PR to be clearer, since it looks like the dep. bump was done in a separate PR.

saucam · 2015-06-05T17:30:05Z

incorporated review comments

retest please

AmplabJenkins · 2015-06-05T17:32:12Z

Merged build triggered.

AmplabJenkins · 2015-06-05T17:32:19Z

Merged build started.

SparkQA · 2015-06-05T17:33:23Z

Test build #34291 has started for PR 5889 at commit d1bf41e.

SparkQA · 2015-06-05T19:18:53Z

Test build #34291 has finished for PR 5889 at commit d1bf41e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-06-05T19:18:58Z

Merged build finished. Test PASSed.

saucam · 2015-06-11T06:39:57Z

@liancheng looks ok to you now ?

liancheng · 2015-06-12T05:43:27Z

@saucam Sorry for the late reply. This LGTM now. The inefficient code path in Parquet still exists (sequentially retrieving FileStatus), but now it only affects client side metadata retrieving, which is deprecated. So I'm going to merge this to master. Thanks for working on this!

This brings in major improvement in that footers are not read on the driver. This also cleans up the code in parquetTableOperations, where we had to override getSplits to eliminate multiple listStatus calls. cc liancheng are there any other changes we need for this ? Author: Yash Datta <Yash.Datta@guavus.com> Closes apache#5889 from saucam/parquet_1.6 and squashes the following commits: d1bf41e [Yash Datta] SPARK-7340: Fix scalastyle and incorporate review comments c9aa042 [Yash Datta] SPARK-7340: Use the new user defined filter predicate for pushing down inset into parquet 56bc750 [Yash Datta] SPARK-7340: Change parquet version to latest release

saucam force-pushed the parquet_1.6 branch from 7e8db22 to 695f6d9 Compare May 27, 2015 06:57

liancheng mentioned this pull request Jun 3, 2015

[SPARK-7743] [SQL] Parquet 1.7 #6597

Closed

Yash Datta added 2 commits June 5, 2015 10:05

SPARK-7340: Change parquet version to latest release

56bc750

SPARK-7340: Use the new user defined filter predicate for pushing dow…

c9aa042

…n inset into parquet

saucam force-pushed the parquet_1.6 branch from 695f6d9 to c9aa042 Compare June 5, 2015 05:20

liancheng reviewed Jun 5, 2015
View reviewed changes

saucam changed the title ~~[SPARK-6566][SQL]: Change parquet version to latest release~~ [SPARK-6566][SQL]: Related changes for newer parquet version Jun 5, 2015

SPARK-7340: Fix scalastyle and incorporate review comments

d1bf41e

asfgit closed this in e428b3a Jun 12, 2015

[SPARK-6566][SQL]: Related changes for newer parquet version #5889

[SPARK-6566][SQL]: Related changes for newer parquet version #5889

Conversation

saucam commented May 4, 2015

AmplabJenkins commented May 4, 2015

AmplabJenkins commented May 4, 2015

SparkQA commented May 4, 2015

SparkQA commented May 4, 2015

AmplabJenkins commented May 4, 2015

AmplabJenkins commented May 4, 2015

liancheng commented May 5, 2015

liancheng commented May 5, 2015

AmplabJenkins commented May 7, 2015

AmplabJenkins commented May 7, 2015

SparkQA commented May 7, 2015

SparkQA commented May 7, 2015

AmplabJenkins commented May 7, 2015

AmplabJenkins commented May 7, 2015

AmplabJenkins commented May 27, 2015

AmplabJenkins commented May 27, 2015

SparkQA commented May 27, 2015

SparkQA commented May 27, 2015

AmplabJenkins commented May 27, 2015

AmplabJenkins commented May 27, 2015

AmplabJenkins commented Jun 5, 2015

AmplabJenkins commented Jun 5, 2015

AmplabJenkins commented Jun 5, 2015

saucam commented Jun 5, 2015

liancheng commented Jun 5, 2015

AmplabJenkins commented Jun 5, 2015

AmplabJenkins commented Jun 5, 2015

SparkQA commented Jun 5, 2015

liancheng commented Jun 5, 2015

SparkQA commented Jun 5, 2015

AmplabJenkins commented Jun 5, 2015

liancheng Jun 5, 2015

Choose a reason for hiding this comment

liancheng Jun 5, 2015

Choose a reason for hiding this comment

JoshRosen commented Jun 5, 2015

saucam commented Jun 5, 2015

AmplabJenkins commented Jun 5, 2015

AmplabJenkins commented Jun 5, 2015

SparkQA commented Jun 5, 2015

SparkQA commented Jun 5, 2015

AmplabJenkins commented Jun 5, 2015

saucam commented Jun 11, 2015

liancheng commented Jun 12, 2015