[SPARK-17791][SQL] Join reordering using star schema detection #15363

ioana-delaney · 2016-10-05T22:41:28Z

What changes were proposed in this pull request?

Star schema consists of one or more fact tables referencing a number of dimension tables. In general, queries against star schema are expected to run fast because of the established RI constraints among the tables. This design proposes a join reordering based on natural, generally accepted heuristics for star schema queries:

Finds the star join with the largest fact table and places it on the driving arm of the left-deep join. This plan avoids large tables on the inner, and thus favors hash joins.
Applies the most selective dimensions early in the plan to reduce the amount of data flow.

The design document was included in SPARK-17791.

Link to the google doc: StarSchemaDetection

How was this patch tested?

A new test suite StarJoinSuite.scala was implemented.

gatorsmile · 2016-10-05T22:43:39Z

ok to test

gatorsmile · 2016-10-05T23:07:52Z

CC @rxin @hvanhovell @cloud-fan @srinathshankar @davies @marmbrus @sameeragarwal @liancheng : )

gatorsmile · 2016-10-05T23:12:10Z

The design doc can be downloaded from the link: https://issues.apache.org/jira/secure/attachment/12831827/StarJoinReordering1005.doc

Below is the slides with the performance number:
https://issues.apache.org/jira/secure/attachment/12829643/StarSchemaJoinReordering.pptx

The performance testing using 1TB TPC-DS workload shows an overall improvement of 19%. Compared to baseline (Negative = improvement; Positive = Degradation):

End to end improved (%)              -19%   
Mean time improved (%)               -19%
Geomean improved (%)                 -24%
End to end improved (seconds)      -3,603
Number of queries improved (>10%)      45
Number of queries degraded (>10%)       6
Number of queries unchanged            48
Top 10 queries improved (%)          -20%

hvanhovell · 2016-10-06T01:15:57Z

ok to test

SparkQA · 2016-10-06T03:30:55Z

Test build #66416 has finished for PR 15363 at commit 518d8e5.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class ReorderJoin(conf: CatalystConf) extends Rule[LogicalPlan] with PredicateHelper
- * Helper case class to hold (plan, size) pairs.

gatorsmile · 2016-10-20T21:44:26Z

sql/core/src/test/scala/org/apache/spark/sql/StarJoinSuite.scala

+  //      |
+  //      s2 - d3
+  // Uses Local Relations to easily control the size of the tables.
+  // e.g. f1 > s2 > d1 > d2 > d3


Here, you might need a description about the snowflake schema.

@gatorsmile I’ve updated the comments and made some changes to the schema.

gatorsmile · 2016-10-20T22:00:10Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala

+   */
+  private def isSelectiveStarJoin(
+      starJoinPlan: Seq[LogicalPlan],
+      conditions: Seq[Expression]): Boolean = {


How about changing the function signature to?

private def isSelectiveStarJoin( factTable: LogicalPlan, dimTables: Seq[LogicalPlan], conditions: Seq[Expression]): Boolean = {

@gatorsmile Thank you for reviewing the changes. I agree with your suggestions. It’s more clear if we pass fact + dimension tables.

SparkQA · 2016-10-24T22:51:17Z

Test build #67474 has finished for PR 15363 at commit 9bddb86.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

davies

This PR looks great overall, the benchmark result is great. Currently we do not have a good cost estimation (may not have in long term due to the fact that Spark SQL is an open engine for many different data sources) that limit the affect from this optimization. I think we should be more defensive to avoid potential regressions (user will see any regression as a blocker for them to use this feature or upgrade). Have you check the queries that regressed in the benchmark? It will be good know that what the cases it make a bad assumption.

davies · 2016-10-27T17:29:37Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala

+      val predicates = splitConjunctivePredicates(filterCond).filter(canEvaluate(_, t))
+      Some(t, predicates)
+
+    case Filter(filterCond, p @ Project(_, t: LeafNode)) =>


Can we use the pattern recursively to avoid these combinations?

case t: LeafNode => case Project(_, BaseTableAccess(t, cons)) => case Filter(c, BaseTableAccess(t, cons)) =>

Yes, I will look into that. Thank you.

davies · 2016-10-27T17:34:34Z

sql/core/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala

+  val STARJOIN_OPTIMIZATION = SQLConfigBuilder("spark.sql.starJoinOptimization")
+    .doc("When true, it enables join reordering based on star schema detection. ")
+    .booleanConf
+    .createWithDefault(false)


Should we use a internal config and enable this by default to have better test coverage?

Do you suggest to have another internal config that will be enabled for testing purposes?

I means make this config internal, and true by default, if we got enough confidence.

@ioana-delaney Could you address @davies 's comment? Thanks!

davies · 2016-10-27T17:36:25Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala

-      createOrderedJoin(input, conditions)
+    case ExtractFiltersAndInnerJoins(input, conditions)
+      if input.size >= 2 && conditions.nonEmpty =>
+      val starJoinPlan = findEligibleStarJoinPlan(input, input, conditions)


Should we put this behind the feature flag (in case that we have a bug in it, we could use the feature flag to workaround it)?

I agree. I will change that.

davies · 2016-10-27T17:46:52Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala

+          // Return an empty plan list and fall back to the default join reordering.
+          Seq.empty[(LogicalPlan, InnerLike)]
+
+        case table1 :: table2 :: _ if table1.size == table2.size =>


Should we replace this equality check with a approximate one?

The size of bytes or cardinality are usually not accurate, should we just use the scale (log of size) instead of accurate size?

The “size” should represent table cardinality after applying the pushed down local predicates i.e. num_rows*selectivity. Temporarily, I used the sizeinBytes value since the join strategies are also using this value to make planning decisions. Long term, a fact table will be determined based on the referential integrity constraints with the other tables. Then, the star join will be planned based on joins’ selectivity.

davies · 2016-10-27T17:47:12Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala

+          Seq.empty[(LogicalPlan, InnerLike)]
+
+        case table1 :: table2 :: _ if table1.size == table2.size =>
+          // There are more tables with the same size. Conservatively, fall back to the


same size => similar size?

Or should we assume that the fact table should be much larger (1+ magnitude) than others?

@davies Yesterday, I forgot to reply to this comment.

Here, we are considering the case when we have multiple fact tables in the query, or a fact table is referenced multiple times. For example, if we have multiple star joins with the fact table referencing the same base table, we cannot make good planning decisions. Therefore, I am conservatively falling back to the positional join.

Similarly, if the query references multiple fact tables that have comparable sizes, we might want to fall back to the positional join. For this case, I also thought of introducing some scale factor, but it's hard to come up with an estimate. I can follow up with some people that have more experience with the warehouse db design and find out what they think.

davies · 2016-10-27T18:03:48Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala

+            // This is a selective star join and all dimensions are base table access.
+            // Compute the size of the dimensions and return the star join
+            // with the most selective dimensions joined lower in the plan.
+            val sortedDims = eligibleDimPlans.map { plan =>


I think the order of dimension table should be based on the selective other than the size of dimension table.

Without a good approximate of selectivity, I'd preserve the order of them so user have the ability to adjust them.

Another thing could be useful is that placing the selective broadcast join before shuffle join.

I agree that the order of the dimensions should be determine based on the join selectivity. Using table size is a temporary approximation. But preserving the order of the dimensions would be a too conservative approach. Based on our performance results in most cases this heuristic led to a good join ordering. In case we make a mistake, we can always switch to the default, positional join order.

I will look into the broadcast vs shuffle join ordering and get back to you.

@davies Sorry for the delay in replying. Regarding the broadcast vs shuffle join comment, I’ve looked at the join strategies. The broadcast join is the default strategy and applies if the inner is smaller than the recommended threshold. Given that the algorithm reorders the dimensions with the smallest dimension lower in the plan, the broadcast join is favored over the repartition/shuffle join. In the future, I assume that the two join alternatives will be evaluated as part of the CBO cost model.

ioana-delaney · 2016-10-28T06:20:01Z

@davies Thank you for reviewing the code! I see this work as evolving and improving with the support of CBO. Without statistics and features such as cardinality and selectivity, we cannot provide an optimal join reordering.

There were two types of regressions. The first type was caused by reordering a non-selective star join. The query did not apply any local predicate on the dimension tables and the join between two large fact tables happen to be very selective. To fix this category of queries, the algorithm will not attempt to reorder a non-selective join. A non-selective join is a join that does not apply local predicates on dimension tables.

The other category of problem was caused by the more general issue of lacking predicate selectivity. To overcome this problem, we introduced the “predicate selectivity hint” feature, to allow the user to specify the selectivity of the predicate. With that, we are able to plan selective dimension first. The JIRA for predicate selectivity was not yet opened.

Then, to further guard against bad plans, we put the feature under the starJoinOptimization option. I was thinking that, to be more conservatives, I can further enforce a certain number of joins in the star. In general, a star join consist of a fact table and at least two dimensions. I can add this restriction to the algorithm.

SparkQA · 2016-11-03T22:25:56Z

Test build #68089 has finished for PR 15363 at commit c21de3e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

ioana-delaney · 2016-11-04T00:35:13Z

retest this please

SparkQA · 2016-11-04T02:57:43Z

Test build #68099 has finished for PR 15363 at commit cca4b9f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

ioana-delaney · 2016-12-15T00:17:31Z

The following updates were made:

Incorporate table and column statistics into the star join detection algorithm. Fact table is chosen based on table cardinality, and dimensions are chosen based on the RI constraints. To infer column uniqueness, the algorithm uses table and column statistics. It compares the number of distinct values with the total number of rows in the table. If their relative difference is within certain limits, the column is assumed to be unique. The updated design document is uploaded to https://issues.apache.org/jira/secure/attachment/12843316/StarJoinReordering1214.doc.
Move star join test cases under Hive test suite, which currently supports statistics.
Rerun TPCDS 1TB with the new table and column statistics. The results are shown in the design doc.

@wzhfy and @davies Would you please review the changes?

SparkQA · 2016-12-15T02:56:26Z

Test build #70159 has finished for PR 15363 at commit 9151a13.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-01-04T01:43:08Z

Test build #70839 has finished for PR 15363 at commit ed46536.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

soubhik-c · 2017-01-04T12:41:08Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala

@@ -42,7 +366,7 @@ object ReorderJoin extends Rule[LogicalPlan] with PredicateHelper {
   * @param conditions a list of condition for join.
   */
  @tailrec
-  def createOrderedJoin(input: Seq[(LogicalPlan, InnerLike)], conditions: Seq[Expression])
+  private def createOrderedJoin(input: Seq[(LogicalPlan, InnerLike)], conditions: Seq[Expression])


can we avoid making it private ? Like we in snappydata plugin external rules for join order and utilize this from that other rule. I suppose there might be others too.

The compiler will complain if the method is public. But we can keep it final.

SparkQA · 2017-03-07T06:36:19Z

Test build #74068 has finished for PR 15363 at commit 072e3a9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

ioana-delaney · 2017-03-07T17:24:37Z

@gatorsmile @wzhfy Would you please review this PR. Thank you.

gatorsmile · 2017-03-07T17:44:48Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala

+   * 2) If the top largest tables have comparable number of rows, fall back to the default
+   *    join reordering. This will prevent changing the position of the large tables in the join.
+   */
+  def findStarJoinPlan(


Nit: -> private def

@gatorsmile The star join is called from join reordering.

gatorsmile · 2017-03-07T17:46:13Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala

+        case Nil =>
+          emptyStarJoinPlan
+        case table1 :: table2 :: _ if table2.size.get.toDouble >
+            conf.starJoinFactTableRatio * table1.size.get.toDouble =>


Nit: style issue.

case table1 :: table2 :: _ if table2.size.get.toDouble > conf.starJoinFactTableRatio * table1.size.get.toDouble => // The largest tables have comparable number of rows. emptyStarJoinPlan

@gatorsmile Done.

gatorsmile · 2017-03-07T17:49:15Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala

+
+          // Verify if the join columns have valid statistics
+          val areStatsAvailable = allFactJoins.forall { plan =>
+            val dimTable = plan._1


I found you used plan._1 multiple times. We prefer to using another way:

val areStatsAvailable = allFactJoins.forall { case (dimTable, _) =>

@gatorsmile Done.

gatorsmile · 2017-03-07T17:57:13Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala

+  /**
+   * Computes table cardinality after applying the predicates.
+   * Currently, the function returns table cardinality.
+   * When predicate selectivity is implemented in Catalyst,


Is it possible we can use the work in the resolved JIRA SPARK-17075: Cardinality Estimation of Predicate Expressions?

@gatorsmile Yes, thank you. I forgot about the recent cbo cardinality changes. I've incorporated them.

gatorsmile · 2017-03-07T17:59:00Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala

+                  val distinctCount = colStats.get.distinctCount
+                  val relDiff = math.abs((distinctCount.toDouble / rowCount.toDouble) - 1.0d)
+                  // ndvMaxErr adjusted based on TPCDS 1TB data results
+                  if (relDiff <= conf.ndvMaxError * 2) true else false


This line can be simplified to relDiff <= conf.ndvMaxError * 2

SparkQA · 2017-03-19T06:00:13Z

Test build #74804 has finished for PR 15363 at commit 1f6a3d6.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class StarSchemaDetection(conf: SQLConf) extends PredicateHelper
case class ReorderJoin(conf: SQLConf) extends Rule[LogicalPlan] with PredicateHelper

… dropped ### What changes were proposed in this pull request? This PR is to fix the following test failure in maven and the PR apache#15363. > org.apache.spark.sql.hive.orc.OrcSourceSuite SPARK-19459/SPARK-18220: read char/varchar column written by Hive The[ test history](https://spark-tests.appspot.com/test-details?suite_name=org.apache.spark.sql.hive.orc.OrcSourceSuite&test_name=SPARK-19459%2FSPARK-18220%3A+read+char%2Fvarchar+column+written+by+Hive) shows all the maven builds failed this test case with the same error message. ``` FAILED: SemanticException [Error 10072]: Database does not exist: db2 org.apache.spark.sql.execution.QueryExecutionException: FAILED: SemanticException [Error 10072]: Database does not exist: db2 at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$runHive$1.apply(HiveClientImpl.scala:637) at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$runHive$1.apply(HiveClientImpl.scala:621) at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:288) at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:229) at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:228) at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:271) at org.apache.spark.sql.hive.client.HiveClientImpl.runHive(HiveClientImpl.scala:621) at org.apache.spark.sql.hive.client.HiveClientImpl.runSqlHive(HiveClientImpl.scala:611) at org.apache.spark.sql.hive.orc.OrcSuite$$anonfun$7.apply$mcV$sp(OrcSourceSuite.scala:160) at org.apache.spark.sql.hive.orc.OrcSuite$$anonfun$7.apply(OrcSourceSuite.scala:155) at org.apache.spark.sql.hive.orc.OrcSuite$$anonfun$7.apply(OrcSourceSuite.scala:155) at org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22) at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85) at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104) at org.scalatest.Transformer.apply(Transformer.scala:22) at org.scalatest.Transformer.apply(Transformer.scala:20) at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:166) at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:68) at org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:163) at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175) ``` ### How was this patch tested? N/A Author: Xiao Li <gatorsmile@gmail.com> Closes apache#17344 from gatorsmile/testtest.

…eorder.

…order.

…r suite.

ioana-delaney · 2017-03-20T02:22:07Z

@gatorsmile @cloud-fan I rewrote the test cases to align to the join reorder suite. Please take a look. Thanks.

SparkQA · 2017-03-20T03:37:01Z

Test build #74842 has finished for PR 15363 at commit 891813f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

ioana-delaney · 2017-03-20T03:43:33Z

retest this please

SparkQA · 2017-03-20T06:10:45Z

Test build #74847 has finished for PR 15363 at commit 891813f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2017-03-20T08:02:52Z

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/StarJoinReorderSuite.scala

+    //  and d3_fk1 = s3_pk1
+    //
+    // Default join reordering: d1, f1, d2, d3, s3
+    // Star join reordering: f1, d1, d3, d2,, d3


2 d3, typo?

@cloud-fan It's a typo. I will fix in my next PR.

cloud-fan · 2017-03-20T08:03:53Z

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/StarJoinReorderSuite.scala

+    //  and d3_fk1 = s3_pk1
+    //
+    // Default join reordering: d1, f1, d2, d3, s3
+    // Star join reordering: f1, d1, d3, d2, d3


the last d3 should be s3

@cloud-fan Yes, it's a typo like above. I did some small changes to the queries when I rewrote the test suite and didn't update the code comments properly. I will fix. Thanks!

cloud-fan · 2017-03-20T08:07:34Z

thanks, merging to master!

The next step is consolidating this with CBO, looking forward to it :)

gatorsmile reviewed Oct 20, 2016

View reviewed changes

ioana-delaney force-pushed the starJoinReord2 branch from 518d8e5 to 9bddb86 Compare October 24, 2016 20:41

davies suggested changes Oct 27, 2016

View reviewed changes

ioana-delaney force-pushed the starJoinReord2 branch from 9bddb86 to c21de3e Compare November 3, 2016 21:17

ioana-delaney force-pushed the starJoinReord2 branch from c21de3e to cca4b9f Compare November 4, 2016 00:37

ioana-delaney force-pushed the starJoinReord2 branch from cca4b9f to 9151a13 Compare December 15, 2016 00:01

ioana-delaney force-pushed the starJoinReord2 branch from 9151a13 to ed46536 Compare January 3, 2017 23:20

soubhik-c reviewed Jan 4, 2017

View reviewed changes

ioana-delaney force-pushed the starJoinReord2 branch from ed46536 to 072e3a9 Compare March 7, 2017 04:23

gatorsmile reviewed Mar 7, 2017

View reviewed changes

ioana-delaney added 17 commits March 19, 2017 18:03

[SPARK-17791] Join reordering using star schema detection.

77ec528

[SPARK-17791] Incorporate review comments.

2e3a54b

[SPARK-17791] Rebase and address review comments.

061d3bc

[SPARK-17791] Incorporate statistics.

49a94e7

[SPARK-17791] Rebase and incorporate statistics.

b8c8152

[SPARK-17791] Rebase and incorporate statistics.

ffe2405

[SPARK-17791] Rebase

323c600

[SPARK-17791] Rebase

9ebc26f

[SPARK-17791] Incorporate comments

a40e6ef

[SPARK-17791] Incorporate comments

d4df989

[SPARK-17791] Split method to call from CBO.

0c0c699

[SPARK-17791] Rebase based on recent SQLConf changes.

3a472b9

[SPARK-17791] Rebase and remove unused star-join call from CostBasedR…

2741ed0

…eorder.

[SPARK-17791] Add TODO for star-join integration into the CostBasedRe…

32d8c09

…order.

[SPARK-17791] Fix code alignment.

e8725bd

[SPARK-17791] Fix code comments.

57d1755

[SPARK-17791] Move test cases under catalyst and align to join reorde…

891813f

…r suite.

ioana-delaney force-pushed the starJoinReord2 branch from 1f6a3d6 to 891813f Compare March 20, 2017 02:15

cloud-fan reviewed Mar 20, 2017

View reviewed changes

asfgit closed this in 8163911 Mar 20, 2017

gatorsmile mentioned this pull request Apr 5, 2017

[SPARK-20231] [SQL] Refactor star schema code for the subsequent star join detection in CBO #17544

Closed

[SPARK-17791][SQL] Join reordering using star schema detection #15363

[SPARK-17791][SQL] Join reordering using star schema detection #15363

Conversation

ioana-delaney commented Oct 5, 2016 • edited

What changes were proposed in this pull request?

How was this patch tested?

gatorsmile commented Oct 5, 2016

gatorsmile commented Oct 5, 2016 • edited

gatorsmile commented Oct 5, 2016 • edited

hvanhovell commented Oct 6, 2016

SparkQA commented Oct 6, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Oct 24, 2016

davies left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ioana-delaney commented Oct 28, 2016

SparkQA commented Nov 3, 2016

ioana-delaney commented Nov 4, 2016

SparkQA commented Nov 4, 2016

ioana-delaney commented Dec 15, 2016

SparkQA commented Dec 15, 2016

SparkQA commented Jan 4, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Mar 7, 2017

ioana-delaney commented Mar 7, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Mar 19, 2017

ioana-delaney commented Mar 20, 2017

SparkQA commented Mar 20, 2017

ioana-delaney commented Mar 20, 2017

SparkQA commented Mar 20, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cloud-fan commented Mar 20, 2017 • edited

ioana-delaney commented Oct 5, 2016 •

edited

gatorsmile commented Oct 5, 2016 •

edited

gatorsmile commented Oct 5, 2016 •

edited

cloud-fan commented Mar 20, 2017 •

edited