[SPARK-34952][SQL] DSv2 Aggregate push down APIs #33352

huaxingao · 2021-07-15T02:42:21Z

What changes were proposed in this pull request?

Add interfaces and APIs to push down Aggregates to V2 Data Source.
Also add JDBC implementation so we can test the new APIs.
Parquet implementation will be added using a separate PR.

Why are the changes needed?

improve performance

Does this PR introduce any user-facing change?

SupportsPushDownAggregates is added. Data sources can implement this interface to push down aggregates.
JDBC_PUSHDOWN_AGGREGATE is added. If sets to true, we will push down aggregate to JDBC data source.

How was this patch tested?

Add new tests in JDBCV2Suite to test aggregate push down.

SparkQA · 2021-07-15T03:30:18Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45562/

SparkQA · 2021-07-15T04:05:37Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45562/

SparkQA · 2021-07-15T04:58:22Z

Test build #141047 has finished for PR 33352 at commit 0cce896.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class Aggregation(aggregateExpressions: Seq[AggregateFunc],
case class Min(column: Expression, dataType: DataType) extends AggregateFunc
case class Max(column: Expression, dataType: DataType) extends AggregateFunc
case class Sum(column: Expression, dataType: DataType, isDistinct: Boolean)
case class Count(column: Expression, dataType: DataType, isDistinct: Boolean)
case class ScanBuilderHolder(

SparkQA · 2021-07-15T06:32:13Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45568/

SparkQA · 2021-07-15T07:11:51Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45568/

SparkQA · 2021-07-15T10:18:22Z

Test build #141053 has finished for PR 33352 at commit a5386f0.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class Aggregation(aggregateExpressions: Seq[AggregateFunc],
case class Min(column: Expression, dataType: DataType) extends AggregateFunc
case class Max(column: Expression, dataType: DataType) extends AggregateFunc
case class Sum(column: Expression, dataType: DataType, isDistinct: Boolean)
case class Count(column: Expression, dataType: DataType, isDistinct: Boolean)
case class ScanBuilderHolder(

cloud-fan · 2021-07-15T17:35:07Z

sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SupportsPushDownAggregates.java

+
+/**
+ * A mix-in interface for {@link ScanBuilder}. Data source can implement this interface to
+ * push down aggregates. Depends on the data source implementation, the aggregates may not


A mix-in interface for {@link ScanBuilder}. Data sources can implement this interface to push down aggregates. Spark assumes that the data source can't fully complete the grouping work, and will group the data source output again. For queries like "SELECT min(value) AS m FROM t GROUP BY key", after pushing down the aggregate to the data source, the data source can still output data with duplicated keys, which is OK as Spark will do GROUP BY key again. The final query plan can be something like this: {{{ Aggregate [key#1], [min(min(value)#2) AS m#3] +- RelationV2[key#1, min(value)#2] }}} Similarly, if there is no grouping expression, the data source can still output more than one rows.

Also let's use valid Java doc syntax, e.g., add <p> between paragraphs, properly format code blocks, etc.

cloud-fan · 2021-07-15T17:40:24Z

sql/catalyst/src/main/scala/org/apache/spark/sql/connector/expressions/aggregates.scala

+case class Max(column: Expression, dataType: DataType) extends AggregateFunc
+case class Sum(column: Expression, dataType: DataType, isDistinct: Boolean)
+  extends AggregateFunc
+case class Count(column: Expression, dataType: DataType, isDistinct: Boolean)


sorry to change my mind at the last second. I think it's very unlikely that a data source can support something like max(a + b), group by a + b. I think it's clearer to use NamedReference instead of Expression here.

For v2 partitioning, it's always named. e.g. CREATE TABLE ... PARTITIONED BY year(ts), the partitioning has a name and you can get it by DESC TABLE, which calls SupportsPartitionManagement.partitionSchema.

For count(1), let's create a special one class CountOne.

BTW count doesn't need a data type? It always returns long.

cloud-fan · 2021-07-15T17:41:41Z

sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala

@@ -870,6 +870,12 @@ object SQLConf {
      .checkValue(threshold => threshold >= 0, "The threshold must not be negative.")
      .createWithDefault(10)

+  val PARQUET_AGGREGATE_PUSHDOWN_ENABLED = buildConf("spark.sql.parquet.aggregatePushdown")


I think the config is per source. For this API-only PR, we don't need any config.

cloud-fan · 2021-07-15T17:46:21Z

sql/catalyst/src/main/scala/org/apache/spark/sql/connector/expressions/aggregates.scala

+// Aggregate Functions in SQL statement.
+// e.g. SELECT COUNT(EmployeeID), Max(salary) FROM dept GROUP BY deptID
+// aggregateExpressions are (COUNT(EmployeeID), Max(salary)), groupByColumns are (deptID)
+case class Aggregation(aggregateExpressions: Seq[AggregateFunc],


This is public DS v2 API, can we write it in Java?

cloud-fan · 2021-07-15T17:52:08Z

...re/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2ScanRelationPushDown.scala

+                  // No need to do column pruning because only the aggregate columns are used as
+                  // DataSourceV2ScanRelation output columns. All the other columns are not
+                  // included in the output. Since PushDownUtils.pruneColumns is not called,
+                  // ScanBuilder.requiredSchema is not pruned, but ScanBuilder.requiredSchema is


There is no ScanBuilder.requiredSchema.

cloud-fan · 2021-07-15T17:53:26Z

...re/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2ScanRelationPushDown.scala

+                      """.stripMargin)
+
+                  val scanRelation = DataSourceV2ScanRelation(sHolder.relation, scan, output)
+                  assert(scanRelation.output.length ==


can we check this earlier? right after val newOutput = scan.readSchema().toAttributes

...re/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2ScanRelationPushDown.scala

SparkQA · 2021-07-16T17:42:54Z

Test build #141171 has finished for PR 33352 at commit 1f6d4ff.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public class Aggregation implements Serializable
case class Min(column: FieldReference, dataType: DataType) extends AggregateFunc
case class Max(column: FieldReference, dataType: DataType) extends AggregateFunc
case class Sum(column: FieldReference, dataType: DataType, isDistinct: Boolean)
case class Count(column: FieldReference, isDistinct: Boolean)
case class CountOne() extends AggregateFunc

SparkQA · 2021-07-16T18:44:45Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45683/

SparkQA · 2021-07-16T18:56:35Z

Test build #141175 has finished for PR 33352 at commit b1d177b.

This patch fails Java style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-07-16T19:28:40Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45687/

SparkQA · 2021-07-16T20:02:26Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45687/

SparkQA · 2021-07-16T21:26:44Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45694/

SparkQA · 2021-07-16T22:02:27Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45694/

SparkQA · 2021-07-17T01:45:36Z

Test build #141182 has finished for PR 33352 at commit b6c7d58.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2021-07-21T13:52:44Z

...core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/jdbc/JDBCScanBuilder.scala

+    val dialect = JdbcDialects.get(jdbcOptions.url)
+    val compiledAgg = JDBCRDD.compileAggregates(aggregation.getAggregateExpressions, dialect)
+
+    var pushedSchema = new StructType()


outputSchema is a better name

cloud-fan · 2021-07-21T13:56:37Z

...re/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2ScanRelationPushDown.scala

+            f.pushedFilters()
+          case _ => Array.empty[sources.Filter]
+        }
+        V1ScanWrapper(v1, Array.empty[sources.Filter], pushedFilters, aggregation)


V1ScanWrapper.translatedFilters is always Nil now?

It seems like only for display purposes. I'm OK to remove it but let's do it more explicitly and remove this field from V1ScanWrapper

SparkQA · 2021-07-22T00:49:29Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45969/

SparkQA · 2021-07-22T01:23:06Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45969/

SparkQA · 2021-07-22T05:55:50Z

Test build #141450 has finished for PR 33352 at commit 1746fa3.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public final class CountStar implements AggregateFunc

cloud-fan · 2021-07-22T09:08:09Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala

@@ -332,6 +334,7 @@ object DataSourceStrategy
        l.output.toStructType,
        Set.empty,
        Set.empty,
+        Option.empty[Aggregation],


super nit: we can just write None

cloud-fan · 2021-07-22T09:12:48Z

...re/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2ScanRelationPushDown.scala

+    case aggNode @ Aggregate(groupingExpressions, resultExpressions, child) =>
+      child match {
+        case ScanOperation(project, filters, sHolder: ScanBuilderHolder)
+          if project.forall(_.isInstanceOf[AttributeReference]) =>


nit: we can add filter.isEmpty here, instead of writing if (filters.length == 0) in the body

cloud-fan · 2021-07-22T09:15:52Z

...core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/jdbc/JDBCScanBuilder.scala

+    if (!jdbcOptions.pushDownAggregate) return false
+
+    val dialect = JdbcDialects.get(jdbcOptions.url)
+    val compiledAgg = JDBCRDD.compileAggregates(aggregation.aggregateExpressions, dialect)


shall we return false earlier if there are nested fields? Otherwise we will hit assertion error in compileAggregates

Right, we should return false earlier if there are nested fields. Fixed. Please check one more time.

cloud-fan · 2021-07-22T09:16:42Z

...core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/jdbc/JDBCScanBuilder.scala

+    var outputSchema = new StructType()
+    aggregation.groupByColumns.foreach { col =>
+      val structField = getStructFieldForCol(col)
+      outputSchema = outputSchema.add(StructField(structField.name, structField.dataType))


outputSchema.add(structField)?

cloud-fan · 2021-07-22T09:22:27Z

sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCV2Suite.scala

@@ -214,4 +237,204 @@ class JDBCV2Suite extends QueryTest with SharedSparkSession {
      checkAnswer(sql("SELECT name, id FROM h2.test.abc"), Row("bob", 4))
    }
  }
+
+  test("scan with aggregate push-down") {
+    val df1 = sql("select MAX(SALARY), MIN(BONUS) FROM h2.test.employee where dept > 0" +


can we create one test for each of them? then we can a title for each test. e.g. this one is aggregate pushdown with GROUP BY, and the next is aggregate pushdown without GROUP BY

cloud-fan

LGTM except for some minor comments, thanks for your patience!

SparkQA · 2021-07-23T04:04:00Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46050/

SparkQA · 2021-07-23T07:47:41Z

Test build #141532 has finished for PR 33352 at commit fae570a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2021-07-26T08:01:35Z

thanks, merging to master/3.2!

### What changes were proposed in this pull request? Add interfaces and APIs to push down Aggregates to V2 Data Source ### Why are the changes needed? improve performance ### Does this PR introduce _any_ user-facing change? SQLConf.PARQUET_AGGREGATE_PUSHDOWN_ENABLED was added. If this is set to true, Aggregates are pushed down to Data Source. ### How was this patch tested? New tests were added to test aggregates push down in #32049. The original PR is split into two PRs. This PR doesn't contain new tests. Closes #33352 from huaxingao/aggPushDownInterface. Authored-by: Huaxin Gao <huaxin_gao@apple.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> (cherry picked from commit c561ee6) Signed-off-by: Wenchen Fan <wenchen@databricks.com>

viirya · 2021-07-26T08:24:05Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRDD.scala

@@ -143,6 +172,8 @@ object JDBCRDD extends Logging {
   * @param parts - An array of JDBCPartitions specifying partition ids and
   *    per-partition WHERE clauses.
   * @param options - JDBC options that contains url, table and other information.
+   * @param requiredSchema - The schema of the columns to SELECT.
+   * @param aggregation - The pushed down aggregation


Is the param doc correct? I don't see aggregation parameter but outputSchema and groupByColumns.

viirya · 2021-07-26T08:28:51Z

sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SupportsPushDownAggregates.java

+   * be: grouping columns, aggregate columns (in the same order as the aggregate functions in
+   * the given Aggregation).
+   */
+  boolean pushAggregation(Aggregation aggregation);


For public API, we should document what the returned value means.

+1. What is the returned boolean for?

viirya

BTW, although there is an option to control JDBC aggregate pushdown, do we need an overall SQL config to control it? Said we have other data source implementing the API, we may not have an option to disable it?

sunchao

Sorry for reviewing this late! I was interested in taking a look too missed it. Added a bunch of comments and I saw @viirya also left some. Perhaps we can address them in a separate PR?

Also, it seems this PR not only added APIs but also implementation for JDBC data sources? If so, it's better to update the PR description accordingly.

sunchao · 2021-07-26T21:32:39Z

sql/catalyst/src/main/java/org/apache/spark/sql/connector/expressions/Aggregation.java

+ */
+@Evolving
+public final class Aggregation implements Serializable {
+  private AggregateFunc[] aggregateExpressions;


nit: mark these as final?

sunchao · 2021-07-26T21:33:56Z

sql/catalyst/src/main/java/org/apache/spark/sql/connector/expressions/Count.java

+    public FieldReference column() {
+        return column;
+    }
+    public boolean isDinstinct() {


typo: isDinstinct -> isDistinct.

sunchao · 2021-07-26T21:34:04Z

sql/catalyst/src/main/java/org/apache/spark/sql/connector/expressions/Count.java

+ */
+@Evolving
+public final class Count implements AggregateFunc {
+    private FieldReference column;


ditto: make these final

sunchao · 2021-07-26T21:35:41Z

sql/catalyst/src/main/java/org/apache/spark/sql/connector/expressions/Count.java

+    private boolean isDistinct;
+
+    public Count(FieldReference column, boolean isDistinct) {
+        this.column = column;


2 space indentation?

sunchao · 2021-07-26T21:35:50Z

sql/catalyst/src/main/java/org/apache/spark/sql/connector/expressions/Count.java

+    }
+
+    public FieldReference column() {
+        return column;


sunchao · 2021-07-26T21:38:29Z

sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SupportsPushDownAggregates.java

+ * "SELECT min(value) AS m FROM t GROUP BY key", after pushing down the aggregate
+ * to the data source, the data source can still output data with duplicated keys, which is OK
+ * as Spark will do GROUP BY key again. The final query plan can be something like this:
+ * {{{


this is not properly rendered, you can use:

* <pre> * Aggregate [key#1], [min(min(value)#2) AS m#3] * +- RelationV2[key#1, min(value)#2] * </pre> * Similarly, if there is no grouping expression, the data source can still output more than one * rows.

instead. Note that the following <p> is also removed.

sunchao · 2021-07-26T21:39:59Z

sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SupportsPushDownAggregates.java

+   * be: grouping columns, aggregate columns (in the same order as the aggregate functions in
+   * the given Aggregation).
+   */
+  boolean pushAggregation(Aggregation aggregation);


+1. What is the returned boolean for?

sunchao · 2021-07-26T21:42:35Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/PushDownUtils.scala

+        }
+
+        val agg = new Aggregation(translatedAggregates.toArray, translatedGroupBys.toArray)
+        if (r.pushAggregation(agg)) {


nit: you can just use Some(agg).filter(r.pushAggregation)

sunchao · 2021-07-26T21:44:29Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/PushDownUtils.scala

+
+    scanBuilder match {
+      case r: SupportsPushDownAggregates =>
+        val translatedAggregates = aggregates.map(DataSourceStrategy.translateAggregate).flatten


nit: can use flatMap instead of map + flatten.

sunchao · 2021-07-26T21:46:24Z

...re/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2ScanRelationPushDown.scala

+      filterCondition.map(Filter(_, sHolder)).getOrElse(sHolder)
+  }
+
+  def pushdownAggregate(plan: LogicalPlan): LogicalPlan = plan.transform {


nit: rename to pushDownAggregate to keep it consistent with pushDownFilters?

viirya · 2021-07-26T21:54:27Z

@huaxingao created a followup #33526.

huaxingao · 2021-07-27T00:21:04Z

@viirya

BTW, although there is an option to control JDBC aggregate pushdown, do we need an overall SQL config to control it? Said we have other data source implementing the API, we may not have an option to disable it?

The config is per data source. For example, when we implement the parquet aggregate push down later, we will add a config for parquet, something like PARQUET_AGGREGATE_PUSHDOWN_ENABLED in SQLConf.

huaxingao · 2021-07-27T00:22:26Z

@sunchao Thanks for your comments! I will address your comments in the follow up #33526

### What changes were proposed in this pull request? This is a followup of #33352 , to simplify the JDBC aggregate pushdown: 1. We should get the schema of the aggregate query by asking the JDBC server, instead of calculating it by ourselves. This can simplify the code a lot, and is also more robust: the data type of SUM may vary in different databases, it's fragile to assume they are always the same as Spark. 2. because of 1, now we can remove the `dataType` property from the public `Sum` expression. This PR also contains some small improvements: 1. Spark should deduplicate the aggregate expressions before pushing them down. 2. Improve the `toString` of public aggregate expressions to make them more SQL. ### Why are the changes needed? code and API simplification ### Does this PR introduce _any_ user-facing change? this API is not released yet. ### How was this patch tested? existing tests Closes #33579 from cloud-fan/dsv2. Authored-by: Wenchen Fan <wenchen@databricks.com> Signed-off-by: Liang-Chi Hsieh <viirya@gmail.com>

### What changes were proposed in this pull request? This is a followup of #33352 , to simplify the JDBC aggregate pushdown: 1. We should get the schema of the aggregate query by asking the JDBC server, instead of calculating it by ourselves. This can simplify the code a lot, and is also more robust: the data type of SUM may vary in different databases, it's fragile to assume they are always the same as Spark. 2. because of 1, now we can remove the `dataType` property from the public `Sum` expression. This PR also contains some small improvements: 1. Spark should deduplicate the aggregate expressions before pushing them down. 2. Improve the `toString` of public aggregate expressions to make them more SQL. ### Why are the changes needed? code and API simplification ### Does this PR introduce _any_ user-facing change? this API is not released yet. ### How was this patch tested? existing tests Closes #33579 from cloud-fan/dsv2. Authored-by: Wenchen Fan <wenchen@databricks.com> Signed-off-by: Liang-Chi Hsieh <viirya@gmail.com> (cherry picked from commit 387a251) Signed-off-by: Liang-Chi Hsieh <viirya@gmail.com>

github-actions bot added the SQL label Jul 15, 2021

huaxingao mentioned this pull request Jul 15, 2021

[SPARK-34952][SQL] Aggregate (Min/Max/Count) push down for Parquet #32049

Closed

[SPARK-34952][SQL] Aggregate push down APIs

a5386f0

huaxingao force-pushed the aggPushDownInterface branch from 0cce896 to a5386f0 Compare July 15, 2021 05:32

cloud-fan reviewed Jul 15, 2021

View reviewed changes

...re/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2ScanRelationPushDown.scala Outdated Show resolved Hide resolved

cloud-fan reviewed Jul 15, 2021

View reviewed changes

...re/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2ScanRelationPushDown.scala Outdated Show resolved Hide resolved

address comments

1f6d4ff

fix lint scala

b1d177b

fix lint java

b6c7d58

JDBC pushed down fitler should be in explain

7988d9c

cloud-fan reviewed Jul 21, 2021

View reviewed changes

address comments

1746fa3

cloud-fan reviewed Jul 22, 2021

View reviewed changes

cloud-fan approved these changes Jul 22, 2021

View reviewed changes

address comments

fae570a

cloud-fan closed this in c561ee6 Jul 26, 2021

viirya reviewed Jul 26, 2021

View reviewed changes

sunchao reviewed Jul 26, 2021

View reviewed changes

huaxingao mentioned this pull request Jul 28, 2021

[SPARK-22390][SPARK-32833][SQL] JDBC V2 Datasource aggregate push down #29695

Closed

huaxingao deleted the aggPushDownInterface branch July 28, 2021 23:41

cloud-fan mentioned this pull request Jul 29, 2021

[SPARK-34952][SQL][FOLLOWUP] Simplify JDBC aggregate pushdown #33579

Closed

[SPARK-34952][SQL] DSv2 Aggregate push down APIs #33352

[SPARK-34952][SQL] DSv2 Aggregate push down APIs #33352

Conversation

huaxingao commented Jul 15, 2021 • edited

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

SparkQA commented Jul 15, 2021

SparkQA commented Jul 15, 2021

SparkQA commented Jul 15, 2021

SparkQA commented Jul 15, 2021

SparkQA commented Jul 15, 2021

SparkQA commented Jul 15, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Jul 16, 2021

SparkQA commented Jul 16, 2021

SparkQA commented Jul 16, 2021

SparkQA commented Jul 16, 2021

SparkQA commented Jul 16, 2021

SparkQA commented Jul 16, 2021

SparkQA commented Jul 16, 2021

SparkQA commented Jul 17, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Jul 22, 2021

SparkQA commented Jul 22, 2021

SparkQA commented Jul 22, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cloud-fan left a comment

Choose a reason for hiding this comment

SparkQA commented Jul 23, 2021

SparkQA commented Jul 23, 2021

cloud-fan commented Jul 26, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

viirya left a comment

Choose a reason for hiding this comment

sunchao left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

viirya commented Jul 26, 2021

huaxingao commented Jul 27, 2021

huaxingao commented Jul 27, 2021

huaxingao commented Jul 15, 2021 •

edited