[SPARK-25474][SQL][DOCS] Update the docs for spark.sql.statistics.fallBackToHdfs #24715

wangyum · 2019-05-27T00:53:38Z

What changes were proposed in this pull request?

This PR update spark.sql.statistics.fallBackToHdfs's doc:

This flag is effective only if it is Hive table.
For non-partitioned data source table, it will be automatically recalculated if table statistics are not available
For partitioned data source table, It is 'spark.sql.defaultSizeInBytes' if table statistics are not available.

Related code:

Non-partitioned data source table:
SizeInBytesOnlyStatsPlanVisitor.default() -> LogicalRelation.computeStats() -> HadoopFsRelation.sizeInBytes() -> PartitioningAwareFileIndex.sizeInBytes()
PartitioningAwareFileIndex.sizeInBytes() is calculated by allFiles().map(_.getLen).sum if table statistics are not available.
Partitioned data source table:
SizeInBytesOnlyStatsPlanVisitor.default() -> LogicalRelation.computeStats() -> CatalogFileIndex.sizeInBytes
CatalogFileIndex.sizeInBytes is spark.sql.defaultSizeInBytes if table statistics are not available.

How was this patch tested?

N/A

HyukjinKwon · 2019-05-27T02:19:14Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala

+  override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
+    case logicalRelation @ LogicalRelation(_, _, catalogTable, _) if catalogTable.nonEmpty &&
+        catalogTable.forall(DDLUtils.isDatasourceTable) && catalogTable.forall(_.stats.isEmpty) =>
+      val sizeInBytes = if (session.sessionState.conf.fallBackToHdfsForStatsEnabled) {


Looks like this code will be a duplicate at #24712. Are there more places that similar patterns happen? Let's do that PR first and deduplicate it if there are some more places to deduplicate.

I'm move DetermineTableStats from HiveStrategies to DataSourceStrategy to reduce duplicate.

SparkQA · 2019-05-27T03:08:37Z

Test build #105808 has finished for PR 24715 at commit dd5a125.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class DetermineDataSourceTableStats(session: SparkSession) extends Rule[LogicalPlan]

# Conflicts: # sql/core/src/test/scala/org/apache/spark/sql/StatisticsCollectionSuite.scala # sql/hive/src/test/scala/org/apache/spark/sql/hive/StatisticsSuite.scala

SparkQA · 2019-06-28T11:48:33Z

Test build #106995 has finished for PR 24715 at commit 70d3557.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class DetermineTableStats(session: SparkSession) extends Rule[LogicalPlan]

SparkQA · 2019-06-28T15:18:42Z

Test build #107010 has finished for PR 24715 at commit dd5f356.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-06-29T07:05:02Z

Test build #107034 has finished for PR 24715 at commit 22dd26e.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-06-29T07:11:35Z

retest this please

SparkQA · 2019-06-29T10:18:32Z

Test build #107035 has finished for PR 24715 at commit 22dd26e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-07-01T19:03:34Z

Test build #107080 has finished for PR 24715 at commit 7748a32.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala

cloud-fan · 2019-08-15T06:18:25Z

The idea LGTM, can you rebase this PR?

# Conflicts: # sql/core/src/main/scala/org/apache/spark/sql/internal/BaseSessionStateBuilder.scala # sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala

wangyum · 2019-08-15T11:34:48Z

I did some benchmark.

Prepare data:

spark.range(100000000).repartition(10000).write.saveAsTable("test_non_partition_10000")
spark.range(100000000).repartition(300000).write.saveAsTable("test_non_partition_300000")
spark.range(100000000).selectExpr("id", "id % 5000 as c2", "id as c3").repartition(org.apache.spark.sql.functions.col("c2")).write.partitionBy("c2").saveAsTable("test_partition_5000")
spark.range(100000000).selectExpr("id", "id % 10000 as c2", "id as c3").repartition(org.apache.spark.sql.functions.col("c2")).write.partitionBy("c2").saveAsTable("test_partition_10000")

Add these lines to LogicalRelation.computeStats:

val time1 = System.currentTimeMillis()
val relationSize = relation.sizeInBytes
val time2 = System.currentTimeMillis()
val fallBackToHdfsSize = CommandUtils.getSizeInBytesFallBackToHdfs(relation.sqlContext.sparkSession, catalogTable.get)
val time3 = System.currentTimeMillis()
// scalastyle:off
println(s"Get size from relation: $relationSize, time: ${time2 - time1}")
println(s"Get size fall back to HDFS: $fallBackToHdfsSize, time: ${time3 - time2}")
// scalastyle:on

Non-partitioned table benchmark result:

scala> spark.sql("explain cost select * from test_non_partition_10000 limit 1").show
Get size from relation: 576588171, time: 22
Get size fall back to HDFS: 576588171, time: 41
+--------------------+
|                plan|
+--------------------+
|== Optimized Logi...|
+--------------------+


scala> spark.sql("explain cost select * from test_non_partition_10000 limit 1").show
Get size from relation: 576588171, time: 3
Get size fall back to HDFS: 576588171, time: 28
+--------------------+
|                plan|
+--------------------+
|== Optimized Logi...|
+--------------------+


scala>

scala> spark.sql("explain cost select * from test_non_partition_300000 limit 1").show
Get size from relation: 706507984, time: 135
Get size fall back to HDFS: 706507984, time: 2038
+--------------------+
|                plan|
+--------------------+
|== Optimized Logi...|
+--------------------+


scala> spark.sql("explain cost select * from test_non_partition_300000 limit 1").show
Get size from relation: 706507984, time: 168
Get size fall back to HDFS: 706507984, time: 3629
+--------------------+
|                plan|
+--------------------+
|== Optimized Logi...|
+--------------------+

Partitioned table benchmark result:

scala> spark.sql("explain cost select * from test_partition_5000 limit 1").show
Get size from relation: 9223372036854775807, time: 0
Get size fall back to HDFS: 1018560794, time: 46
+--------------------+
|                plan|
+--------------------+
|== Optimized Logi...|
+--------------------+


scala> spark.sql("explain cost select * from test_partition_10000 limit 1").show
Get size from relation: 9223372036854775807, time: 0
Get size fall back to HDFS: 1036799332, time: 43
+--------------------+
|                plan|
+--------------------+
|== Optimized Logi...|
+--------------------+

Partitioned table with spark.sql.hive.manageFilesourcePartitions=false (set it by --conf) benchmark result:

scala> spark.sql("set spark.sql.hive.manageFilesourcePartitions").show
+--------------------+-----+
|                 key|value|
+--------------------+-----+
|spark.sql.hive.ma...|false|
+--------------------+-----+


scala> spark.sql("explain cost select * from test_partition_5000 limit 1").show
Get size from relation: 1018560794, time: 3
Get size fall back to HDFS: 1018560794, time: 45
+--------------------+
|                plan|
+--------------------+
|== Optimized Logi...|
+--------------------+


scala> spark.sql("explain cost select * from test_partition_10000 limit 1").show
Get size from relation: 1036799332, time: 865
Get size fall back to HDFS: 1036799332, time: 69
+--------------------+
|                plan|
+--------------------+
|== Optimized Logi...|
+--------------------+

cloud-fan · 2019-08-15T12:39:46Z

@wangyum do you mean CommandUtils.getSizeInBytesFallBackToHdfs is very slow if there are many files?

cloud-fan · 2019-08-15T12:41:41Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala

+
+  override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
+    // For the data source table, we only recalculate the table statistics when it creates
+    // the CatalogFileIndex using defaultSizeInBytes. See SPARK-25474 for more details.


when it creates the CatalogFileIndex using defaultSizeInBytes -> when the table stats are not available

cloud-fan · 2019-08-15T12:42:29Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala

+ */
+class DetermineTableStats(session: SparkSession) extends Rule[LogicalPlan] {
+
+  private val sessionConf = session.sessionState.conf


nit: just call it conf

cloud-fan · 2019-08-15T12:47:01Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala

+      val withStats = table.copy(stats = Some(CatalogStatistics(sizeInBytes = BigInt(sizeInBytes))))
+      logical.copy(catalogTable = Some(withStats))
+
+    case relation: HiveTableRelation


shall we catch InsertIntoTable(HiveTableRelation) as well?

@advancedxy Already work on this:c86a27b

cloud-fan · 2019-08-15T12:47:48Z

sql/core/src/test/scala/org/apache/spark/sql/StatisticsCollectionSuite.scala

+    // Non-partitioned table
+    withTempDir { dir =>
+      Seq(false, true).foreach { fallBackToHDFSForStats =>
+        withSQLConf(SQLConf.ENABLE_FALL_BACK_TO_HDFS_FOR_STATS.key -> s"$fallBackToHDFSForStats") {


why this config has no effect in this test?

// fallBackToHDFSForStats = true: The table stats will be recalculated by DetermineTableStats
// fallBackToHDFSForStats = false: The table stats will be recalculated by FileIndex

cloud-fan · 2019-08-15T12:48:33Z

sql/core/src/test/scala/org/apache/spark/sql/StatisticsCollectionSuite.scala

+    }
+
+    // Partitioned table
+    Seq(false, true).foreach { fallBackToHDFSForStats =>


please create a test case for it. e.g.

test("partitioned data source tables support fallback to HDFS for size estimation")

wangyum · 2019-08-15T15:30:04Z

@wangyum do you mean CommandUtils.getSizeInBytesFallBackToHdfs is very slow if there are many files?

CommandUtils.getSizeInBytesFallBackToHdfs is not very slow.
I have no idea why PartitioningAwareFileIndex.sizeInBytes is faster than CommandUtils.getSizeInBytesFallBackToHdfs.
It may be related to the cluster load, I plan to switch to an idle cluster to test tomorrow.

SparkQA · 2019-08-20T17:46:14Z

Test build #109415 has finished for PR 24715 at commit cc32b48.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2019-08-21T10:30:50Z

sql/core/src/test/scala/org/apache/spark/sql/StatisticsCollectionSuite.scala

+    }
+  }
+
+  test("External partitioned data source table does not support fallback to HDFS " +


how is this implemented?

We do not support it:

if (fallBackToHDFS) { assert(relation.stats.sizeInBytes === 0) } else { assert(relation.stats.sizeInBytes === conf.defaultSizeInBytes) }

cloud-fan · 2019-08-21T11:56:48Z

sql/core/src/test/scala/org/apache/spark/sql/StatisticsCollectionSuite.scala

+              val relation = spark.table("spark_25474").queryExecution.analyzed.children.head
+              assert(spark.table("spark_25474").count() === 5)
+              if (fallBackToHDFS) {
+                assert(relation.stats.sizeInBytes === 0)


We have a serious issue here: the wrong stats may mislead Spark to broadcast a very large table and OOM.

I think we can only fallback to HDFS size for non-partitioned tables.

+1 Partitioned table usually very large.

@shahidki31 @dongjoon-hyun What do you think ?

+1 for disabling for partitioned tables.
This is master branch status, isn't it? Previously, Spark returns 8EB in this case safely.

I think, non-partitioned data source table is already getting the correct statistics. I am not sure, we need to support fallback to HDFS for size for non partitioned table.

@wangyum . Why do we need to revert that? You can revert the functional part here and keep the test codes.

For me, this PR already contains the reverting here(#24715 (comment)).

Personally, I'm -1 for removing the existing test cases.

@dongjoon-hyun It is expensive to support partitioned table with external partitions. Please see this test case. Its data size is incorrect.

Related discussion:
#24715 (comment)
#24715 (comment)

So we plan do not fallback to HDFS size for partitioned tables.

+1 We need to document it.

@wangyum May be, in this PR only you can do both, if we doesn't support fallback config.

dongjoon-hyun · 2019-08-22T06:27:22Z

sql/core/src/test/scala/org/apache/spark/sql/StatisticsCollectionSuite.scala

+              s"PARTITIONED BY(a) LOCATION '${dir.toURI}'")
+
+            withTempDir { partitionDir =>
+              spark.range(5).write.mode(SaveMode.Overwrite).parquet(partitionDir.getCanonicalPath)


spark.range(5).toDF("b") instead of spark.range(5)?

# Conflicts: # sql/core/src/main/scala/org/apache/spark/sql/execution/command/CommandUtils.scala # sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/HadoopFsRelation.scala # sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala

wangyum · 2019-08-23T15:49:11Z

sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala

+      "determining if a table is small enough to use auto broadcast joins. " +
+      "For non-partitioned data source table, it will be automatically recalculated if table " +
+      "statistics are not available. For partitioned data source table, It is " +
+      s"'${DEFAULT_SIZE_IN_BYTES.key}' if table statistics are not available.")
    .booleanConf


@shahidki31 I have updated the documentation. Please take a look.

Thanks @wangyum . Looks good. Shall we add it in the configuration.md also. It seems these configs are not there?

Hi, @wangyum . We had better minimize the patch diff always. Please keep the original location of this conf. I don't see any difference for the following three lines.

val ENABLE_FALL_BACK_TO_HDFS_FOR_STATS = buildConf("spark.sql.statistics.fallBackToHdfs")
.booleanConf
.createWithDefault(false)

This is because we added DEFAULT_SIZE_IN_BYTES to ENABLE_FALL_BACK_TO_HDFS_FOR_STATS. So we need to move DEFAULT_SIZE_IN_BYTES before ENABLE_FALL_BACK_TO_HDFS_FOR_STATS , otherwise:

[error] [warn] /home/jenkins/workspace/SparkPullRequestBuilder/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:1224: Reference to uninitialized value DEFAULT_SIZE_IN_BYTES [error] [warn] s"'${DEFAULT_SIZE_IN_BYTES.key}' if table statistics are not available.") [error] [warn] [warn] 8 warnings found

@shahidki31 Yes. Running the SET -v command will show the entire list of the SQL configuration:
http://spark.apache.org/docs/latest/configuration.html#spark-sql

SparkQA · 2019-08-23T15:58:56Z

Test build #109645 has finished for PR 24715 at commit 3b0c234.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-08-23T20:41:43Z

Test build #109647 has finished for PR 24715 at commit d23fd47.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2019-08-26T07:42:43Z

sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala

@@ -1230,6 +1224,16 @@ object SQLConf {
    .bytesConf(ByteUnit.BYTE)
    .createWithDefault(Long.MaxValue)

+  val ENABLE_FALL_BACK_TO_HDFS_FOR_STATS = buildConf("spark.sql.statistics.fallBackToHdfs")
+    .doc("This flag is effective only if it is Hive table. When true, it will fall back to HDFS " +


Hive table -> Non-partitioned Hive table?

Currently Hive partitioned table is supported, do you think we need to disable it?

spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala

Lines 116 to 136 in 07c4b9b

override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {

case relation: HiveTableRelation

if DDLUtils.isHiveTable(relation.tableMeta) && relation.tableMeta.stats.isEmpty =>

val table = relation.tableMeta

val sizeInBytes = if (session.sessionState.conf.fallBackToHdfsForStatsEnabled) {

try {

val hadoopConf = session.sessionState.newHadoopConf()

val tablePath = new Path(table.location)

val fs: FileSystem = tablePath.getFileSystem(hadoopConf)

fs.getContentSummary(tablePath).getLength

} catch {

case e: IOException =>

logWarning("Failed to get table size from hdfs.", e)

session.sessionState.conf.defaultSizeInBytes

}

} else {

session.sessionState.conf.defaultSizeInBytes

}

val withStats = table.copy(stats = Some(CatalogStatistics(sizeInBytes = BigInt(sizeInBytes))))

relation.copy(tableMeta = withStats)

Yea I think so. Can you send a new PR to fix it?

cloud-fan · 2019-08-28T06:22:14Z

sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala

@@ -1230,6 +1224,16 @@ object SQLConf {
    .bytesConf(ByteUnit.BYTE)
    .createWithDefault(Long.MaxValue)

+  val ENABLE_FALL_BACK_TO_HDFS_FOR_STATS = buildConf("spark.sql.statistics.fallBackToHdfs")
+    .doc("When true, it will fall back to HDFS if the table statistics are not available from " +
+      "table metadata. This is useful in determining if a table is small enough to use auto " +


auto broadcast join? maybe just say broadcast join

cloud-fan · 2019-08-28T06:22:45Z

sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala

+  val ENABLE_FALL_BACK_TO_HDFS_FOR_STATS = buildConf("spark.sql.statistics.fallBackToHdfs")
+    .doc("When true, it will fall back to HDFS if the table statistics are not available from " +
+      "table metadata. This is useful in determining if a table is small enough to use auto " +
+      "broadcast joins. This flag is effective only if it is non-partitioned Hive table. " +


only for non-partitioned Hive tables

SparkQA · 2019-08-28T07:05:02Z

Test build #109850 has finished for PR 24715 at commit 44ac6cc.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-08-28T07:05:02Z

Test build #109851 has finished for PR 24715 at commit 55d59e3.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-08-28T07:11:02Z

retest this please

SparkQA · 2019-08-28T10:45:46Z

Test build #109853 has finished for PR 24715 at commit 55d59e3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2019-08-28T11:15:36Z

thanks, merging to master!

Data source tables support fallback to HDFS for size estimation

dd5a125

HyukjinKwon reviewed May 27, 2019

View reviewed changes

dongjoon-hyun added the SQL label Jun 14, 2019

wangyum mentioned this pull request Jun 27, 2019

[SPARK-25474][SQL] Support spark.sql.statistics.fallBackToHdfs in data source tables #22502

Closed

wangyum added 2 commits June 28, 2019 13:29

Merge remote-tracking branch 'upstream/master' into SPARK-25474

1dbf2ca

# Conflicts: # sql/core/src/test/scala/org/apache/spark/sql/StatisticsCollectionSuite.scala # sql/hive/src/test/scala/org/apache/spark/sql/hive/StatisticsSuite.scala

Move DetermineTableStats to DataSourceStrategy

70d3557

Fix test error

dd5f356

Fix test error.

22dd26e

Revert InMemoryColumnarQuerySuite

7748a32

wangyum commented Jul 2, 2019

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala Outdated Show resolved Hide resolved

wangyum added 3 commits August 15, 2019 15:54

Merge remote-tracking branch 'upstream/master' into SPARK-25474

d47d420

# Conflicts: # sql/core/src/main/scala/org/apache/spark/sql/internal/BaseSessionStateBuilder.scala # sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala

fix

43fc310

Add comment

001e1e1

wangyum mentioned this pull request Aug 15, 2019

[SPARK-25474][SQL][FOLLOW-UP] fallback to hdfs when relation table stats is not available #25460

Closed

cloud-fan reviewed Aug 15, 2019

View reviewed changes

cloud-fan reviewed Aug 21, 2019

View reviewed changes

dongjoon-hyun reviewed Aug 22, 2019

View reviewed changes

wangyum closed this Aug 23, 2019

wangyum added 2 commits August 23, 2019 23:32

Update doc

3b0c234

wangyum reopened this Aug 23, 2019

wangyum commented Aug 23, 2019

View reviewed changes

wangyum changed the title ~~[SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation~~ [SPARK-25474][SQL] Update the documentation for spark.sql.statistics.fallBackToHdfs Aug 23, 2019

Fix test error

d23fd47

dongjoon-hyun changed the title ~~[SPARK-25474][SQL] Update the documentation for spark.sql.statistics.fallBackToHdfs~~ [SPARK-25474][SQL][DOCS] Update the docs for spark.sql.statistics.fallBackToHdfs Aug 24, 2019

cloud-fan reviewed Aug 26, 2019

View reviewed changes

update doc

44ac6cc

cloud-fan reviewed Aug 28, 2019

View reviewed changes

Address comment

55d59e3

cloud-fan closed this in e3b32da Aug 28, 2019

wangyum deleted the SPARK-25474 branch August 28, 2019 11:20

	override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
	case relation: HiveTableRelation
	if DDLUtils.isHiveTable(relation.tableMeta) && relation.tableMeta.stats.isEmpty =>
	val table = relation.tableMeta
	val sizeInBytes = if (session.sessionState.conf.fallBackToHdfsForStatsEnabled) {
	try {
	val hadoopConf = session.sessionState.newHadoopConf()
	val tablePath = new Path(table.location)
	val fs: FileSystem = tablePath.getFileSystem(hadoopConf)
	fs.getContentSummary(tablePath).getLength
	} catch {
	case e: IOException =>
	logWarning("Failed to get table size from hdfs.", e)
	session.sessionState.conf.defaultSizeInBytes
	}
	} else {
	session.sessionState.conf.defaultSizeInBytes
	}

	val withStats = table.copy(stats = Some(CatalogStatistics(sizeInBytes = BigInt(sizeInBytes))))
	relation.copy(tableMeta = withStats)

[SPARK-25474][SQL][DOCS] Update the docs for spark.sql.statistics.fallBackToHdfs #24715

[SPARK-25474][SQL][DOCS] Update the docs for spark.sql.statistics.fallBackToHdfs #24715

Conversation

wangyum commented May 27, 2019 • edited Loading

What changes were proposed in this pull request?

How was this patch tested?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented May 27, 2019

SparkQA commented Jun 28, 2019

SparkQA commented Jun 28, 2019

SparkQA commented Jun 29, 2019

wangyum commented Jun 29, 2019

SparkQA commented Jun 29, 2019

SparkQA commented Jul 1, 2019

cloud-fan commented Aug 15, 2019

wangyum commented Aug 15, 2019

cloud-fan commented Aug 15, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wangyum commented Aug 15, 2019

SparkQA commented Aug 20, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shahidki31 Aug 22, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shahidki31 Aug 22, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wangyum Aug 23, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Aug 23, 2019

SparkQA commented Aug 23, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Aug 28, 2019

SparkQA commented Aug 28, 2019

wangyum commented Aug 28, 2019

SparkQA commented Aug 28, 2019

cloud-fan commented Aug 28, 2019

wangyum commented May 27, 2019 •

edited

Loading

shahidki31 Aug 22, 2019 •

edited

Loading

shahidki31 Aug 22, 2019 •

edited

Loading

wangyum Aug 23, 2019 •

edited

Loading