[SPARK-43402][SQL] FileSourceScanExec supports push down data filter with scalar subquery #41088

ulysses-you · 2023-05-08T05:11:46Z

What changes were proposed in this pull request?

Scalar subquery can be pushed down as data filter at runtime, since we always execute subquery first. Ideally, we can rewrite ScalarSubquery to Literal before pushing down filter. The main issue before we do not support that is ReuseSubquery is ineffective, see #22518. It is not a issue now.

For example:

SELECT * FROM t1 WHERE c1 > (SELECT min(c2) FROM t2)

Why are the changes needed?

Improve peformance if data filter have scalar subquery.

Does this PR introduce any user-facing change?

no

How was this patch tested?

add test

ulysses-you · 2023-05-08T06:03:28Z

cc @cloud-fan @viirya thank you

cloud-fan · 2023-05-09T06:16:27Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala

@@ -188,7 +188,13 @@ object FileSourceStrategy extends Strategy with PredicateHelper with Logging {
      // Partition keys are not available in the statistics of the files.
      // `dataColumns` might have partition columns, we need to filter them out.
      val dataColumnsWithoutPartitionCols = dataColumns.filterNot(partitionSet.contains)
-      val dataFilters = normalizedFiltersWithoutSubqueries.flatMap { f =>
+      // Scalar subquery can be pushed down as data filter at runtime, since we always


how about partition filters?

partition filter has already supported since #23802

cloud-fan · 2023-05-09T13:10:30Z

sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala

@@ -543,7 +561,7 @@ case class FileSourceScanExec(
        dataSchema = relation.dataSchema,
        partitionSchema = relation.partitionSchema,
        requiredSchema = requiredSchema,
-        filters = pushedDownFilters,
+        filters = dynamicallyPushedDownFilters,


One implicit change is, now we will trigger subquery execution when getting the rdd of the file scan node. This worries me a little bit. Can we trigger subquery execution in the AQE framework? e.g. before executing a query stage, transform the scan node and update its dataFilters to replace evaluated scalar subqueries with literals.

We do not change the timing of trigger for subquery execution. If we call execute at SparkPlan, then it would always first prepare and call updateResult to executing subquery. So it seems there is no change with this pr. This pr only uses the result of subquery and push it down.

Could we split the dynamicallyPushedDownFilters to pushedDownFilters and pushedDownRuntimeFilters and combine them here? Something like this: https://github.com/apache/spark/pull/36128/files#diff-089285f1484c1598cb2839b86b6a9e65b98ab5b30462aedc210fe4bbf44cae78R374-R455
This is because pushedDownFilters are also used in metadata, while pushedDownRuntimeFilters can only be used here.

I did not change pushedDownFilters, it should not be a issue with metadata. But it's a pity that we can not make metadata aware of scalar subquery filter.

It is not the issue. Just wanted to make it clearer. dynamicallyPushedDownFilters also includes pushedDownFilters.

If we first filter out scalar subqueries and then add them back, then the ordering of pushed filter is changed. e.g.,
c1 > (select min(x) from t) and c2 > 1 -> c2 > 1 and c1 > (select min(x) from t)
I'm not sure if it affects the performance in parquet side. So I simply re-translate the whole data filter.

viirya · 2023-05-10T05:53:26Z

sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala

+      pushedDownFilters
+    }
+  }
+
  override lazy val metadata: Map[String, String] = {


This metadata also contains pushedDownFilters, do we need to update it? Or, are we able to update it?

I think we are not able to update it. metadata would be used by explain which is invoked before executing and we get the pushed filter with scalar subqueris at runtime.

viirya · 2023-05-10T06:04:12Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala

+      // execute subquery first.
+      // It has no meaning to push down bloom filter, so skip it.
+      val normalizedFiltersWithScalarSubqueries = normalizedFilters
+        .filterNot(e => e.containsPattern(PLAN_EXPRESSION) && !e.containsPattern(SCALAR_SUBQUERY))


Is it possible the scalar subquery is correlated?

it seems impossible, we have rewritten the correlated scalar subquery at Optimizer side. Otherwise it would break tons of things, e.g., the partition filter with sclar subquery.

viirya · 2023-05-10T19:21:08Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala

+      val normalizedFiltersWithScalarSubqueries = normalizedFilters
+        .filterNot(e => e.containsPattern(PLAN_EXPRESSION) && !e.containsPattern(SCALAR_SUBQUERY))
+        .filterNot(_.isInstanceOf[BloomFilterMightContain])
+      val dataFilters = normalizedFiltersWithScalarSubqueries.flatMap { f =>


Hmm, dataFilters will go through translateFilter. Does translateFilter support translate scalar subquery?

no, translateFilter does not support scalar subquery. We will first transform scalar subquery to literal then go through translateFilter at runtime. The translateFilter related code in FileSourceStrategy is only used to print log which does not matter.

Oh, yea, I remember this now. 😄

ulysses-you · 2023-05-16T09:57:01Z

@cloud-fan @viirya @wangyum do you have other thought ?

viirya

Looks okay. Maybe @cloud-fan can take another look though.

ulysses-you · 2023-07-28T02:13:10Z

I'm going to merge this pr to master(4.0.0) if no any more comments in a few days, cc @viirya @cloud-fan @wangyum

ulysses-you · 2023-07-31T02:15:58Z

thank you for review, merged to master!

cloud-fan · 2023-08-02T09:13:32Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala

      val bucketSpec: Option[BucketSpec] = fsRelation.bucketSpec
      val bucketSet = if (shouldPruneBuckets(bucketSpec)) {
+        // subquery expressions are filtered out because they can't be used to prune buckets


seems it's fine to not filter out scalar subquery, as the code of genBucketSet can skip it already.

cloud-fan · 2023-08-02T09:27:02Z

sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala

+      // Replace scalar subquery to literal so that `DataSourceStrategy.translateFilter` can
+      // support translate it. The subquery must has been materialized since SparkPlan always
+      // execute subquery first.
+      val normalized = dataFilters.map(_.transform {


should selectedPartitions also use the normalized data filters?

I feel a bit of inconsistency here. The partition filter can also contain scala subquery, when is it normalized?

Seems selectedPartitions does not accept subquery expression, it filter out the dpp filter first.

spark/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala

Lines 252 to 257 in e326b9b

@transient lazy val selectedPartitions: Array[PartitionDirectory] = {

val optimizerMetadataTimeNs = relation.location.metadataOpsTimeNs.getOrElse(0L)

val startTime = System.nanoTime()

val ret =

relation.location.listFiles(

partitionFilters.filterNot(isDynamicPruningFilter), dataFilters)

Then in dynamicallySelectedPartitions, normalized the partition filter.

spark/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala

Lines 268 to 281 in e326b9b

@transient protected lazy val dynamicallySelectedPartitions: Array[PartitionDirectory] = {

val dynamicPartitionFilters = partitionFilters.filter(isDynamicPruningFilter)

if (dynamicPartitionFilters.nonEmpty) {

val startTime = System.nanoTime()

// call the file index for the files matching all filters except dynamic partition filters

val predicate = dynamicPartitionFilters.reduce(And)

val partitionColumns = relation.partitionSchema

val boundPredicate = Predicate.create(predicate.transform {

case a: AttributeReference =>

val index = partitionColumns.indexWhere(a.name == _.name)

BoundReference(index, partitionColumns(index).dataType, nullable = true)

}, Nil)

val ret = selectedPartitions.filter(p => boundPredicate.eval(p.values))

For scalar subquery, I think we should turn it into literal so that it can be used to prune partitions?

The outputPartitioning and outputOrdering use selectedPartitions when it is bucket scan which is before do execution. Besides, selectedPartitions is a public field, I'm afraid it would break something if we assume it only be called at runtime...

cloud-fan · 2023-08-02T09:32:20Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala

+      // execute subquery first.
+      // It has no meaning to push down bloom filter, so skip it.
+      val normalizedFiltersWithScalarSubqueries = normalizedFilters
+        .filterNot(e => e.containsPattern(PLAN_EXPRESSION) && !e.containsPattern(SCALAR_SUBQUERY))


thinking about this again. I think all non-correlated subqueries can be allowed? They will be evaluated before the actual plan execution.

Actually, subquery expressions must be non-correlated when we reach here (they must have been rewritten into joins before). Then we need to deal with two kinds of subqueries:

foldable ones like scalar subquery. We can turn them into literals at the beginning and do filtering.

non-foldable ones like the DPP subquery. We will evaluate them when we have an initial partition list and need to filter it further.

yes, and list subquery should also be rewritten to join. So we can only match scalar subquery here for data filter. DPP is a partition filter.

I think we need to make it clear about the 3 different kinds of predicates

simple predicates that can be used to prune partitions, or anything that needs to be accessed during planning

foldable subquery expressions that can be turned into literal before further use (is data source filter pushdown the only use?)

subquery expressions that can be used to do additional pruning after planning but before execution.

it seems 2 and 3 are same but 2 is data filter and 3 is partition filter ?

Technically it can be. But there is a issue if we want to use data filte to help pruning partition directory #41088 (comment). Or, add one more field to do one more list partition directory with normalized data filter.

I mean, something like part_col = (select ... )

yes, it has been worked since #23802. So, both partitionFilters and dataFilters support contain subquery expression.

ok, so we only need to covert scala subquery to literal because filter pushdown needs v1 filter which doesn't support subquery?

yes, v1 pushdown filter seems only support literal.

…a/partition filters ### What changes were proposed in this pull request? This is a followup of #41088 to make the code cleaner. There are two kinds of data/partition filters: static filters that can be executed during the planning phase, and dynamic filters that can only be executed after the planning phase. This PR makes sure these two kinds of filters are properly used and adds code comments to explain it. ### Why are the changes needed? code cleanup ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? existing tests Closes #42318 from cloud-fan/minor. Authored-by: Wenchen Fan <wenchen@databricks.com> Signed-off-by: Kent Yao <yao@apache.org>

epa095 · 2023-11-06T17:36:08Z

@ulysses-you Does this solve https://issues.apache.org/jira/browse/SPARK-34444 ?

ulysses-you · 2023-11-07T01:33:48Z

@epa095 yes, updated that jira status

…y produces in NPE in Parquet filter ### What changes were proposed in this pull request? This issue has been introduced in #41088 where we convert scalar subqueries to literals and then convert the literals to org.apache.spark.sql.sources.Filters. These filters are then pushed down to parquet. If the literal is a comparison with null then the parquet filter conversion code throws NPE. dateToDays does not expect null https://github.com/apache/spark/blob/9b53b803998001e4b706666e37e5f86f900a7430/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala#L153 but Gt passes it directly https://github.com/apache/spark/blob/9b53b803998001e4b706666e37e5f86f900a7430/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala#L403C29-L403C41 ``` Caused by: scala.MatchError: null at org.apache.spark.sql.execution.datasources.parquet.ParquetFilters.org$apache$spark$sql$execution$datasources$parquet$ParquetFilters$$dateToDays(ParquetFilters.scala:168) at org.apache.spark.sql.execution.datasources.parquet.ParquetFilters$$anonfun$5.$anonfun$applyOrElse$73(ParquetFilters.scala:419) at org.apache.spark.sql.execution.datasources.parquet.ParquetFilters.$anonfun$createFilterHelper$9(ParquetFilters.scala:720) at scala.Option.map(Option.scala:230) at org.apache.spark.sql.execution.datasources.parquet.ParquetFilters.createFilterHelper(ParquetFilters.scala:720) at org.apache.spark.sql.execution.datasources.parquet.ParquetFilters.createFilter(ParquetFilters.scala:621) at org.apache.spark.sql.execution.datasources.parquet.ParquetFilterPredicateFactory.$anonfun$createFilter$2(ParquetFilters.scala:885) at scala.collection.immutable.List.flatMap(List.scala:366) at org.apache.spark.sql.execution.datasources.parquet.ParquetFilterPredicateFactory.createFilter(ParquetFilters.scala:885) at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anon$1.apply(ParquetFileFormat.scala:410) at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anon$1.apply(ParquetFileFormat.scala:265) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1$$anon$2.getNext(FileScanRDD.scala:601) ``` ### Why are the changes needed? NPE when subquery returns a NULL literal NPE when comparing against a NULL literal with NullPropagation rule turned off. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? unit test ### Was this patch authored or co-authored using generative AI tooling? No Closes #45202 from cosmind-db/cosmind-db/filter-null. Lead-authored-by: cosmind-db <cosmin.dumitru@databricks.com> Co-authored-by: Hyukjin Kwon <gurwls223@gmail.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>

…y produces in NPE in Parquet filter ### What changes were proposed in this pull request? This issue has been introduced in apache#41088 where we convert scalar subqueries to literals and then convert the literals to org.apache.spark.sql.sources.Filters. These filters are then pushed down to parquet. If the literal is a comparison with null then the parquet filter conversion code throws NPE. dateToDays does not expect null https://github.com/apache/spark/blob/9b53b803998001e4b706666e37e5f86f900a7430/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala#L153 but Gt passes it directly https://github.com/apache/spark/blob/9b53b803998001e4b706666e37e5f86f900a7430/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala#L403C29-L403C41 ``` Caused by: scala.MatchError: null at org.apache.spark.sql.execution.datasources.parquet.ParquetFilters.org$apache$spark$sql$execution$datasources$parquet$ParquetFilters$$dateToDays(ParquetFilters.scala:168) at org.apache.spark.sql.execution.datasources.parquet.ParquetFilters$$anonfun$5.$anonfun$applyOrElse$73(ParquetFilters.scala:419) at org.apache.spark.sql.execution.datasources.parquet.ParquetFilters.$anonfun$createFilterHelper$9(ParquetFilters.scala:720) at scala.Option.map(Option.scala:230) at org.apache.spark.sql.execution.datasources.parquet.ParquetFilters.createFilterHelper(ParquetFilters.scala:720) at org.apache.spark.sql.execution.datasources.parquet.ParquetFilters.createFilter(ParquetFilters.scala:621) at org.apache.spark.sql.execution.datasources.parquet.ParquetFilterPredicateFactory.$anonfun$createFilter$2(ParquetFilters.scala:885) at scala.collection.immutable.List.flatMap(List.scala:366) at org.apache.spark.sql.execution.datasources.parquet.ParquetFilterPredicateFactory.createFilter(ParquetFilters.scala:885) at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anon$1.apply(ParquetFileFormat.scala:410) at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anon$1.apply(ParquetFileFormat.scala:265) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1$$anon$2.getNext(FileScanRDD.scala:601) ``` ### Why are the changes needed? NPE when subquery returns a NULL literal NPE when comparing against a NULL literal with NullPropagation rule turned off. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? unit test ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#45202 from cosmind-db/cosmind-db/filter-null. Lead-authored-by: cosmind-db <cosmin.dumitru@databricks.com> Co-authored-by: Hyukjin Kwon <gurwls223@gmail.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>

…with scalar subquery ### What changes were proposed in this pull request? Scalar subquery can be pushed down as data filter at runtime, since we always execute subquery first. Ideally, we can rewrite `ScalarSubquery` to `Literal` before pushing down filter. The main issue before we do not support that is `ReuseSubquery` is ineffective, see apache#22518. It is not a issue now. For example: ```sql SELECT * FROM t1 WHERE c1 > (SELECT min(c2) FROM t2) ``` ### Why are the changes needed? Improve peformance if data filter have scalar subquery. ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? add test Closes apache#41088 from ulysses-you/SPARK-43402. Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Xiduo You <ulyssesyou@apache.org>

…a/partition filters ### What changes were proposed in this pull request? This is a followup of apache#41088 to make the code cleaner. There are two kinds of data/partition filters: static filters that can be executed during the planning phase, and dynamic filters that can only be executed after the planning phase. This PR makes sure these two kinds of filters are properly used and adds code comments to explain it. ### Why are the changes needed? code cleanup ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? existing tests Closes apache#42318 from cloud-fan/minor. Authored-by: Wenchen Fan <wenchen@databricks.com> Signed-off-by: Kent Yao <yao@apache.org>

…y produces in NPE in Parquet filter ### What changes were proposed in this pull request? This issue has been introduced in apache#41088 where we convert scalar subqueries to literals and then convert the literals to org.apache.spark.sql.sources.Filters. These filters are then pushed down to parquet. If the literal is a comparison with null then the parquet filter conversion code throws NPE. dateToDays does not expect null https://github.com/apache/spark/blob/9b53b803998001e4b706666e37e5f86f900a7430/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala#L153 but Gt passes it directly https://github.com/apache/spark/blob/9b53b803998001e4b706666e37e5f86f900a7430/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala#L403C29-L403C41 ``` Caused by: scala.MatchError: null at org.apache.spark.sql.execution.datasources.parquet.ParquetFilters.org$apache$spark$sql$execution$datasources$parquet$ParquetFilters$$dateToDays(ParquetFilters.scala:168) at org.apache.spark.sql.execution.datasources.parquet.ParquetFilters$$anonfun$5.$anonfun$applyOrElse$73(ParquetFilters.scala:419) at org.apache.spark.sql.execution.datasources.parquet.ParquetFilters.$anonfun$createFilterHelper$9(ParquetFilters.scala:720) at scala.Option.map(Option.scala:230) at org.apache.spark.sql.execution.datasources.parquet.ParquetFilters.createFilterHelper(ParquetFilters.scala:720) at org.apache.spark.sql.execution.datasources.parquet.ParquetFilters.createFilter(ParquetFilters.scala:621) at org.apache.spark.sql.execution.datasources.parquet.ParquetFilterPredicateFactory.$anonfun$createFilter$2(ParquetFilters.scala:885) at scala.collection.immutable.List.flatMap(List.scala:366) at org.apache.spark.sql.execution.datasources.parquet.ParquetFilterPredicateFactory.createFilter(ParquetFilters.scala:885) at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anon$1.apply(ParquetFileFormat.scala:410) at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anon$1.apply(ParquetFileFormat.scala:265) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1$$anon$2.getNext(FileScanRDD.scala:601) ``` ### Why are the changes needed? NPE when subquery returns a NULL literal NPE when comparing against a NULL literal with NullPropagation rule turned off. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? unit test ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#45202 from cosmind-db/cosmind-db/filter-null. Lead-authored-by: cosmind-db <cosmin.dumitru@databricks.com> Co-authored-by: Hyukjin Kwon <gurwls223@gmail.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>

github-actions bot added the SQL label May 8, 2023

cloud-fan reviewed May 9, 2023

View reviewed changes

viirya reviewed May 10, 2023

View reviewed changes

viirya reviewed May 20, 2023

View reviewed changes

viirya approved these changes May 20, 2023

View reviewed changes

ulysses-you force-pushed the SPARK-43402 branch from abd841b to 6859759 Compare June 8, 2023 01:45

ulysses-you force-pushed the SPARK-43402 branch from 6859759 to a825793 Compare June 27, 2023 01:36

ulysses-you added 2 commits July 25, 2023 10:08

FileSourceScanExec supports push down data filter with scalar subquery

def43d6

fix test

f7cf116

ulysses-you force-pushed the SPARK-43402 branch from a825793 to f7cf116 Compare July 25, 2023 02:09

wangyum approved these changes Jul 28, 2023

View reviewed changes

viirya approved these changes Jul 28, 2023

View reviewed changes

ulysses-you closed this in b8cb9d5 Jul 31, 2023

ulysses-you deleted the SPARK-43402 branch July 31, 2023 02:16

cloud-fan reviewed Aug 2, 2023

View reviewed changes

cloud-fan mentioned this pull request Aug 3, 2023

[SPARK-44655][SQL] Make the code cleaner about static and dynamic data/partition filters #42318

Closed

maytasm mentioned this pull request Oct 25, 2023

[SPARK-45621] Add feature to evaluate subquery before push down filter Optimizer rule #43471

Closed

cosmind-db mentioned this pull request Feb 21, 2024

[SPARK-47120][SQL] Null comparison push down data filter from subquery produces in NPE in Parquet filter #45202

Closed

	@transient lazy val selectedPartitions: Array[PartitionDirectory] = {
	val optimizerMetadataTimeNs = relation.location.metadataOpsTimeNs.getOrElse(0L)
	val startTime = System.nanoTime()
	val ret =
	relation.location.listFiles(
	partitionFilters.filterNot(isDynamicPruningFilter), dataFilters)

	@transient protected lazy val dynamicallySelectedPartitions: Array[PartitionDirectory] = {
	val dynamicPartitionFilters = partitionFilters.filter(isDynamicPruningFilter)

	if (dynamicPartitionFilters.nonEmpty) {
	val startTime = System.nanoTime()
	// call the file index for the files matching all filters except dynamic partition filters
	val predicate = dynamicPartitionFilters.reduce(And)
	val partitionColumns = relation.partitionSchema
	val boundPredicate = Predicate.create(predicate.transform {
	case a: AttributeReference =>
	val index = partitionColumns.indexWhere(a.name == _.name)
	BoundReference(index, partitionColumns(index).dataType, nullable = true)
	}, Nil)
	val ret = selectedPartitions.filter(p => boundPredicate.eval(p.values))

[SPARK-43402][SQL] FileSourceScanExec supports push down data filter with scalar subquery #41088

[SPARK-43402][SQL] FileSourceScanExec supports push down data filter with scalar subquery #41088

Conversation

ulysses-you commented May 8, 2023

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

ulysses-you commented May 8, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ulysses-you commented May 16, 2023

viirya left a comment

Choose a reason for hiding this comment

ulysses-you commented Jul 28, 2023

ulysses-you commented Jul 31, 2023

cloud-fan Aug 2, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

epa095 commented Nov 6, 2023

ulysses-you commented Nov 7, 2023

cloud-fan Aug 2, 2023 •

edited