[GLUTEN-3547][CORE] [VL] Add native parquet writer in spark 3.4 #3690

JkSelf · 2023-11-13T03:28:21Z

What changes were proposed in this pull request?

Since the introduction of the WriteFilesExec operator in Spark 3.4 to facilitate write operations (SPARK-41708), we can now utilize this operator to enable native Parquet write. This PR introduces the WriteFilesExecTransformer to delegate the process to the Velox TableWriteNode, and ColumnarWriteFilesExec is added to implement the doExecuteWrite() method by converting RDD[ColumnarBatch] to RDD[WriterCommitMessage]。

Due to the differences between vanilla Spark and Velox, this PR has dependencies on the following three upstream Velox PRs:
facebookincubator/velox#8089
facebookincubator/velox#8090
facebookincubator/velox#8091

Limitations and failed unit test recorded here.

How was this patch tested?

Enable spark.gluten.sql.native.writer.enabled config in spark 3.4 unit test.

github-actions · 2023-11-13T03:28:42Z

#3547

github-actions · 2023-11-13T03:28:53Z

Run Gluten Clickhouse CI

github-actions · 2023-11-13T09:44:55Z

Run Gluten Clickhouse CI

github-actions · 2023-11-21T07:55:12Z

Run Gluten Clickhouse CI

github-actions · 2023-11-21T10:16:12Z

Run Gluten Clickhouse CI

github-actions · 2023-11-28T07:43:14Z

Run Gluten Clickhouse CI

github-actions · 2023-12-06T10:07:34Z

Run Gluten Clickhouse CI

github-actions · 2023-12-06T10:42:30Z

Run Gluten Clickhouse CI

github-actions · 2023-12-06T13:03:00Z

Run Gluten Clickhouse CI

github-actions · 2023-12-06T13:44:16Z

Run Gluten Clickhouse CI

github-actions · 2023-12-07T00:20:42Z

Run Gluten Clickhouse CI

github-actions · 2023-12-07T05:06:03Z

Run Gluten Clickhouse CI

github-actions · 2023-12-07T10:04:03Z

Run Gluten Clickhouse CI

github-actions · 2023-12-07T10:26:06Z

Run Gluten Clickhouse CI

github-actions · 2023-12-08T04:12:29Z

Run Gluten Clickhouse CI

github-actions · 2023-12-08T08:38:30Z

Run Gluten Clickhouse CI

github-actions · 2023-12-08T10:29:55Z

Run Gluten Clickhouse CI

github-actions · 2023-12-08T12:56:52Z

Run Gluten Clickhouse CI

github-actions · 2023-12-08T23:13:40Z

Run Gluten Clickhouse CI

github-actions · 2023-12-25T12:52:29Z

Run Gluten Clickhouse CI

github-actions · 2023-12-26T02:39:42Z

Run Gluten Clickhouse CI

github-actions · 2023-12-26T04:59:43Z

Run Gluten Clickhouse CI

github-actions · 2023-12-26T06:33:42Z

Run Gluten Clickhouse CI

github-actions · 2023-12-26T07:32:34Z

Run Gluten Clickhouse CI

github-actions · 2023-12-26T07:34:12Z

Run Gluten Clickhouse CI

...ends-velox/src/test/scala/org/apache/spark/sql/execution/VeloxParquetWriteForHiveSuite.scala

...ds-velox/src/test/scala/io/glutenproject/execution/VeloxParquetDataTypeValidationSuite.scala

backends-velox/src/main/scala/org/apache/spark/sql/execution/VeloxColumnarWriteFilesExec.scala

github-actions · 2023-12-27T03:01:08Z

Run Gluten Clickhouse CI

ulysses-you

thank you @JkSelf

github-actions · 2023-12-27T07:48:15Z

Run Gluten Clickhouse CI

zhztheplayer · 2023-12-27T08:57:23Z

gluten-core/src/main/scala/io/glutenproject/extension/columnar/TransformHintRule.scala

@@ -267,7 +268,7 @@ case class FallbackEmptySchemaRelation() extends Rule[SparkPlan] {
          TransformHints.tagNotTransformable(p, "at least one of its children has empty output")
          p.children.foreach {
            child =>
-              if (child.output.isEmpty) {
+              if (child.output.isEmpty && !child.isInstanceOf[WriteFilesExec]) {


What's the reason of making the change? Thanks.

@zhztheplayer The output of WriteFilesExec is empty. So it will fallback if no this limitation.

zhztheplayer · 2023-12-27T09:00:19Z

backends-velox/src/main/scala/io/glutenproject/backendsapi/velox/SparkPlanExecApiImpl.scala

+    val Array(major, minor, _) = SparkShimLoader.getSparkShims.getShimDescriptor.toString.split('.')
+    if (major.toInt > 3 || (major.toInt == 3 && (minor.toInt >= 4))) {
+      List()
+    } else {
+      List(spark => NativeWritePostRule(spark))
+    }


Can we somehow do this in shim? Would that require for lot of effort?

@zhztheplayer Good catch. I will add this in shim in the following PRs. Thanks.

JkSelf force-pushed the write-transformer branch from 2264510 to c1aceb5 Compare November 13, 2023 09:44

zhouyuan mentioned this pull request Nov 16, 2023

[VL] partition write support #2813

Closed

JkSelf force-pushed the write-transformer branch from c1aceb5 to 24f899e Compare November 21, 2023 07:54

JkSelf mentioned this pull request Nov 23, 2023

[GLUTEN-3547][VL][CORE] Add WriteFiles into spark32 & spark33 shim #3800

Merged

JkSelf force-pushed the write-transformer branch from b8abf14 to acb1897 Compare November 28, 2023 07:42

JkSelf force-pushed the write-transformer branch from acb1897 to 44fd20a Compare December 6, 2023 10:07

JkSelf force-pushed the write-transformer branch from 44fd20a to 97c9b92 Compare December 6, 2023 10:41

JkSelf force-pushed the write-transformer branch from 0fa26d0 to f211d53 Compare December 7, 2023 10:03

JkSelf force-pushed the write-transformer branch from f211d53 to bf51603 Compare December 7, 2023 10:25

JkSelf force-pushed the write-transformer branch from 247dda2 to 91e1e8f Compare December 8, 2023 12:56

JkSelf force-pushed the write-transformer branch from 91e1e8f to aa51057 Compare December 8, 2023 23:13

JkSelf force-pushed the write-transformer branch from 9e43eae to 230950e Compare December 26, 2023 04:59

JkSelf force-pushed the write-transformer branch from 230950e to ced397b Compare December 26, 2023 06:33

JkSelf force-pushed the write-transformer branch from 9890d72 to 243084c Compare December 26, 2023 07:33

ayushi-agarwal reviewed Dec 26, 2023

View reviewed changes

...ends-velox/src/test/scala/org/apache/spark/sql/execution/VeloxParquetWriteForHiveSuite.scala Outdated Show resolved Hide resolved

ayushi-agarwal reviewed Dec 26, 2023

View reviewed changes

...ds-velox/src/test/scala/io/glutenproject/execution/VeloxParquetDataTypeValidationSuite.scala Show resolved Hide resolved

ulysses-you reviewed Dec 27, 2023

View reviewed changes

backends-velox/src/main/scala/org/apache/spark/sql/execution/VeloxColumnarWriteFilesExec.scala Outdated Show resolved Hide resolved

JkSelf force-pushed the write-transformer branch from 243084c to 00c4e56 Compare December 27, 2023 03:00

ulysses-you previously approved these changes Dec 27, 2023

View reviewed changes

JkSelf dismissed ulysses-you’s stale review via 2c10863 December 27, 2023 07:47

JkSelf force-pushed the write-transformer branch from 00c4e56 to 2c10863 Compare December 27, 2023 07:47

zhztheplayer approved these changes Dec 27, 2023

View reviewed changes

zhztheplayer reviewed Dec 27, 2023

View reviewed changes

JkSelf merged commit fd33a93 into apache:main Dec 27, 2023
18 checks passed

JkSelf added 5 commits December 27, 2023 23:02

Add native write support in spark 3.4

23ac4df

Resolve comments

54deba6

Resolve comments

9f72037

code format

f19cf3a

Resolve comments

2c10863

Surbhi-Vijay mentioned this pull request Dec 28, 2023

[GLUTEN-4218] fix: Fix Gluten CPP build breaks #4217

Merged

JkSelf mentioned this pull request Dec 29, 2023

[VL]Move getExtendedColumnarPostRules method into shim layer #4226

Merged

ulysses-you mentioned this pull request Jan 2, 2024

[VL] Followup for native write files #4246

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[GLUTEN-3547][CORE] [VL] Add native parquet writer in spark 3.4 #3690

[GLUTEN-3547][CORE] [VL] Add native parquet writer in spark 3.4 #3690

JkSelf commented Nov 13, 2023 •

edited

github-actions bot commented Nov 13, 2023

github-actions bot commented Nov 13, 2023

github-actions bot commented Nov 13, 2023

github-actions bot commented Nov 21, 2023

github-actions bot commented Nov 21, 2023

github-actions bot commented Nov 28, 2023

github-actions bot commented Dec 6, 2023

github-actions bot commented Dec 6, 2023

github-actions bot commented Dec 6, 2023

github-actions bot commented Dec 6, 2023

github-actions bot commented Dec 7, 2023

github-actions bot commented Dec 7, 2023

github-actions bot commented Dec 7, 2023

github-actions bot commented Dec 7, 2023

github-actions bot commented Dec 8, 2023

github-actions bot commented Dec 8, 2023

github-actions bot commented Dec 8, 2023

github-actions bot commented Dec 8, 2023

github-actions bot commented Dec 8, 2023

github-actions bot commented Dec 25, 2023

github-actions bot commented Dec 26, 2023

github-actions bot commented Dec 26, 2023

github-actions bot commented Dec 26, 2023

github-actions bot commented Dec 26, 2023

github-actions bot commented Dec 26, 2023

github-actions bot commented Dec 27, 2023

ulysses-you left a comment

github-actions bot commented Dec 27, 2023

zhztheplayer Dec 27, 2023

JkSelf Dec 28, 2023

zhztheplayer Dec 27, 2023

JkSelf Dec 28, 2023

[GLUTEN-3547][CORE] [VL] Add native parquet writer in spark 3.4 #3690

[GLUTEN-3547][CORE] [VL] Add native parquet writer in spark 3.4 #3690

Conversation

JkSelf commented Nov 13, 2023 • edited

What changes were proposed in this pull request?

How was this patch tested?

github-actions bot commented Nov 13, 2023

github-actions bot commented Nov 13, 2023

github-actions bot commented Nov 13, 2023

github-actions bot commented Nov 21, 2023

github-actions bot commented Nov 21, 2023

github-actions bot commented Nov 28, 2023

github-actions bot commented Dec 6, 2023

github-actions bot commented Dec 6, 2023

github-actions bot commented Dec 6, 2023

github-actions bot commented Dec 6, 2023

github-actions bot commented Dec 7, 2023

github-actions bot commented Dec 7, 2023

github-actions bot commented Dec 7, 2023

github-actions bot commented Dec 7, 2023

github-actions bot commented Dec 8, 2023

github-actions bot commented Dec 8, 2023

github-actions bot commented Dec 8, 2023

github-actions bot commented Dec 8, 2023

github-actions bot commented Dec 8, 2023

github-actions bot commented Dec 25, 2023

github-actions bot commented Dec 26, 2023

github-actions bot commented Dec 26, 2023

github-actions bot commented Dec 26, 2023

github-actions bot commented Dec 26, 2023

github-actions bot commented Dec 26, 2023

github-actions bot commented Dec 27, 2023

ulysses-you left a comment

Choose a reason for hiding this comment

github-actions bot commented Dec 27, 2023

zhztheplayer Dec 27, 2023

Choose a reason for hiding this comment

JkSelf Dec 28, 2023

Choose a reason for hiding this comment

zhztheplayer Dec 27, 2023

Choose a reason for hiding this comment

JkSelf Dec 28, 2023

Choose a reason for hiding this comment

JkSelf commented Nov 13, 2023 •

edited