[MINOR][CH] Drop Spark 3.2 / 3.1-era compat workarounds in ClickHouse backend#12541
Merged
jackylee-ch merged 2 commits intoJul 17, 2026
Merged
Conversation
|
Run Gluten Clickhouse CI on x86 |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR removes two Spark 3.2 compatibility workarounds in the ClickHouse backend now that Gluten targets Spark 3.3+ only, simplifying the code by delegating to stable upstream APIs.
Changes:
- Replace reflective access to
WriteJobDescription.bucketSpecwith a directdescription.bucketSpec.isDefinedcheck inCHColumnarWrite. - Remove the local reimplementation of
NestedColumnAliasing.getAttributeToExtractValuesand delegate to Spark’s upstream implementation. - Retain the Gluten-specific nested
GetStructFieldextractor and document why it’s required at the call site.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| backends-clickhouse/src/main/scala/org/apache/spark/sql/execution/CHColumnarWrite.scala | Drops Spark 3.2-era reflection and uses bucketSpec directly for bucketed-write file pattern selection. |
| backends-clickhouse/src/main/scala/org/apache/gluten/extension/ExtendedColumnPruning.scala | Removes duplicated Spark logic and relies on upstream NestedColumnAliasing.getAttributeToExtractValues, keeping the Gluten-specific extractor where needed. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Contributor
…nd CHColumnarWrite Two Spark 3.2-era pieces no longer justified now that Gluten supports only Spark 3.3+: * `ExtendedGeneratorNestedColumnAliasing` used to reimplement `NestedColumnAliasing.getAttributeToExtractValues` locally because the Spark 3.2 version only took two arguments. Every currently supported branch (3.3, 3.4, 3.5, 4.0, 4.1) exposes the three-argument overload, so both call sites now delegate to upstream and the private reimplementation (`collectRootReferenceAndExtractValue`, `getAttributeToExtractValues`, `totalFieldNum`) is removed. The Gluten-specific `collectNestedGetStructFields` extractor is kept, and its call site now carries a comment explaining why it is needed. * `HadoopMapReduceAdapter.isBucketWrite` used reflection to read `WriteJobDescription.bucketSpec`, guarding against Spark 3.2 where the field did not exist. `bucketSpec` is a public `val` on every currently supported branch, so the reflection helper is replaced with a direct `description.bucketSpec.isDefined` check. No behavior change on any supported Spark version.
LuciferYang
force-pushed
the
gluten-minor-ch-spark32-refactor
branch
from
July 17, 2026 05:19
fa2d79c to
f2b5e01
Compare
|
Run Gluten Clickhouse CI on x86 |
…Writer `FakeRowOutputWriter` implements `OutputWriter.path()` without the `override` keyword and carries a comment saying it is left off for Spark 3.1 compatibility. Spark's `OutputWriter.path(): String` is abstract on every currently supported branch (3.3, 3.4, 3.5, 4.0, 4.1), so `override` is legal there and now matches the sibling `override def write(...)` / `override def close()` in the same class. No behavior change on any supported Spark version.
LuciferYang
changed the title
|
Run Gluten Clickhouse CI on x86 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Small ClickHouse-side Spark 3.2 / 3.1-era workarounds that are no longer justified now that Gluten supports only Spark 3.3+:
ExtendedGeneratorNestedColumnAliasing(backends-clickhouse/.../ExtendedColumnPruning.scala)Previously reimplemented
NestedColumnAliasing.getAttributeToExtractValueslocally because the Spark 3.2 version only accepted two arguments. Every currently supported branch (3.3, 3.4, 3.5, 4.0, 4.1) exposes the three-argument overload with theextractorparameter, so both call sites now delegate to upstream and the private reimplementation (collectRootReferenceAndExtractValue,getAttributeToExtractValues,totalFieldNum) is removed. The Gluten-specificcollectNestedGetStructFieldsextractor is kept and the call site now has a comment explaining why it is needed.HadoopMapReduceAdapter.isBucketWrite(backends-clickhouse/.../CHColumnarWrite.scala)Previously used reflection to read
WriteJobDescription.bucketSpec, guarding against Spark 3.2 where the field did not exist.bucketSpecis a publicvalon every currently supported branch, so the reflection helper is replaced with a directdescription.bucketSpec.isDefinedcheck.FakeRowOutputWriter.path()(backends-clickhouse/.../FakeRowOutputWriter.scala)FakeRowOutputWriterimplementsOutputWriter.path()without theoverridekeyword, with a comment saying it is left off for Spark 3.1 compatibility. Spark'sOutputWriter.path(): Stringis abstract on every currently supported branch, sooverrideis legal there and now matches the siblingoverride def write(...)/override def close()in the same class.No behavior change on any supported Spark version.
How was this patch tested?
./build/mvn -Pbackends-clickhouse -Pspark-3.3 -Pdelta -pl backends-clickhouse -am test-compile -DskipTests: SUCCESS./dev/format-scala-code.sh: no diffNestedColumnAliasing.getAttributeToExtractValuesbodies onupstream/branch-3.3,branch-3.4,branch-3.5,branch-4.0,branch-4.1: three-argument overload present on every branch. The internalcanAliasguard reshuffled between 3.4 and 3.5, but for the Gluten call site usingcollectNestedGetStructFieldsthe guard is a semantic no-op (single-attribute chains, never traverses lambda variables).WriteJobDescription.bucketSpecon the same five branches:val bucketSpec: Option[WriterBucketSpec]on all of them.abstract class OutputWriter { def path(): String }on the same five branches:path()is abstract on all of them.GlutenClickHouseHiveTableSuite's "Nested column pruning for Project(Filter(Generate))" cases,GlutenClickHouseTPCHParquetBucketSuite, MergeTree write paths that useFakeRowOutputWriter) rely on native artifacts and Linux, so they run in CI rather than locally.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude claude-opus-4-7