refactor(spark): dedup catalyst utils and analysis rules across Spark version modules - #19149
Merged
Merged
Conversation
…o a common base class Introduce BaseHoodieCatalystExpressionUtils in hudi-spark-common carrying the method bodies that are byte-identical across Spark 3.3-4.2 (normalizeExprs, extractPredicatesWithinOutputSet, canUpCast, and the order-preserving transformation matcher). The ParseToDate/ParseToTimestamp patterns, whose case-class shapes differ across Spark versions, are kept per version behind the unapplyOrderPreservingDateParsing hook. HoodieSpark4CatalystExpressionUtils in hudi-spark4-common now carries the bodies uniform within the Spark 4.x family (getEncoder, matchCast, unapplyCastExpression, date-parsing hook), turning the Spark 4.x objects into empty declarations. The Spark 3.x objects keep only genuine per-version deltas (RowEncoder vs ExpressionEncoder, AnsiCast, EvalMode). The empty HoodieSpark3CatalystExpressionUtils shim is removed; adapter-facing object names are unchanged.
…base and family classes Move the method bodies that are byte-identical across all six Spark version modules into the BaseHoodieCatalystPlanUtils trait in hudi-spark-common (unapplyResolvedTable, projectOverSchema, isRepairTable, getRepairTableChildren, the four index-plan matchers, unapplyInsertIntoStatement, and createProjectForByNameQuery, replacing the dead 'None' default that every version overrode identically). Add family-level bases for bodies identical only within a Spark major version: HoodieSpark3CatalystPlanUtils in hudi-spark3-common (unapplyUpdateAction with the 2-field UpdateAction, extractJsonFromSerializedOffset with the pre-4.1 package) and HoodieSpark4CatalystPlanUtils in hudi-spark4-common (7-field MergeIntoTable, 5-field LogicalRelation scan, error-class based failAnalysisForMIT and failTableNotFound). Per-version objects keep only genuine Spark API deltas: pattern arities of MergeIntoTable/ScanOperation/LogicalRelation/UpdateAction, the SerializedOffset package move in Spark 4.1, pre-error-class failure messages on 3.3, the _LEGACY_ERROR_TEMP_2309 error class on 3.4, and the Spark 3.4-only default-columns workaround in unapplyInsertIntoStatement. Adapter-facing object names are unchanged.
…e classes Nothing in this family is identical across all six Spark versions, so the bodies are shared at the Spark-major-version level: HoodieSpark3SchemaUtils in hudi-spark3-common carries getSchema (pre-4.0 JdbcUtils.getSchema signature without the Connection argument), and HoodieSpark4SchemaUtils in hudi-spark4-common carries all three methods, which are uniform across 4.0-4.2, turning those objects into empty declarations. Spark 3.x objects keep the per-version deltas: the 3-arg SchemaUtils.checkColumnNameDuplication on 3.3 (colType parameter removed in Spark 3.4) and StructType#toAttributes on 3.3/3.4 (moved to DataTypeUtils in Spark 3.5, SPARK-44353).
…ss Spark 4.x The three-level preprocess logic (partition-spec normalization, user-specified column projection, output-column resolution with default-value support, and the arity-mismatch error rewrite) is byte-identical across Spark 4.0-4.2, so it moves into the abstract HoodieSpark4ResolveColumnsForInsertInto base class in hudi-spark4-common. The per-version case classes keep only apply(), whose InsertIntoStatement pattern arity differs in Spark 4.2 (two fields added). The DataSourceV2ToV1Fallback rules stay fully per-version: every adjacent pair differs in InsertIntoStatement and/or DataSourceV2Relation case-class shapes, so no clean seam exists within the family. Spark 3.5 keeps its full copy of ResolveColumnsForInsertInto since hudi-spark3-common also compiles against Spark 3.3/3.4, which lack the required APIs. Rule class names are unchanged (HoodieAnalysis instantiates them reflectively by name).
Collaborator
wombatu-kun
approved these changes
Jul 3, 2026
This was referenced Jul 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe the issue this Pull Request addresses
The six Spark version modules (hudi-spark3.3.x through hudi-spark4.2.x) carry near-duplicate copies of
HoodieSparkXXCatalystExpressionUtils(92-99% mutually similar),HoodieSparkXXCatalystPlanUtils(88-99%),HoodieSparkXXSchemaUtils(93-98%), and the Spark 4.xResolveColumnsForInsertIntoanalysis rule. Every shared line is maintained and covered six times.Summary and Changelog
BaseHoodieCatalystExpressionUtils(hudi-spark-common) now carries the bodies byte-identical across Spark 3.3-4.2 (normalizeExprs,extractPredicatesWithinOutputSet,canUpCast, the order-preserving transformation matcher). Version-shape-dependentParseToDate/ParseToTimestamppatterns sit behind a protectedunapplyOrderPreservingDateParsinghook.BaseHoodieCatalystPlanUtils(hudi-spark-common) gains the six-way-identical bodies (unapplyResolvedTable,projectOverSchema,isRepairTable,getRepairTableChildren, index-plan matchers,unapplyInsertIntoStatement,createProjectForByNameQuery). New family basesHoodieSpark3CatalystPlanUtils/HoodieSpark4CatalystPlanUtilscarry family-uniform bodies.HoodieSpark3SchemaUtils/HoodieSpark4SchemaUtilshost the family-uniform schema utils; the Spark 4.x objects become empty declarations.preprocesschain of the Spark 4.xResolveColumnsForInsertIntorule moves to an abstract base inhudi-spark4-common; 4.0/4.1/4.2 keep onlyapply()(Spark 4.2'sInsertIntoStatementhas a different arity).MergeIntoTable5/6/7 fields,ScanOperation,LogicalRelation,UpdateAction), encoder differences (RowEncodervsExpressionEncoder),AnsiCastvsEvalMode, error-class differences, the Spark 4.1SerializedOffsetpackage move, and the Spark 3.4-only default-columns insert workaround ([SPARK-38795][SQL] Support INSERT INTO user specified column lists with DEFAULT values spark#36077, fixed in 3.5).Impact
No behavior change intended; where copies genuinely differ, the delta remains as a per-version override. Shared bodies are compiled and covered once instead of six times.
Risk Level
low. Every moved body is line-identical to what already compiles on all versions the hosting module builds against; imports were cross-checked per module compile matrix; CI compiles all six Spark profiles.
Documentation Update
none
Contributor's checklist