Proposal: drop Spark 4.0 in Hudi 1.3.0 -- the only Spark 4 profile below the parquet 1.16 VARIANT floor

[voonhous]

Proposal: drop the spark4.0 profile and hudi-spark-datasource/hudi-spark4.0.x in 1.3.0. Supported Spark matrix becomes 3.5, 4.1, 4.2.

We want to push hard on VARIANT. Spark 4.0 is not "one minor behind" on variant -- its native variant support sits on the other side of a format boundary. It also reaches EOL on 2026-11-23.

Spark 4.0 ships parquet-java 1.15.2; 4.1 ships 1.16.0. Concretely, on 4.0:

• No VariantLogicalTypeAnnotation -- added in parquet-java 1.16.0 (apache/parquet-java#3070). Variant groups are written as bare, unannotated metadata/value structs, so nothing in the file metadata says "this is a VARIANT". Hudi falls back to a name/arity shape heuristic to recognise them.
• Spark's native read path mis-maps variant fields. 4.0's ParquetUnshreddedVariantConverter looks value/metadata up by name but installs the converters at fixed [value, metadata] array positions; parquet then calls getConverter(fieldIndex) by schema position. So a spec-ordered [metadata, value] file hands metadata bytes to the value converter -> MALFORMED_VARIANT (Spark 4.0.x reads Variant Parquet group fields by position instead of by name, causing MALFORMED_VARIANT #18334). Hudi carries a forked ReadSupport that rewrites the requested schema to [value, metadata] purely to dodge this. Fixed in 4.1 by SPARK-54410. (Hudi's AVRO read path does not go through Spark's row converters and is unaffected.)
• No shredded-variant reconstruction -- buildFullVariantReadSchema is implemented for 4.1/4.2 only; 4.0 falls through to the base None.

4.1 and 4.2 have none of these. This is a capability boundary, not a version gap.

The core problem: a compile floor, not a runtime gate

The annotation gap is not something a runtime check can paper over. hudi-spark4-common is compiled once per Spark profile, and under -Dspark4.0 parquet.version resolves to 1.15.2 -- so while spark4.0 is in the build, shared Spark 4 code cannot reference VariantLogicalTypeAnnotation at all. No config flag, version check, or reflection shim buys that back.

Profile	Spark	parquet-java	VariantLogicalTypeAnnotation
spark3.5	3.5.3	1.13.1	n/a (no VariantType)
spark4.0	4.0.2	1.15.2	no
spark4.1	4.1.1	1.16.0	yes
spark4.2	4.2.0	1.17.0	yes

This is why BaseSpark4Adapter carries a no-op that 4.1 and 4.2 each override identically:

// TODO(#18935) drop-spark4.0: when all remaining 4.x adapters are parquet 1.16+,
// apply variantType() in this base and delete the no-op default plus the
// Spark4_1Adapter override.
protected def applyVariantLogicalType(builder: Types.GroupBuilder[GroupType]) = builder

hudi-spark4-common is exactly where cross-version variant logic belongs. The floor evicts that logic into per-version modules -- the duplication the version-module layout exists to prevent.

What it costs today

Variant features get pushed out of shared code. In #18961, Spark41VariantShreddingSchemaInferrer had to go in hudi-spark4.1.x rather than hudi-spark4-common, because it needs parquet 1.16-era APIs. Compare Spark4VariantShreddingProvider, which is shared across all three minors because it only needs Spark's variant library. That split is the compile floor drawn in code.

Every variant PR pays a per-profile compile tax. #18961 records compile checks under "the spark3.5, spark4.0 and spark4.1 profiles" -- three passes, soon four with 4.2.

~5,000 LOC. hudi-spark4.0.x is 21 files, eight with no 4.1 counterpart -- the forked ReadSupport above, a forked legacy parquet format and reader, and a version-specific extended SQL parser and AST builder.

Refactor multiplication. 34 commits touched the module since 2025-01-01, mostly cross-cutting refactors (#19193, #19195, #19147-#19150, #19132, #19455, #19460) applied N times because N version modules exist.

Why now: sequencing against #18961

#18961 (auto-infer per-file shredding schemas) is what makes shredded files common in the wild for the first time. That flips a latent Spark 4.0 gap into a live one.

• Writes are safe. The inferrer lives in hudi-spark4.1.x, so a 4.0 writer has none on the classpath and keeps writing unshredded.
• The AVRO read path is safe on 4.0. HoodieVariantReconstruction resolves Spark4VariantShreddingProvider from hudi-spark4-common, and fails loudly rather than dropping typed_value.
• The gap is a 4.0 native read of a shredded file written by a 4.1/4.2 writer in a mixed fleet. There buildFullVariantReadSchema returns the base None, plus SPARK-54410.

I have not verified whether that last case throws MALFORMED_VARIANT or returns wrong results -- and it is awkward to test today because no unit job runs hudi-spark4.0.x (all spark4.0 rows in bot.yml are commented out under [CI-TRIM]; only spark3.5 and spark4.2 run unit tests).

If we keep 4.0, that test plus likely an explicit read guard become prerequisites for flipping the inference default (#18937). If we drop it, neither is needed.

Scope: what this does not fix

Worth stating plainly so expectations are right.

This does not make Hudi's variant files self-describing to other engines. AVRO is the default write-path record type, and it converts variants via AvroSchemaConverterWithTimestampNTZ, which emits a bare GroupType with no annotation. That converter lives in hudi-hadoop-common, which must keep compiling under spark3.5 at parquet 1.13.1 -- so it cannot reference the annotation regardless of what happens to 4.0. Only HoodieRowParquetWriteSupport reaches applyVariantLogicalType.

Consequences:

1. The floor rises for hudi-spark4-common only.
2. isVariantPhysicalSchema (the name/arity shape heuristic) stays as the fallback for unannotated files. It cannot be deleted by this change.

Annotating the AVRO path is a follow-up blocked on Spark 3.5, not on 4.0. I will file it separately.

Who is actually affected

Vendors shipping Spark 4.0 do not ship a Hudi version this change would reach:

Runtime	Spark	Hudi
EMR emr-spark-8.0	4.0	1.0.2
EMR 7.10.0	3.5.5-amzn-1	0.15.0-amzn-7
Dataproc 2.3	3.5.3	0.15.0 (optional component)

Databricks Runtime 17.3 LTS ships Spark 4.0.0 into Oct 2028, but does not bundle Hudi.

So the only population this could strand is self-managed deployments on Hudi 1.3.0+ with Spark 4.0 -- who control both versions and can move to 4.1/4.2. Existing Spark 4.0 users also keep working 1.2.x bundles; this removes a build profile going forward, it does not break deployments.

Happy to be corrected if there is a population I have missed.

Proposed motion

Drop spark4.0 in 1.3.0:

• Remove the profile and module; update the README build matrix.
• Move hudi_trino_e2e.yml (currently hardcoded to -Dspark4.0 / the spark402 compose) onto a higher Spark compose -- needs trinocoordinator added to the target pair.
• Collapse applyVariantLogicalType into BaseSpark4Adapter, delete both overrides.
• Delete the bundle-validation spark4.0.0 rows and spark400 base images.
• Keep isVariantPhysicalSchema -- still needed for unannotated files.

Open question: does Hudi require a deprecation-notice release before removal? If so, deprecate in 1.3.0 and remove in 1.4.0.

References

• Parquet 1.16.0 release notes | parquet-java#3070 "Add Variant Logical Type" | Parquet variant spec
• Spark versioning policy | Spark EOL dates
• Spark 4.0 on Amazon EMR | EMR 7.10.0 | Dataproc 2.3
• drop-spark4.0: apply variantType() in BaseSpark4Adapter and remove parquet 1.15 variant shims #18935 (cleanup umbrella) | Spark 4.0.x reads Variant Parquet group fields by position instead of by name, causing MALFORMED_VARIANT #18334 (4.0 field-order workaround) | feat(variant): auto-infer per-file variant shredding schemas on shredding inference #18961 (auto-inference) | Auto-infer variant shredding schema and enable shredding by default (SPARK + AVRO record types) #18937 (default flip)
• SPARK-54410 (read variant fields by name, fixed in 4.1) | SPARK-53659 (per-file shredding inference)

[voonhous]

@CTTY @wombatu-kun @yihua For visibility

Removing Spark 4.0 pretty much reduces our test coverage scope too if we want to push for Variant.

[wombatu-kun]

+1, though the load-bearing argument is narrower than the TL;DR: hudi-spark4-common compiles once per profile, so while spark4.0 builds it the source cannot reference VariantLogicalTypeAnnotation at all. No config or runtime gate buys that back. Two corrections before this goes to the dev list.

Shredding is unreachable at default config. HoodieSparkSchemaConverters always builds an unshredded createVariant(...), and both write supports shred only when the schema already carries typed_value or the test-only hoodie.parquet.variant.force.shredding.schema.for.test is set. "Spark 4.0 writes shredded files its reader cannot rebuild" is not reachable today; the annotation gap is unconditional, so I would lead with that and drop the shredding paragraph.

The drop does not make variant files self-describing. AVRO is the default write-path record type, and the avro path writes the variant group through AvroSchemaConverterWithTimestampNTZ as a bare GroupType on every profile; only HoodieRowParquetWriteSupport reaches applyVariantLogicalType. That converter sits in hudi-hadoop-common, which still compiles against parquet 1.13.1 under spark3.5, so the floor rises for hudi-spark4-common only and 4.1/4.2 files stay as opaque to other engines as 4.0 ones. Two consequences: annotating on the avro path is a follow-up worth filing alongside this, and isVariantPhysicalSchema has to stay as a fallback rather than being replaced by an annotation check.

One practical note for the drop PR: hudi_trino_e2e.yml hardcodes -Dspark4.0 and the spark402 composes are the only ones carrying a trinocoordinator service, so that workflow needs moving onto the spark411 compose in the same change.

On gate (2): DBR 17.3 LTS ships Spark 4.0.0 with an LTS window into 2028, so that is the population to name in the deprecation note.

[voonhous]

Thanks @wombatu-kun -- all three land, and I've rewritten the top post around them rather than patching it.

Shredding reachability. Confirmed: HoodieSparkSchemaConverters only ever builds createVariant(...), and the only production route to typed_value is a schema that already carries it. Removed that argument entirely.

Self-describing. Confirmed, and the most useful catch. AvroSchemaConverterWithTimestampNTZ emits a bare GroupType, and it sits in hudi-hadoop-common, which must keep compiling under spark3.5 at parquet 1.13.1 -- so the annotation is out of reach regardless of 4.0. The top post now has an explicit "what this does not fix" section, and the checklist says keep isVariantPhysicalSchema. Bucket B item 2 of #18935 is dead; I'll file the avro-path annotation follow-up separately, blocked on 3.5.

CI. Right, hudi_trino_e2e.yml runs -Dspark4.0 on push/PR. Narrowed to the accurate claim: no unit job exercises hudi-spark4.0.x. The compose move is mechanical -- we already have higher Spark composes to switch onto, it just needs trinocoordinator on the target pair. It's a checklist item, not a blocker.

I've also taken your framing as the TL;DR, since it's sharper than what I had: hudi-spark4-common compiles once per profile, so while spark4.0 is in the build the source cannot reference VariantLogicalTypeAnnotation at all.

Two things added:

Sequencing against #18961. Auto-inference is what makes shredded files common in the wild for the first time, which flips the 4.0 gap from latent to live. Writes are guarded and the AVRO read path is fine on 4.0; the exposure is a 4.0 native read of a 4.1/4.2-written shredded file. Keep 4.0 and that needs a test plus probably a read guard before #18937 flips; drop it and neither is needed.

On DBR 17.3 LTS -- I chased the vendor picture and I think it argues the other way. Every runtime shipping Spark 4.0 ships a Hudi far behind this change: EMR's Spark 4.0 (emr-spark-8.0) bundles Hudi 1.0.2, EMR 7.10.0 bundles 0.15.0-amzn-7 on Spark 3.5.5, Dataproc 2.3 is Spark 3.5.3 with Hudi 0.15.0. DBR doesn't bundle Hudi at all. So the only population 1.3.0 could strand is self-managed on Spark 4.0, who control both versions. Table's in the top post -- happy to be corrected if I've missed a population.