Skip to content

[WIP][GLUTEN-10215][VL] Delta 4 native write support for Spark 4.0#11924

Draft
malinjawi wants to merge 2 commits intoapache:mainfrom
malinjawi:wip-delta40-native-write
Draft

[WIP][GLUTEN-10215][VL] Delta 4 native write support for Spark 4.0#11924
malinjawi wants to merge 2 commits intoapache:mainfrom
malinjawi:wip-delta40-native-write

Conversation

@malinjawi
Copy link
Copy Markdown
Contributor

What changes are proposed in this pull request?

This PR adds native Delta Lake write support for Spark 4.0 + Delta 4.0 in the Velox backend.

Main changes:

  • Add Delta 4 write offload integration in Delta write operators and command offload path.
  • Wire Delta-specific parquet file format and optimistic transaction write path for native execution.
  • Add native write statistics tracking plumbing used by Delta write.
  • Add Delta optimized writer offload integration for the Delta 4 path.

Scope note:

  • This PR is focused on native write support only.
  • Delta 4 test-suite expansion is split into separate PR(s) for reviewability.

Part of #10215.

How was this patch tested?

  • Compile/build validation with Spark 4.0 + Delta 4 profile:
./build/mvn -pl backends-velox -am -Pbackends-velox,delta,spark-4.0,scala-2.13,java-17 -DskipTests -Dcheckstyle.skip=true -Dspotless.skip=true -Dscalastyle.skip=true compile

@github-actions github-actions bot added the VELOX label Apr 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant