Hudi uniform support #2333

the-other-tim-brown · 2023-11-26T16:52:15Z

Which Delta project/connector is this regarding?

Description

This change aims to add support for Hudi in Uniform
The changes were mostly copied from OneTable which has a working version of Delta to Hudi already

How was this patch tested?

Some basic tests are added

Does this PR introduce any user-facing changes?

Yes, this allows users to expose their Delta tables as Hudi

the-other-tim-brown · 2023-11-26T16:58:19Z

This is my first time contributing to Delta Lake and I need some assistance on a few points:

How do I use intellij to run and debug tests? I run into kerberos issues still after many hours of trying to get this environment setup. Is there a quickstart guide I've missed?
How do I run a test suite against a particular version of spark/scala. Hudi only supports up to spark 3.4 as of the 0.14.0 release so we won't be able to run with 3.5 in the tests.
General guidance on UniForm integration. I've just copied over what exists for Iceberg and currently put the code in the same module. I'm guessing we'll want to create a similar module for Hudi if your goal is to keep the dependencies lean and only include them if required.

vkorukanti · 2023-11-27T16:35:59Z

This is my first time contributing to Delta Lake and I need some assistance on a few points:

How do I use intellij to run and debug tests? I run into kerberos issues still after many hours of trying to get this environment setup. Is there a quickstart guide I've missed?

These are the instructions that work for me. Can you tell bit more about the Kerberos issue?

How do I run a test suite against a particular version of spark/scala. Hudi only supports up to spark 3.4 as of the 0.14.0 release so we won't be able to run with 3.5 in the tests.

@lzlfred ^

General guidance on UniForm integration. I've just copied over what exists for Iceberg and currently put the code in the same module. I'm guessing we'll want to create a similar module for Hudi if your goal is to keep the dependencies lean and only include them if required.

@lzlfred ^

the-other-tim-brown · 2023-11-28T02:56:44Z

This is my first time contributing to Delta Lake and I need some assistance on a few points:

How do I use intellij to run and debug tests? I run into kerberos issues still after many hours of trying to get this environment setup. Is there a quickstart guide I've missed?

These are the instructions that work for me. Can you tell bit more about the Kerberos issue?

I run into this error which seems like it may be a version issue?

An exception or error caused a run to abort: 'void org.apache.hadoop.security.HadoopKerberosName.setRuleMechanism(java.lang.String)' 
java.lang.NoSuchMethodError: 'void org.apache.hadoop.security.HadoopKerberosName.setRuleMechanism(java.lang.String)'
	at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:84)
	at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:315)
	at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:300)
	at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:575)
	at org.apache.spark.util.Utils$.$anonfun$getCurrentUserName$1(Utils.scala:2416)
	at scala.Option.getOrElse(Option.scala:189)
	at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2416)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:334)
	at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2888)
	at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:1099)
	at scala.Option.getOrElse(Option.scala:189)
	at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:1093)
	at org.apache.spark.sql.delta.ConvertToHudiSuite.createSparkSession(ConvertToHudiSuite.scala:105)
	at org.apache.spark.sql.delta.ConvertToHudiSuite.beforeAll(ConvertToHudiSuite.scala:43)```

Tagar · 2023-11-28T05:33:26Z

java.lang.NoSuchMethodError: 'void org.apache.hadoop.security.HadoopKerberosName.setRuleMechanism

Not sure how this was tested, perhaps hadoop and hadoop-common both need to be pinned to the same version, something like 3.3.6

lzlfred · 2023-11-29T23:08:21Z

@the-other-tim-brown Thanks for putting this together. Delta appreciate this effort and lets work together to add the support.

the current master & delta 3.0 branch is on Spark 3.5; I am not aware of any way to build and run with another Spark version.
Agree... We should have a separate module for Hudi.

the-other-tim-brown · 2023-11-30T02:31:17Z

@the-other-tim-brown Thanks for putting this together. Delta appreciate this effort and lets work together to add the support.

the current master & delta 3.0 branch is on Spark 3.5; I am not aware of any way to build and run with another Spark version.

Agree... We should have a separate module for Hudi.

If we are ok just comparing the files in the current view of the table, we can avoid bringing in the hudi spark bundle which will solve this issue. Is that ok?
I will need help with this, this was my first time using sbt and I'm not sure what all needs to be shaded vs not.

lzlfred · 2023-12-05T02:27:49Z

If we are ok just comparing the files in the current view of the table, we can avoid bringing in the hudi spark bundle which will solve this issue. Is that ok?

Yes sure. at this point we do not need to read the hudi table by Spark yet.

I will need help with this, this was my first time using sbt and I'm not sure what all needs to be shaded vs not.

myself is not a sbt expert either... i will circle this back to the committers and see.

lzlfred · 2024-02-14T04:53:25Z

without being able to review this in detail yet, here are a few high level feedbacks:

we should make hudi in its own package/module to avoid fat jars that contains dependencies of multiple formats.
we should make sure hudi metadata dir is whitelisted in Delta's vacuum logic, which deletes everything Not on its white list or not tracked by delta log. code pointer: https://github.com/delta-io/delta/blob/master/spark/src/main/scala/org/apache/spark/sql/delta/DeltaTable.scala#L217
I suggest we should add more unit tests such as doing insert/update/delete with Delta, and then read with hudi to see file lists matches. More unit tests including schema evolution, partitioning, statistics conversion can come in future.

I will review this in more detail in the next few weeks. Thanks for the effort !

lzlfred

looks great ! plz do

clean up and make separate Hudi package as previously mentioned.
do a grep and fix other Iceberg occurrences mostly in comments.

lzlfred · 2024-03-07T05:09:02Z

iceberg/src/main/scala/org/apache/spark/sql/delta/icebergShaded/HudiConversionTransaction.scala

+import scala.collection.JavaConverters._
+
+/**
+ * Used to prepare (convert) and then commit a set of Delta actions into the Iceberg table located


Suggested change

* Used to prepare (convert) and then commit a set of Delta actions into the Iceberg table located

* Used to prepare (convert) and then commit a set of Delta actions into the Hudi table metadata located

lzlfred · 2024-03-07T05:09:15Z

iceberg/src/main/scala/org/apache/spark/sql/delta/icebergShaded/HudiConversionTransaction.scala

+ * at the same path as [[postCommitSnapshot]]
+ *
+ *
+ * @param conf Configuration for Iceberg Hadoop interactions.


Suggested change

* @param conf Configuration for Iceberg Hadoop interactions.

* @param conf Configuration for Hudi Hadoop interactions.

lzlfred · 2024-03-07T05:09:26Z

iceberg/src/main/scala/org/apache/spark/sql/delta/icebergShaded/HudiConversionTransaction.scala

+ *
+ *
+ * @param conf Configuration for Iceberg Hadoop interactions.
+ * @param postCommitSnapshot Latest Delta snapshot associated with this Iceberg commit.


Suggested change

* @param postCommitSnapshot Latest Delta snapshot associated with this Iceberg commit.

* @param postCommitSnapshot Latest Delta snapshot associated with this Hudi commit.

lzlfred · 2024-03-07T05:09:37Z

iceberg/src/main/scala/org/apache/spark/sql/delta/icebergShaded/HudiConversionTransaction.scala

+ *
+ * @param conf Configuration for Iceberg Hadoop interactions.
+ * @param postCommitSnapshot Latest Delta snapshot associated with this Iceberg commit.
+ * @param tableOp How to instantiate the underlying Iceberg table. Defaults to WRITE_TABLE.


Suggested change

* @param tableOp How to instantiate the underlying Iceberg table. Defaults to WRITE_TABLE.

* @param tableOp How to instantiate the underlying Hudi table. Defaults to WRITE_TABLE.

lzlfred · 2024-03-07T05:11:46Z

iceberg/src/main/scala/org/apache/spark/sql/delta/icebergShaded/HudiConverter.scala

+import scala.util.control.NonFatal
+
+object HudiConverter {
+


nit: remove empty line

lzlfred · 2024-03-07T05:35:53Z

iceberg/src/main/scala/org/apache/spark/sql/delta/icebergShaded/HudiConverter.scala

+    if (!UniversalFormat.hudiEnabled(snapshotToConvert.metadata)) {
+      return None
+    }
+    txn.catalogTable match {


Iceberg almost always require a catalog so we had this requirement. If thats not true for Hudi, I would suggest to release this contraint, otherwise it wont work if we access/write to the Delta table by path.

However on the other hand I get the fact that we need to pass in the tableName to Hudi table creator.

Can we add a TODO here saying requiring a catalogTable is a constraint that we can release in the future ?

We don't require a catalog for Hudi. I just figured if there is a name present we can use it when creating the table.

lzlfred · 2024-03-07T05:36:57Z

spark/src/main/scala/org/apache/spark/sql/delta/hooks/HudiConverterHook.scala

+        !UniversalFormat.hudiEnabled(postCommitSnapshot.metadata)) {
+      return
+    }
+


extra line.

lzlfred · 2024-03-07T05:37:29Z

iceberg/src/main/scala/org/apache/spark/sql/delta/icebergShaded/HudiConversionTransaction.scala

+  }
+
+  private def runArchiver(table: HoodieJavaTable[_ <: HoodieAvroPayload],
+                          config: HoodieWriteConfig,


ident... plz ident 4 space from private.

lzlfred · 2024-03-07T05:37:37Z

iceberg/src/main/scala/org/apache/spark/sql/delta/icebergShaded/HudiConversionTransaction.scala

+  }
+
+  private def markInstantsAsCleaned(table: HoodieJavaTable[_],
+                                    writeConfig: HoodieWriteConfig,


lzlfred · 2024-03-07T05:37:59Z

iceberg/src/main/scala/org/apache/spark/sql/delta/icebergShaded/HudiConversionTransaction.scala

+  }
+
+  private def getWriteConfig(schema: Schema,
+                             numCommitsToKeep: Int,


the-other-tim-brown · 2024-03-07T10:02:12Z

looks great ! plz do

clean up and make separate Hudi package as previously mentioned.

do a grep and fix other Iceberg occurrences mostly in comments.

@lzlfred I've moved the code into a separate package but I do not have any experience with setting up a new sbt module. After moving the code, the iceberg and hudi modules fail to compile. I've pushed an update so you or someone on your team can help debug this.

build.sbt

… that

lzlfred

what a epic feature ! thanks for all efforts.

lzlfred · 2024-03-08T23:03:56Z

hudi/src/main/scala/org/apache/spark/sql/delta/hudi/HudiConverter.scala

+   * removed from the wait queue. Only one snapshot is queued at any point of time.
+   *
+   */
+  override def enqueueSnapshotForConversion(


Ideally we want to consolidate all these async logic with iceberg conversion... I will take that part and I am not asking for a TODO from this PR.. .

lzlfred · 2024-03-08T23:04:38Z

hudi/src/main/scala/org/apache/spark/sql/delta/hudi/HudiSchemaUtils.scala

+          isNullable)
+      // TODO: Add List and Map support: https://github.com/delta-io/delta/issues/2738
+      case ArrayType(elementType, containsNull) =>
+        throw new UnsupportedOperationException("UniForm doesn't support Array columns")


nit: say Uniform Hudi

lzlfred · 2024-03-08T23:04:45Z

hudi/src/main/scala/org/apache/spark/sql/delta/hudi/HudiSchemaUtils.scala

+        throw new UnsupportedOperationException("UniForm doesn't support Array columns")
+
+      case MapType(keyType, valueType, valueContainsNull) =>
+        throw new UnsupportedOperationException("UniForm doesn't support Map columns")


same as above.

lzlfred · 2024-03-08T23:04:57Z

hudi/src/main/scala/org/apache/spark/sql/delta/hudi/HudiSchemaUtils.scala

+      case atomicType: AtomicType => convertAtomic(atomicType, isNullable)
+
+      case other =>
+        throw new UnsupportedOperationException(s"Cannot convert Delta type $other to Iceberg")


Suggested change

throw new UnsupportedOperationException(s"Cannot convert Delta type $other to Iceberg")

throw new UnsupportedOperationException(s"Cannot convert Delta type $other to Hudi")

scottsand-db

SBT related changes LGTM

lzlfred · 2024-03-13T02:09:31Z

spark/src/main/scala/org/apache/spark/sql/delta/hooks/HudiConverterHook.scala

+
+package org.apache.spark.sql.delta.hooks
+
+import org.apache.spark.sql.SparkSession


idk why scala style does not report error, while plz follow

/spark/src/main/scala/org/apache/spark/sql/delta/hooks/IcebergConverterHook.scala

and make import to be

import org.apache.spark.sql.delta.{OptimisticTransactionImpl, Snapshot, UniversalFormat}
import org.apache.spark.sql.delta.actions.Action
import org.apache.spark.sql.delta.metering.DeltaLogging

import org.apache.spark.sql.SparkSession

spark/src/main/scala/org/apache/spark/sql/delta/UniversalFormat.scala

lzlfred · 2024-03-13T20:32:52Z

spark/src/main/scala/org/apache/spark/sql/delta/UniversalFormat.scala

 import org.apache.spark.sql.SparkSession
 import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.types.{ArrayType, NullType, MapType}


plz swap NullType and MapType to be alphabetical.

the-other-tim-brown added 3 commits November 22, 2023 18:33

basic transaction flow compiles

992fc5e

add hooks for Hudi, convert to avro with hudi utils

a965d49

setup test, add schema converter

1de8899

update suite to use table path, remove column mapping

9747207

the-other-tim-brown and others added 5 commits March 5, 2024 18:03

fix bugs around tracking progress

a4f3e75

Merge branch 'master' into hudi-uniform-support

2f0419d

fix local debug and spark session issue, don't require catalog for hudi

375243a

update logging

74e01fa

add case to vacuum suite, error out if delete vector enabled

070d043

lzlfred reviewed Mar 7, 2024

View reviewed changes

the-other-tim-brown added 5 commits March 7, 2024 01:13

update test cases, address minor comments

8c0d1e0

remove more reference to iceberg, fix indentation

b9a829a

fix whitespace

ca9178e

try to create a new module

b945ff1

minor sbt updates

88b8f97

the-other-tim-brown added 3 commits March 7, 2024 02:12

fix hudi module compile issues

23f3088

update converter reflection path

e692ab0

fix test imports

80a00a9

scottsand-db requested changes Mar 7, 2024

View reviewed changes

build.sbt Show resolved Hide resolved

the-other-tim-brown added 2 commits March 7, 2024 12:24

change timeouts, add to sparkGroups in build.sbt

88e660e

make paths and table names unique per test case

a4e6bb2

the-other-tim-brown added 3 commits March 7, 2024 18:15

fix whitespace

7737272

fix spark dependency

8cda724

add TODOs and schema check when creating table and add test cases for…

8cd2fbd

… that

the-other-tim-brown changed the title ~~[WIP] Hudi uniform support~~ Hudi uniform support Mar 8, 2024

lzlfred approved these changes Mar 8, 2024

View reviewed changes

fix error messages

78e8652

scottsand-db approved these changes Mar 8, 2024

View reviewed changes

lzlfred reviewed Mar 13, 2024

View reviewed changes

the-other-tim-brown added 4 commits March 13, 2024 20:49

style fixes

d43028a

local bundle jar works for creating table

59d83a0

remove print statements from build.sbt, add readme with instructions

eea6a7b

cleanup

e4e9e83

scottsand-db merged commit 9028303 into delta-io:master Mar 29, 2024
7 checks passed

the-other-tim-brown deleted the hudi-uniform-support branch March 29, 2024 16:27

vkorukanti mentioned this pull request Apr 11, 2024

[Feature Request] Support Hudi in Uniform #2312

Closed

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Hudi uniform support #2333

Hudi uniform support #2333

the-other-tim-brown commented Nov 26, 2023 •

edited

the-other-tim-brown commented Nov 26, 2023

vkorukanti commented Nov 27, 2023

the-other-tim-brown commented Nov 28, 2023

Tagar commented Nov 28, 2023

lzlfred commented Nov 29, 2023

the-other-tim-brown commented Nov 30, 2023

lzlfred commented Dec 5, 2023

lzlfred commented Feb 14, 2024

lzlfred left a comment

lzlfred Mar 7, 2024

lzlfred Mar 7, 2024

lzlfred Mar 7, 2024

lzlfred Mar 7, 2024

lzlfred Mar 7, 2024

the-other-tim-brown Mar 7, 2024

lzlfred Mar 7, 2024

the-other-tim-brown Mar 7, 2024

lzlfred Mar 7, 2024

the-other-tim-brown Mar 7, 2024

lzlfred Mar 7, 2024

the-other-tim-brown Mar 7, 2024

lzlfred Mar 7, 2024

the-other-tim-brown Mar 7, 2024

lzlfred Mar 7, 2024

the-other-tim-brown Mar 7, 2024

the-other-tim-brown commented Mar 7, 2024

lzlfred left a comment

lzlfred Mar 8, 2024

lzlfred Mar 8, 2024

lzlfred Mar 8, 2024

lzlfred Mar 8, 2024

scottsand-db left a comment

lzlfred Mar 13, 2024

lzlfred Mar 13, 2024

	* Used to prepare (convert) and then commit a set of Delta actions into the Iceberg table located
	* Used to prepare (convert) and then commit a set of Delta actions into the Hudi table metadata located

	* @param conf Configuration for Iceberg Hadoop interactions.
	* @param conf Configuration for Hudi Hadoop interactions.

	* @param postCommitSnapshot Latest Delta snapshot associated with this Iceberg commit.
	* @param postCommitSnapshot Latest Delta snapshot associated with this Hudi commit.

	* @param tableOp How to instantiate the underlying Iceberg table. Defaults to WRITE_TABLE.
	* @param tableOp How to instantiate the underlying Hudi table. Defaults to WRITE_TABLE.

	throw new UnsupportedOperationException(s"Cannot convert Delta type $other to Iceberg")
	throw new UnsupportedOperationException(s"Cannot convert Delta type $other to Hudi")


		package org.apache.spark.sql.delta.hooks

		import org.apache.spark.sql.SparkSession

Hudi uniform support #2333

Hudi uniform support #2333

Conversation

the-other-tim-brown commented Nov 26, 2023 • edited

Which Delta project/connector is this regarding?

Description

How was this patch tested?

Does this PR introduce any user-facing changes?

the-other-tim-brown commented Nov 26, 2023

vkorukanti commented Nov 27, 2023

the-other-tim-brown commented Nov 28, 2023

Tagar commented Nov 28, 2023

lzlfred commented Nov 29, 2023

the-other-tim-brown commented Nov 30, 2023

lzlfred commented Dec 5, 2023

lzlfred commented Feb 14, 2024

lzlfred left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

the-other-tim-brown commented Mar 7, 2024

lzlfred left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

scottsand-db left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

the-other-tim-brown commented Nov 26, 2023 •

edited