Hudi Connector

[caneGuy]

As we mentioned on the slack and #9641. We need a new separate connector for Hudi now. This is a parent issue we can add discussion and TODOs here.

• Add Hudi connector #10228
• testing
  • typical unit tests like TestHudiPlugin
  • BaseConnectorTest
• Add product test for Hudi #14365
• Add Hudi connector documentation #14429
• Support MOR snapshot querying for Hudi tables #14432
• Support DDL/DML for Hudi tables using the Hudi connector #14433
• Support table redirections from Hive to Hudi #14750
• Support time travel in Hudi connector #15003
• Add a metadata table showing commits in Hudi connector #16149
• Failed to create view based on hudi table #16384
• Support incremenal query in Hudi connector #17976
• Support querying bootstrapped Hudi table in the Hudi connector #17977
• glue support?

cc @vinothchandar @findepi @hashhar @bvaradar @codope @martint @mxdzs0612

[codope]

@caneGuy I have opened an umbrella JIRA to track several tasks related to the new connector. Feel free to add more subtasks. Once we have concretely scoped out the design, let's update this issue.

Right now I am working on a few higher priority tasks related to the next Hudi release. I'll pick up the connector work again in the last week of November. Meanwhile, I would suggest to join Hudi slack if you're not already there. I am "sagar sumit" on slack. We can discuss more over there.

[caneGuy]

great @codope i will join slack now!

[caneGuy]

@codope do you have any POC for MOR table to use common api instead of inputformat?

[vinothchandar]

We don't yet. But it'll come shortly after we get the MOR/RO support landed. Are you interested in taking a swing at that. We expect to open the official PR early next week

[duanyongvictory]

As we mentioned on the slack and #9641. We need a new separate connector for Hudi now. This is a parent issue we can add discussion and TODOs here.

• add Hudi connector

• testing

  • typical unit tests like TestHudiPlugin
  • BaseConnectorTest
  • product tests against supported hive version(s)

• document: type mapping, support hive versions, glue?

• Support query COW table

• Support query MOR table

• Support time travel query

• glue support?

cc @vinothchandar @findepi @hashhar @bvaradar @codope @martint @mxdzs0612

hi , i am eagerly to use snapshot query for mor table，could you please give me the only code for support this feature?
so i can test by myself.
thanks a lot.

[tooptoop4]

@duanyongvictory see #10228

some other todo items are:

• 'table redirection' from hive ie 74b324c#diff-26fea35b3bf02f4ddff9bc2d58707ae304807a65cfb1a7e7d492eb043d174681
• DDL support ie alter add col or drop/rename/change type col
• DML support ie insert/update/delete/merge

[duanyongvictory]

@duanyongvictory see #10228

some other todo items are:

• 'table redirection' from hive ie 74b324c#diff-26fea35b3bf02f4ddff9bc2d58707ae304807a65cfb1a7e7d492eb043d174681
• DDL support ie alter add col or drop/rename/change type col
• DML support ie insert/update/delete/merge

thanks for you reply.
I have already checked for #10228 and the code, but i found no snapshot query supported，so could you please give me a little tip for this code ? so I can read by my self.

as you mentioned about the ddl/dml，in my case only dql are used，we do dml/ddl through spark。

thanks a lot. it will be a great helpful if you could give me the code file for snapshot query .

[duanyongvictory]

@duanyongvictory see #10228
some other todo items are:

• 'table redirection' from hive ie 74b324c#diff-26fea35b3bf02f4ddff9bc2d58707ae304807a65cfb1a7e7d492eb043d174681
• DDL support ie alter add col or drop/rename/change type col
• DML support ie insert/update/delete/merge

thanks for you reply. I have already checked for #10228 and the code, but i found no snapshot query supported，so could you please give me a little tip for this code ? so I can read by my self.

as you mentioned about the ddl/dml，in my case only dql are used，we do dml/ddl through spark。

thanks a lot. it will be a great helpful if you could give me the code file for snapshot query .

@tooptoop4
sorry to interrupt you again.
as i asked, any ideas could provide for me ? thanks.

[codope]

@duanyongvictory the MOR snapshot query support is planned in the next quarter once the base PR merges. https://issues.apache.org/jira/browse/HUDI-2740

Would you like to contribute to this feature? I can share details of what changes would be required.

[duanyongvictory]

@duanyongvictory the MOR snapshot query support is planned in the next quarter once the base PR merges. https://issues.apache.org/jira/browse/HUDI-2740

Would you like to contribute to this feature? I can share details of what changes would be required.

i would happy to.
what changes do i need to know? i also want to do some contribute.
these days i have been study snapshot query for prestodb, why could prestdb ahead?

[codope]

@duanyongvictory Great to hear that! Your contribution will help a lot in accelerating the write support. We're discussing in Hudi slack in https://apache-hudi.slack.com/archives/C02L9R88RJP/p1663729133329139
Let's take the discussion there.

[duanyongvictory]

@duanyongvictory Great to hear that! Your contribution will help a lot in accelerating the write support. We're discussing in Hudi slack in https://apache-hudi.slack.com/archives/C02L9R88RJP/p1663729133329139 Let's take the discussion there.

Could you invite me to this workplace?
My google account is not related to this workplace.
My google account is duanyong0079@gmail.com.
thanks.

[codope]

@duanyongvictory I've sent an invite to your email. Please check.

[zorrofox]

Do we have any updates for Glue support?

[ebyhr]

I don't think we have.

[MateusCastello]

Do we have any updates for Glue support?
@zorrofox

I'm acctually able to fetch data from Glue managed Tables using "hive.metastore=glue" and passing the correct AWS role using "hive.metastore.glue.iam-role", but Only when I set retry-policy to None, disabling fault-tolerant Query Execution.

When Retry-Policy is set to "TASK" I get a Java Error saying that "this connector don't have memory accounting capabilities for ConnectorSplit"

java.lang.UnsupportedOperationException: This connector does not provide memory accounting capabilities for ConnectorSplit

at io.trino.spi.connector.ConnectorSplit.getRetainedSizeInBytes(ConnectorSplit.java:43)
at io.trino.metadata.Split.getRetainedSizeInBytes(Split.java:91)
at io.airlift.slice.SizeOf.estimatedSizeOf(SizeOf.java:194)
at io.trino.execution.scheduler.TaskDescriptor.lambda$getRetainedSizeInBytes$0(TaskDescriptor.java:99)
at io.airlift.slice.SizeOf.estimatedSizeOf(SizeOf.java:222)
at io.trino.execution.scheduler.TaskDescriptor.getRetainedSizeInBytes(TaskDescriptor.java:99)
at io.trino.execution.scheduler.TaskDescriptorStorage$TaskDescriptors.put(TaskDescriptorStorage.java:187)
at io.trino.execution.scheduler.TaskDescriptorStorage.put(TaskDescriptorStorage.java:89)
at io.trino.execution.scheduler.EventDrivenFaultTolerantQueryScheduler$StagePartition.seal(EventDrivenFaultTolerantQueryScheduler.java:1635)
at io.trino.execution.scheduler.EventDrivenFaultTolerantQueryScheduler$StageExecution.sealPartition(EventDrivenFaultTolerantQueryScheduler.java:1244)
at io.trino.execution.scheduler.EventDrivenFaultTolerantQueryScheduler$Scheduler.lambda$onPartitionsSealed$12(EventDrivenFaultTolerantQueryScheduler.java:1050)
at com.google.common.primitives.ImmutableIntArray.forEach(ImmutableIntArray.java:414)
at io.trino.execution.scheduler.EventDrivenFaultTolerantQueryScheduler$Scheduler.onPartitionsSealed(EventDrivenFaultTolerantQueryScheduler.java:1049)
at io.trino.execution.scheduler.EventDrivenFaultTolerantQueryScheduler$PartitionsSealedEvent.accept(EventDrivenFaultTolerantQueryScheduler.java:2066)
at io.trino.execution.scheduler.EventDrivenFaultTolerantQueryScheduler$Scheduler.processEvents(EventDrivenFaultTolerantQueryScheduler.java:624)
at io.trino.execution.scheduler.EventDrivenFaultTolerantQueryScheduler$Scheduler.run(EventDrivenFaultTolerantQueryScheduler.java:555)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)

[codope]

@caneGuy @ebyhr Can you please edit the description of this issue and add the following TODOs?
#17976
#17977