Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature]: Support integration with Apache Hudi #220

Open
Tracked by #2534
zyclove opened this issue Aug 24, 2022 · 7 comments
Open
Tracked by #2534

[Feature]: Support integration with Apache Hudi #220

zyclove opened this issue Aug 24, 2022 · 7 comments
Labels
type:feature Feature Requests

Comments

@zyclove
Copy link

zyclove commented Aug 24, 2022

We have been using hudi as a data lake.
Looking forward to supporting.

@zhoujinsong zhoujinsong added the type:feature Feature Requests label Aug 25, 2022
@zhoujinsong
Copy link
Contributor

Hi, @zyclove

Thanks for your feed back.
Supporting hudi is a very useful feature for arctic and we are planning put it into our roadmap.
But It has a lot work to do in order to achieve this goal.
And we are very pleased to welcome you to join the discussing and designing for this feature.

@melin
Copy link

melin commented Sep 7, 2022

Hi, @zyclove

Thanks for your feed back. Supporting hudi is a very useful feature for arctic and we are planning put it into our roadmap. But It has a lot work to do in order to achieve this goal. And we are very pleased to welcome you to join the discussing and designing for this feature.

目前比较难的是,hudi 没有想iceberg 保留catalog 扩展能力,社区还在讨论中,需要等很久

@zyclove
Copy link
Author

zyclove commented Mar 20, 2023

请问现在社区有进度吗?很希望可以列出方案和计划,一起共同搞起来。现在很多特性确实hudi支持的很不错,hudi线上使用公司也特别多,对Arctic这种元数据管理服务依赖也很强烈。能不能大佬们讨论讨论搞个计划呢?

Hudi vs Delta Lake vs Iceberg: https://www.onehouse.ai/blog/apache-hudi-vs-delta-lake-vs-apache-iceberg-lakehouse-feature-comparison

hudi很多特性我们一直在线上使用,很期待可以支持一下哦。
@zhoujinsong @melin @fantasyni @radiumce

@zhoujinsong zhoujinsong changed the title 【feat】hudi is need supported [Feature]: hudi is need supported Mar 20, 2023
@zhoujinsong
Copy link
Contributor

zhoujinsong commented Mar 23, 2023

@zyclove
Thanks a lot for bringing this feature up again!
I must admit that right now the Arctic community has no clear plan for Hudi's integration.
However, I think we can start discussing what value the Arctic can bring up to Hudi users after integration so that we can develop a more detailed integration plan later.

As far as I can see Arctic can bring the following values to Hudi users after integration:

  • Centered optimizing task scheduling for Hudi tables to improve resource usage and stability of table optimizing tasks(compaction、clustering、cleaning)
  • A web-based dashboard to show table information and metrics.

However, I would like to get more input from Hudi users about this question, so I would also like to hear your opinion.

@zhoujinsong zhoujinsong mentioned this issue Apr 17, 2023
4 tasks
@zhoujinsong zhoujinsong changed the title [Feature]: hudi is need supported [Feature]: Support integration with Apache Hudi Apr 17, 2023
@zyclove
Copy link
Author

zyclove commented Nov 10, 2023

目前hudi社区也已经有元数据管理服务,也提供接口,现在是不是对接管理开发也更容易了,能不能加快一下排期呢?

@shidayang
Copy link
Contributor

目前hudi社区也已经有元数据管理服务,也提供接口,现在是不是对接管理开发也更容易了,能不能加快一下排期呢?

We are very interested in integrating Hudi. Are you interested in driving this feature?

@baiyangtx
Copy link
Contributor

目前hudi社区也已经有元数据管理服务,也提供接口,现在是不是对接管理开发也更容易了,能不能加快一下排期呢?

As far as I know, Hudi has its own Compaction service. What additional capabilities do you expect Amoro to provide for Hudi?

Do you want visualized Compaction management?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type:feature Feature Requests
Projects
None yet
Development

No branches or pull requests

5 participants