Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -50,4 +50,4 @@ All this said, there are still many ways we can improve upon this foundation.
* While optimistic concurrency control is attractive when serializable snapshot isolation is desired, it's neither optimal nor the only method for dealing with concurrency between writers. We plan to implement a fully lock-free concurrency control using CRDTs and widely adopted stream processing concepts, over our log [merge API](https://github.com/apache/hudi/blob/bc8bf043d5512f7afbb9d94882c4e43ee61d6f06/hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordPayload.java#L38), that has already been [proven](https://hudi.apache.org/blog/2021/09/01/building-eb-level-data-lake-using-hudi-at-bytedance/#functionality-support) to sustain enormous continuous write volumes for the data lake.
* Touching upon key constraints, Hudi is the only lake transactional layer that ensures unique [key](https://hudi.apache.org/docs/key_generation) constraints today, but limited to the record key of the table. We will be looking to expand this capability in a more general form to non-primary key fields, with the said newer concurrency models.

Finally, for data lakes to transform successfully into lakehouses, we must learn from the failing of the "hadoop warehouse" vision, which shared similar goals with the new "lakehouse" vision. Designers did not pay closer attention to the missing technology gaps against warehouses and created unrealistic expectations from the actual software. As transactions and database functionality finally goes mainstream on data lakes, we must apply these lessons and remain candid about the current shortcomings. If you are building a lakehouse, I hope this post encourages you to closely consider various operational and efficiency aspects around concurrency control. Join our fast growing community by trying out [Apache Hudi](https://hudi.apache.org/docs/overview) or join us in conversations on [Slack](https://join.slack.com/t/apache-hudi/shared_invite/enQtODYyNDAxNzc5MTg2LTE5OTBlYmVhYjM0N2ZhOTJjOWM4YzBmMWU2MjZjMGE4NDc5ZDFiOGQ2N2VkYTVkNzU3ZDQ4OTI1NmFmYWQ0NzE).
Finally, for data lakes to transform successfully into lakehouses, we must learn from the failing of the "hadoop warehouse" vision, which shared similar goals with the new "lakehouse" vision. Designers did not pay closer attention to the missing technology gaps against warehouses and created unrealistic expectations from the actual software. As transactions and database functionality finally goes mainstream on data lakes, we must apply these lessons and remain candid about the current shortcomings. If you are building a lakehouse, I hope this post encourages you to closely consider various operational and efficiency aspects around concurrency control. Join our fast growing community by trying out [Apache Hudi](https://hudi.apache.org/docs/overview) or join us in conversations on [Slack](https://join.slack.com/t/apache-hudi/shared_invite/zt-1c44wsfgl-gX_6f50DezdnWVrGsxC~Ug).
4 changes: 2 additions & 2 deletions website/blog/2022-01-06-apache-hudi-2021-a-year-in-review.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ As the year came to end, I took some time to reflect on where we are and what we

**_Community_**

I want to call out how amazing it is to see such a diverse group of people step up and contribute to this project. There were over 30,000 interactions with the [project on github](https://github.com/apache/hudi/), up 2x from last year. Over the last year 300 people have contributed to the project, with over 3,000 PRs over 5 releases. We moved Apache Hudi from release 0.5.X all the way to our feature packed 0.10.0 release. Come and join us on our [active slack channel](https://join.slack.com/t/apache-hudi/shared_invite/enQtODYyNDAxNzc5MTg2LTE5OTBlYmVhYjM0N2ZhOTJjOWM4YzBmMWU2MjZjMGE4NDc5ZDFiOGQ2N2VkYTVkNzU3ZDQ4OTI1NmFmYWQ0NzE)! Over 850 community members engaged on our slack, up about 100% from the year before. I want to add a special shout out to our top slack participants who have helped answer so many questions and drive rich discussions on our channel. Sivabalan Narayanan, Nishith Agarwal, Bhavani Sudha Saktheeswaran, Vinay Patil, Rubens Soto, Dave Hagman, Raghav Tandon, Sagar Sumit, Joyan Sil, Jake D, Felix Jose, Nick Vintila, KimL, Andrew Sukhan, Danny Chan, Biswajit Mohapatra, and Pratyaksh Sharma! I know I am missing plenty of other important callouts, every PR that landed this year has helped shape Hudi into what it is today. Thank you!
I want to call out how amazing it is to see such a diverse group of people step up and contribute to this project. There were over 30,000 interactions with the [project on github](https://github.com/apache/hudi/), up 2x from last year. Over the last year 300 people have contributed to the project, with over 3,000 PRs over 5 releases. We moved Apache Hudi from release 0.5.X all the way to our feature packed 0.10.0 release. Come and join us on our [active slack channel](https://join.slack.com/t/apache-hudi/shared_invite/zt-1c44wsfgl-gX_6f50DezdnWVrGsxC~Ug)! Over 850 community members engaged on our slack, up about 100% from the year before. I want to add a special shout out to our top slack participants who have helped answer so many questions and drive rich discussions on our channel. Sivabalan Narayanan, Nishith Agarwal, Bhavani Sudha Saktheeswaran, Vinay Patil, Rubens Soto, Dave Hagman, Raghav Tandon, Sagar Sumit, Joyan Sil, Jake D, Felix Jose, Nick Vintila, KimL, Andrew Sukhan, Danny Chan, Biswajit Mohapatra, and Pratyaksh Sharma! I know I am missing plenty of other important callouts, every PR that landed this year has helped shape Hudi into what it is today. Thank you!

<img src="/assets/images/powers/logo-wall.png" alt="drawing" width="600"/>

Expand All @@ -29,4 +29,4 @@ Apache Hudi has come a long way in 2021 from v0.5.X to 0.10.0. Throughout this y

2021 may have been our best year so far, but it still feels like we are just getting started when we look at our new year's resolutions for 2022. In the year ahead we have bold plans to realize the first cut of our entire vision and take Hudi 1.0, that includes full-featured multi-modal indexing for faster writes/queries, pathbreaking lock free concurrency, new server components for caching/metadata and finally Flink based incremental materialized views!  _You can find our_ [_detailed roadmap here_](https://hudi.apache.org/roadmap)_._

I look forward to continued collaboration with the growing Hudi community! Come join our [_community events_](https://hudi.apache.org/community/syncs) _and discussions in our_ [_slack channel_](https://join.slack.com/t/apache-hudi/shared_invite/enQtODYyNDAxNzc5MTg2LTE5OTBlYmVhYjM0N2ZhOTJjOWM4YzBmMWU2MjZjMGE4NDc5ZDFiOGQ2N2VkYTVkNzU3ZDQ4OTI1NmFmYWQ0NzE)_! Happy new year 2022!_
I look forward to continued collaboration with the growing Hudi community! Come join our [_community events_](https://hudi.apache.org/community/syncs) _and discussions in our_ [_slack channel_](https://join.slack.com/t/apache-hudi/shared_invite/zt-1c44wsfgl-gX_6f50DezdnWVrGsxC~Ug)_! Happy new year 2022!_
Original file line number Diff line number Diff line change
Expand Up @@ -185,4 +185,4 @@ spark-submit \\

This post introduced the Debezium Source for Hudi Deltastreamer to ingest Debezium changelogs into Hudi tables. Database data can now be ingested into data lakes to provide a cost-effective way to store and analyze database data.

Please follow this [JIRA](https://issues.apache.org/jira/browse/HUDI-1290) to learn more about active development on this new feature. I look forward to more contributions and feedback from the community. Come join our [Hudi Slack](https://join.slack.com/t/apache-hudi/shared_invite/enQtODYyNDAxNzc5MTg2LTE5OTBlYmVhYjM0N2ZhOTJjOWM4YzBmMWU2MjZjMGE4NDc5ZDFiOGQ2N2VkYTVkNzU3ZDQ4OTI1NmFmYWQ0NzE) channel or attend one of our [community events](https://hudi.apache.org/community/syncs) to learn more.
Please follow this [JIRA](https://issues.apache.org/jira/browse/HUDI-1290) to learn more about active development on this new feature. I look forward to more contributions and feedback from the community. Come join our [Hudi Slack](https://join.slack.com/t/apache-hudi/shared_invite/zt-1c44wsfgl-gX_6f50DezdnWVrGsxC~Ug) channel or attend one of our [community events](https://hudi.apache.org/community/syncs) to learn more.
2 changes: 1 addition & 1 deletion website/community/get-involved.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ There are several ways to get in touch with the Hudi community.
| For development discussions | Dev Mailing list ([Subscribe](mailto:dev-subscribe@hudi.apache.org), [Unsubscribe](mailto:dev-unsubscribe@hudi.apache.org), [Archives](https://lists.apache.org/list?dev@hudi.apache.org)). Empty email works for subscribe/unsubscribe. Please use [gists](https://gist.github.com) to share code/stacktraces on the email. |
| For any general questions, user support | Users Mailing list ([Subscribe](mailto:users-subscribe@hudi.apache.org), [Unsubscribe](mailto:users-unsubscribe@hudi.apache.org), [Archives](https://lists.apache.org/list?users@hudi.apache.org)). Empty email works for subscribe/unsubscribe. Please use [gists](https://gist.github.com) to share code/stacktraces on the email. |
| For reporting bugs or issues or discover known issues | Please use [ASF Hudi JIRA](https://issues.apache.org/jira/projects/HUDI/summary). See [#here](#accounts) for access |
| For quick pings & 1-1 chats | Join our [slack group](https://join.slack.com/t/apache-hudi/shared_invite/enQtODYyNDAxNzc5MTg2LTE5OTBlYmVhYjM0N2ZhOTJjOWM4YzBmMWU2MjZjMGE4NDc5ZDFiOGQ2N2VkYTVkNzU3ZDQ4OTI1NmFmYWQ0NzE). In case this does not work, please leave a comment on this [github issue](https://github.com/apache/hudi/issues/143) |
| For quick pings & 1-1 chats | Join our [slack group](https://join.slack.com/t/apache-hudi/shared_invite/zt-1c44wsfgl-gX_6f50DezdnWVrGsxC~Ug). In case this does not work, please leave a comment on this [github issue](https://github.com/apache/hudi/issues/143) |
| For proposing large features, changes | Start a RFC. Instructions [here](/contribute/rfc-process).
| Join weekly sync-up meeting | Follow instructions [here](https://cwiki.apache.org/confluence/display/HUDI/Apache+Hudi+Community+Weekly+Sync). |
See [#here](#accounts) for wiki access |
Expand Down
4 changes: 2 additions & 2 deletions website/docs/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ resources to learn more, engage, and get help as you get started.

### Join in on discussions
See all the ways to [engage with the community here](/community/get-involved). Two most popular methods include:
- [Hudi Slack Channel](https://join.slack.com/t/apache-hudi/shared_invite/enQtODYyNDAxNzc5MTg2LTE5OTBlYmVhYjM0N2ZhOTJjOWM4YzBmMWU2MjZjMGE4NDc5ZDFiOGQ2N2VkYTVkNzU3ZDQ4OTI1NmFmYWQ0NzE)
- [Hudi Slack Channel](https://join.slack.com/t/apache-hudi/shared_invite/zt-1c44wsfgl-gX_6f50DezdnWVrGsxC~Ug)
- [Hudi mailing list](mailto:users-subscribe@hudi.apache.org) - (send any msg to subscribe)

### Come to Office Hours for help
Expand All @@ -65,5 +65,5 @@ Apache Hudi welcomes you to join in on the fun and make a lasting impact on the
[contributor guide](/contribute/how-to-contribute) to learn more, and don’t hesitate to directly reach out to any of the
current committers to learn more.

Have an idea, an ask, or feedback about a pain-point, but don’t have time to contribute? Join the [Hudi Slack Channel](https://join.slack.com/t/apache-hudi/shared_invite/enQtODYyNDAxNzc5MTg2LTE5OTBlYmVhYjM0N2ZhOTJjOWM4YzBmMWU2MjZjMGE4NDc5ZDFiOGQ2N2VkYTVkNzU3ZDQ4OTI1NmFmYWQ0NzE)
Have an idea, an ask, or feedback about a pain-point, but don’t have time to contribute? Join the [Hudi Slack Channel](https://join.slack.com/t/apache-hudi/shared_invite/zt-1c44wsfgl-gX_6f50DezdnWVrGsxC~Ug)
and share!
4 changes: 2 additions & 2 deletions website/docusaurus.config.js
Original file line number Diff line number Diff line change
Expand Up @@ -225,7 +225,7 @@ module.exports = {
'aria-label': 'Hudi Twitter Handle',
},
{
href: 'https://join.slack.com/t/apache-hudi/shared_invite/enQtODYyNDAxNzc5MTg2LTE5OTBlYmVhYjM0N2ZhOTJjOWM4YzBmMWU2MjZjMGE4NDc5ZDFiOGQ2N2VkYTVkNzU3ZDQ4OTI1NmFmYWQ0NzE',
href: 'https://join.slack.com/t/apache-hudi/shared_invite/zt-1c44wsfgl-gX_6f50DezdnWVrGsxC~Ug',
position: 'right',
className: 'header-slack-link',
'aria-label': 'Hudi Slack Channel',
Expand Down Expand Up @@ -331,7 +331,7 @@ module.exports = {
},
{
label: 'Slack',
href: 'https://join.slack.com/t/apache-hudi/shared_invite/enQtODYyNDAxNzc5MTg2LTE5OTBlYmVhYjM0N2ZhOTJjOWM4YzBmMWU2MjZjMGE4NDc5ZDFiOGQ2N2VkYTVkNzU3ZDQ4OTI1NmFmYWQ0NzE',
href: 'https://join.slack.com/t/apache-hudi/shared_invite/zt-1c44wsfgl-gX_6f50DezdnWVrGsxC~Ug',
},
{
label: 'GitHub',
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ There are several ways to get in touch with the Hudi community.
| For development discussions | Dev Mailing list ([Subscribe](mailto:dev-subscribe@hudi.apache.org), [Unsubscribe](mailto:dev-unsubscribe@hudi.apache.org), [Archives](https://lists.apache.org/list?dev@hudi.apache.org)). Empty email works for subscribe/unsubscribe. Please use [gists](https://gist.github.com) to share code/stacktraces on the email. |
| For any general questions, user support | Users Mailing list ([Subscribe](mailto:users-subscribe@hudi.apache.org), [Unsubscribe](mailto:users-unsubscribe@hudi.apache.org), [Archives](https://lists.apache.org/list?users@hudi.apache.org)). Empty email works for subscribe/unsubscribe. Please use [gists](https://gist.github.com) to share code/stacktraces on the email. |
| For reporting bugs or issues or discover known issues | Please use [ASF Hudi JIRA](https://issues.apache.org/jira/projects/HUDI/summary). See [#here](#accounts) for access |
| For quick pings & 1-1 chats | Join our [slack group](https://join.slack.com/t/apache-hudi/shared_invite/enQtODYyNDAxNzc5MTg2LTE5OTBlYmVhYjM0N2ZhOTJjOWM4YzBmMWU2MjZjMGE4NDc5ZDFiOGQ2N2VkYTVkNzU3ZDQ4OTI1NmFmYWQ0NzE). In case this does not work, please leave a comment on this [github issue](https://github.com/apache/hudi/issues/143) |
| For quick pings & 1-1 chats | Join our [slack group](https://join.slack.com/t/apache-hudi/shared_invite/zt-1c44wsfgl-gX_6f50DezdnWVrGsxC~Ug). In case this does not work, please leave a comment on this [github issue](https://github.com/apache/hudi/issues/143) |
| For proposing large features, changes | Start a RFC. Instructions [here](https://cwiki.apache.org/confluence/display/HUDI/RFC+Process).
| Join weekly sync-up meeting | Follow instructions [here](https://cwiki.apache.org/confluence/display/HUDI/Apache+Hudi+Community+Weekly+Sync). |
See [#here](#accounts) for wiki access |
Expand Down
2 changes: 1 addition & 1 deletion website/i18n/cn/docusaurus-theme-classic/footer.json
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@
},
"link.item.label.Slack": {
"message": "Slack",
"description": "The label of footer link with label=Slack linking to https://join.slack.com/t/apache-hudi/shared_invite/enQtODYyNDAxNzc5MTg2LTE5OTBlYmVhYjM0N2ZhOTJjOWM4YzBmMWU2MjZjMGE4NDc5ZDFiOGQ2N2VkYTVkNzU3ZDQ4OTI1NmFmYWQ0NzE"
"description": "The label of footer link with label=Slack linking to https://join.slack.com/t/apache-hudi/shared_invite/zt-1c44wsfgl-gX_6f50DezdnWVrGsxC~Ug"
},
"link.item.label.GitHub": {
"message": "GitHub",
Expand Down
2 changes: 1 addition & 1 deletion website/sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,7 @@ module.exports = {
{
type: 'link',
label: 'Chat with us on Slack',
href: 'https://join.slack.com/t/apache-hudi/shared_invite/enQtODYyNDAxNzc5MTg2LTE5OTBlYmVhYjM0N2ZhOTJjOWM4YzBmMWU2MjZjMGE4NDc5ZDFiOGQ2N2VkYTVkNzU3ZDQ4OTI1NmFmYWQ0NzE',
href: 'https://join.slack.com/t/apache-hudi/shared_invite/zt-1c44wsfgl-gX_6f50DezdnWVrGsxC~Ug',
},
],
};
2 changes: 1 addition & 1 deletion website/sidebarsCommunity.js
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ module.exports = {
{
type: 'link',
label: 'Join Our Slack',
href: 'https://join.slack.com/t/apache-hudi/shared_invite/enQtODYyNDAxNzc5MTg2LTE5OTBlYmVhYjM0N2ZhOTJjOWM4YzBmMWU2MjZjMGE4NDc5ZDFiOGQ2N2VkYTVkNzU3ZDQ4OTI1NmFmYWQ0NzE',
href: 'https://join.slack.com/t/apache-hudi/shared_invite/zt-1c44wsfgl-gX_6f50DezdnWVrGsxC~Ug',
},
],
};
2 changes: 1 addition & 1 deletion website/sidebarsContribute.js
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ module.exports = {
{
type: 'link',
label: 'Join our Slack',
href: 'https://join.slack.com/t/apache-hudi/shared_invite/enQtODYyNDAxNzc5MTg2LTE5OTBlYmVhYjM0N2ZhOTJjOWM4YzBmMWU2MjZjMGE4NDc5ZDFiOGQ2N2VkYTVkNzU3ZDQ4OTI1NmFmYWQ0NzE',
href: 'https://join.slack.com/t/apache-hudi/shared_invite/zt-1c44wsfgl-gX_6f50DezdnWVrGsxC~Ug',
},
],
};
2 changes: 1 addition & 1 deletion website/src/pages/powered-by.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ last_modified_at: 2019-12-31T15:59:57-04:00

Apache Hudi is a [fast growing diverse community](https://hudi.apache.org/blog/2022/01/06/apache-hudi-2021-a-year-in-review)
of people and organizations from all around the globe. The following is a small sample of companies that have adopted or
contributed to the Apache Hudi community! [Join us on slack](https://join.slack.com/t/apache-hudi/shared_invite/enQtODYyNDAxNzc5MTg2LTE5OTBlYmVhYjM0N2ZhOTJjOWM4YzBmMWU2MjZjMGE4NDc5ZDFiOGQ2N2VkYTVkNzU3ZDQ4OTI1NmFmYWQ0NzE),
contributed to the Apache Hudi community! [Join us on slack](https://join.slack.com/t/apache-hudi/shared_invite/zt-1c44wsfgl-gX_6f50DezdnWVrGsxC~Ug),
or come to one of our [virtual community events](https://hudi.apache.org/community/syncs).

<img src="/assets/images/powers/logo-wall.png" alt="drawing"/>
Expand Down
4 changes: 2 additions & 2 deletions website/versioned_docs/version-0.10.0/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ resources to learn more, engage, and get help as you get started.

### Join in on discussions
See all the ways to [engage with the community here](/community/get-involved). Two most popular methods include:
- [Hudi Slack Channel](https://join.slack.com/t/apache-hudi/shared_invite/enQtODYyNDAxNzc5MTg2LTE5OTBlYmVhYjM0N2ZhOTJjOWM4YzBmMWU2MjZjMGE4NDc5ZDFiOGQ2N2VkYTVkNzU3ZDQ4OTI1NmFmYWQ0NzE)
- [Hudi Slack Channel](https://join.slack.com/t/apache-hudi/shared_invite/zt-1c44wsfgl-gX_6f50DezdnWVrGsxC~Ug)
- [Hudi mailing list](mailto:users-subscribe@hudi.apache.org) - (send any msg to subscribe)

### Come to Office Hours for help
Expand All @@ -65,5 +65,5 @@ Apache Hudi welcomes you to join in on the fun and make a lasting impact on the
[contributor guide](/contribute/how-to-contribute) to learn more, and don’t hesitate to directly reach out to any of the
current committers to learn more.

Have an idea, an ask, or feedback about a pain-point, but don’t have time to contribute? Join the [Hudi Slack Channel](https://join.slack.com/t/apache-hudi/shared_invite/enQtODYyNDAxNzc5MTg2LTE5OTBlYmVhYjM0N2ZhOTJjOWM4YzBmMWU2MjZjMGE4NDc5ZDFiOGQ2N2VkYTVkNzU3ZDQ4OTI1NmFmYWQ0NzE)
Have an idea, an ask, or feedback about a pain-point, but don’t have time to contribute? Join the [Hudi Slack Channel](https://join.slack.com/t/apache-hudi/shared_invite/zt-1c44wsfgl-gX_6f50DezdnWVrGsxC~Ug)
and share!
4 changes: 2 additions & 2 deletions website/versioned_docs/version-0.10.1/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ resources to learn more, engage, and get help as you get started.

### Join in on discussions
See all the ways to [engage with the community here](/community/get-involved). Two most popular methods include:
- [Hudi Slack Channel](https://join.slack.com/t/apache-hudi/shared_invite/enQtODYyNDAxNzc5MTg2LTE5OTBlYmVhYjM0N2ZhOTJjOWM4YzBmMWU2MjZjMGE4NDc5ZDFiOGQ2N2VkYTVkNzU3ZDQ4OTI1NmFmYWQ0NzE)
- [Hudi Slack Channel](https://join.slack.com/t/apache-hudi/shared_invite/zt-1c44wsfgl-gX_6f50DezdnWVrGsxC~Ug)
- [Hudi mailing list](mailto:users-subscribe@hudi.apache.org) - (send any msg to subscribe)

### Come to Office Hours for help
Expand All @@ -65,5 +65,5 @@ Apache Hudi welcomes you to join in on the fun and make a lasting impact on the
[contributor guide](/contribute/how-to-contribute) to learn more, and don’t hesitate to directly reach out to any of the
current committers to learn more.

Have an idea, an ask, or feedback about a pain-point, but don’t have time to contribute? Join the [Hudi Slack Channel](https://join.slack.com/t/apache-hudi/shared_invite/enQtODYyNDAxNzc5MTg2LTE5OTBlYmVhYjM0N2ZhOTJjOWM4YzBmMWU2MjZjMGE4NDc5ZDFiOGQ2N2VkYTVkNzU3ZDQ4OTI1NmFmYWQ0NzE)
Have an idea, an ask, or feedback about a pain-point, but don’t have time to contribute? Join the [Hudi Slack Channel](https://join.slack.com/t/apache-hudi/shared_invite/zt-1c44wsfgl-gX_6f50DezdnWVrGsxC~Ug)
and share!
Loading