Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(service/hdfs): enable hdfs append support #3600

Merged
merged 5 commits into from
Nov 17, 2023
Merged

fix(service/hdfs): enable hdfs append support #3600

merged 5 commits into from
Nov 17, 2023

Conversation

qingwen220
Copy link
Contributor

@qingwen220 qingwen220 commented Nov 16, 2023 •
edited by Xuanwo

support append in block_write

close #2500

@qingwen220 qingwen220 force-pushed the hdfs_blocking_append branch 5 times, most recently from 5f76495 to 48eb645 Compare November 16, 2023 13:23
@qingwen220
Copy link
Contributor Author

qingwen220 commented Nov 16, 2023 •
edited

@Xuanwo After some tests, default and cluster tests fail with different reasons. Below are my guess.

  1. default single node may not support append.
  2. for the distributed cluster, we should add configuration 'dfs_replication=1' on the client side as well. That configuration on my own single-node cluster works. But I do not know how to config in the ut.

Looking forward to your input.

@Xuanwo Xuanwo changed the title bugfix(service/hdfs): enable hdfs blocking append fix(service/hdfs): enable hdfs blocking append Nov 16, 2023
Xuanwo
Xuanwo reviewed Nov 16, 2023
View reviewed changes
core/src/services/hdfs/backend.rs Outdated Show resolved Hide resolved
.github/workflows/service_test_hdfs.yml Show resolved Hide resolved
.github/workflows/service_test_hdfs.yml Show resolved Hide resolved
@qingwen220 qingwen220 force-pushed the hdfs_blocking_append branch 2 times, most recently from 7d3998b to 4398cca Compare November 17, 2023 06:18
@qingwen220 qingwen220 force-pushed the hdfs_blocking_append branch 2 times, most recently from 833c4bf to 016a58d Compare November 17, 2023 07:33
Xuanwo
Xuanwo reviewed Nov 17, 2023
View reviewed changes
Copy link
Member

@Xuanwo Xuanwo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other LGTM. We just need to make CI happy. Thanks for your contributions!

core/src/services/hdfs/backend.rs Outdated Show resolved Hide resolved
core/src/services/hdfs/backend.rs Outdated Show resolved Hide resolved
@Xuanwo Xuanwo changed the title fix(service/hdfs): enable hdfs blocking append fix(service/hdfs): enable hdfs append support Nov 17, 2023
Xuanwo
Xuanwo approved these changes Nov 17, 2023
View reviewed changes
Copy link
Member

@Xuanwo Xuanwo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@Xuanwo Xuanwo merged commit c9fcb80 into apache:main Nov 17, 2023
140 of 166 checks passed
@suyanhanx
Copy link
Member

Thanks!

@G-XD G-XD mentioned this pull request Nov 27, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@suyanhanx suyanhanx suyanhanx left review comments

@Xuanwo Xuanwo Xuanwo approved these changes

@PsiACE PsiACE Awaiting requested review from PsiACE PsiACE is a code owner

Assignees
No one assigned
Labels
releases-note/fix
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Support append for hdfs
3 participants
@qingwen220 @suyanhanx @Xuanwo