Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Support DFS as the shuffle data storage #74

Closed
Gustfh opened this issue Mar 15, 2022 · 4 comments
Closed

[FEATURE] Support DFS as the shuffle data storage #74

Gustfh opened this issue Mar 15, 2022 · 4 comments

Comments

@Gustfh
Copy link

Gustfh commented Mar 15, 2022

Add support DFS for the shuffle data storage, such as HDFS, Alluxio. so rss will not care about the disk healthy, the replication. and also the DFS could take the pressure of shuffle read.

@FMX
Copy link
Contributor

FMX commented Mar 17, 2022

@Gustfh Hi, thanks for your advice.
We had supported HDFS before and found that performance on HDFS is unacceptable.
We are working on supporting AliLFS now.

@Gustfh
Copy link
Author

Gustfh commented Mar 17, 2022

i think we should open an abstract for dfs storage or compose storage, so other people could implement different storage as their requirement.

@FMX
Copy link
Contributor

FMX commented Mar 18, 2022

OK, I think you are right. Let me propose a draft.

@FMX
Copy link
Contributor

FMX commented Nov 1, 2022

Check #146 and I'll close this pr.

@FMX FMX closed this as completed Nov 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants