-
Notifications
You must be signed in to change notification settings - Fork 142
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FEATURE] support shuffle partition data write to multiple shuffle servers #1373
Comments
+1 for that. I hope the trigger strategies could be designed as the pluggable mechansim |
How would this differ from |
This may only write one replica but the replica can be stored in different servers. |
You would better give us a google document like https://docs.google.com/document/d/1p1PksBN2LJ-OtGEHvdyEuH9b1Mv1aD_exMPl4TNaTs0/edit#heading=h.c7fauf9cicbi |
OK. added a google doc https://docs.google.com/document/d/1FRGP82R7JBnr3IjmF8PQ6qWP9GnKQsk-N1OWk_f1kSY |
LGTM. Please go ahead. |
…o hold split partition server
…ing from reassignment mechanism (#1445) ### What changes were proposed in this pull request? partition write to multi servers leveraging from reassignment mechanism ### Why are the changes needed? For: #1373 ### Does this PR introduce _any_ user-facing change? 1、add config `rss.server.dynamic.assign.enabled` for whether to reassign a faulty shuffle server. 2、support reassign a new shuffle server for send failed blocks 3、ShuffleReader read partition in muitl server implement will in next pr ### How was this patch tested? UTs --------- Co-authored-by: jam.xu <jam.xu@vipshop.com>
…d from multiple shuffle servers
### What changes were proposed in this pull request? fix partition id type is incorrect ### Why are the changes needed? Fix: (#1373) ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Not necessary.
…n partition data reassign is enabled (#1583) ### What changes were proposed in this pull request? Fix task fail retry parameter not work and add task fail retry parameter in other locations ### Why are the changes needed? Fix: (#1373) ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Not necessary. --------- Co-authored-by: shun01.ding <shun01.ding@vipshop.com>
… range error when reassign faulty shuffle server for tasks
… range error when reassign faulty shuffle server for tasks
… after reassign (#1612) ### What changes were proposed in this pull request? fix partition id inconsistency when reassign new shuffle server For example: when writing data on node a1, the registered partition id is 1003. a1 node fails,and reassign node b1 and register shuffle server b1,but partitionNumPerRange is 1. when writing data to node b1, NO_REGISTER exception will be thrown ### Why are the changes needed? Fix: (#1373) ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? --------- Co-authored-by: shun01.ding <shun01.ding@vipshop.com>
Code of Conduct
Search before asking
Describe the feature
Currently, data from one partition can only be written to a designated shuffle server.
In some scenarios, such as when a large partition occupies a significant amount of shuffle server disk space, this can lead to server crashes in extreme cases.
Based on #825, we could support writing one partition data to multiple shuffle servers.
Motivation
https://docs.google.com/document/d/1FRGP82R7JBnr3IjmF8PQ6qWP9GnKQsk-N1OWk_f1kSY
Describe the solution
Additional context
No response
Are you willing to submit PR?
The text was updated successfully, but these errors were encountered: