Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Improvement] Implement the interface ShuffleDataIO #802

Closed
2 of 3 tasks
jerqi opened this issue Apr 8, 2023 · 7 comments
Closed
2 of 3 tasks

[Improvement] Implement the interface ShuffleDataIO #802

jerqi opened this issue Apr 8, 2023 · 7 comments
Assignees
@summaryzb

Comments

@jerqi
Copy link
Contributor

jerqi commented Apr 8, 2023

Code of Conduct

Search before asking

  • I have searched in the issues and found no similar issues.

What would you like to be improved?

We can implement ShuffleDataIO
https://github.com/apache/spark/blob/master/core/src/main/java/org/apache/spark/shuffle/api/ShuffleDataIO.java
https://github.com/apache/spark/blob/master/core/src/main/java/org/apache/spark/shuffle/api/ShuffleDriverComponents.java
Dynamic Allocation don't need patch any more.

How should we improve?

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!
@beryllw
Copy link
Contributor

beryllw commented Apr 10, 2023

I'm Interested in this, could you assign it to me？

@jerqi
Copy link
Contributor Author

jerqi commented Apr 10, 2023

I'm Interested in this, could you assign it to me？

OK, I have assigned to you.

@beryllw
Copy link
Contributor

beryllw commented Apr 10, 2023

The patch [SPARK-42689] that allows ShuffleDriverComponent to declare if shuffle data is reliably stored has been fixed in Spark 3.5, which has not yet been released. Do I need to test it?

@jerqi
Copy link
Contributor Author

jerqi commented Apr 10, 2023

Yes, you can build a spark client with master branch to test it.

@beryllw
Copy link
Contributor

beryllw commented Apr 10, 2023

Yes, you can build a spark client with master branch to test it.

Okay, I will do that.

@summaryzb
Copy link
Contributor

volunteering

@jerqi
Copy link
Contributor Author

jerqi commented Aug 1, 2023

@Kwafoor If you're busy recently, let @summaryzb finish this issue first.

@summaryzb summaryzb mentioned this issue Oct 8, 2023
Merged
jerqi pushed a commit that referenced this issue Oct 9, 2023
@summaryzb
[#802] feat(spark): Implement ShuffleDataIo (#1226)
98b0d12 
### What changes were proposed in this pull request?
Implement ShuffleDataIo

### Why are the changes needed?
#802

### Does this PR introduce _any_ user-facing change?
Yes. To use spark dynamicAllocation, user should set `spark.shuffle.sort.io.plugin.class` to `org.apache.spark.shuffle.RssShuffleDataIo`, otherwise spark3.5 will fail

### How was this patch tested?
Integration test
Manual test
@jerqi jerqi closed this as completed Oct 9, 2023
@beryllw beryllw removed their assignment Oct 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@summaryzb summaryzb

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jerqi @beryllw @summaryzb