Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Support integrationing into gluten #940

Closed
2 of 3 tasks
xianjingfeng opened this issue Jun 9, 2023 · 13 comments · Fixed by #950
Closed
2 of 3 tasks

[FEATURE] Support integrationing into gluten #940

xianjingfeng opened this issue Jun 9, 2023 · 13 comments · Fixed by #950

Comments

@xianjingfeng
Copy link
Member

Code of Conduct

Search before asking

  • I have searched in the issues and found no similar issues.

Describe the feature

gluten is a spark native engine. When we integrate uniffle into gluten, we need to adjust some codes of uniffle.
In addition, we also need to publish the jar package to the public maven warehouse, like celeborn done.
https://github.com/oap-project/gluten/tree/main/gluten-celeborn

Motivation

No response

Describe the solution

No response

Additional context

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!
@xianjingfeng
Copy link
Member Author

@kaijchen Can you help publish the jar package to the public maven repository.
And my colleagues will help complete the remaining work of this feature.

@xianjingfeng
Copy link
Member Author

cc @jerqi @advancedxy @zuston

@jerqi
Copy link
Contributor

jerqi commented Jun 9, 2023

@kaijchen Can you help publish the jar package to the public maven repository. And my colleagues will help complete the remaining work of this feature.

If we want to publish the jar package, we need to finish binary release. And we only finish source releasenow. We can discuss this issue on Uniffle regular meeting in the next week.

@kaijchen
Copy link
Contributor

kaijchen commented Jun 9, 2023

@kaijchen Can you help publish the jar package to the public maven repository.

Doing so might require releasing process, let's do it in release 0.7.1?
Or we can publish the artifacts to the SNAPSHOT repository.

@jerqi
Copy link
Contributor

jerqi commented Jun 9, 2023

cc @summaryzb

@xianjingfeng
Copy link
Member Author

xianjingfeng commented Jun 9, 2023

@kaijchen Can you help publish the jar package to the public maven repository.

Doing so might require releasing process, let's do it in release 0.7.1? Or we can publish the artifacts to the SNAPSHOT repository.

We can do it in next release version, after we complete this feature.

@summaryzb
Copy link
Contributor

I‘d like to involve this feature, shall you open a discussion on it

@zuston
Copy link
Member

zuston commented Jun 9, 2023

SGTM.

@summaryzb
Copy link
Contributor

we implement the integration in our production pipeline as below
image

@summaryzb
Copy link
Contributor

gluten side change
apache/incubator-gluten#1966

@jerqi
Copy link
Contributor

jerqi commented Jun 15, 2023

@summaryzb Could you write a simple document like https://docs.google.com/document/d/1G0cOFVJbYLf2oX1fiadh7zi2M6DlEcjTQTh4kSkb0LA/edit#heading=h.kl0vgpq0nxcy
? It's more easy if other people want to join the discussion or study the implement of this feature.

@summaryzb
Copy link
Contributor

Sure I'll post it here today later

xianjingfeng pushed a commit that referenced this issue Jun 16, 2023
### What changes were proposed in this pull request?
support read and write serialized columnar data

### Why are the changes needed?
Fix: #940

### Does this PR introduce any user-facing change?
No.

### How was this patch tested?
UnitTest, it's covered by the existing test
@summaryzb
Copy link
Contributor

summaryzb commented Jun 16, 2023

It's more easy if other people want to join the discussion or study the implement of this feature.

https://docs.google.com/document/d/1oi5QqrZejPRWBUIBv84eHbUhn2mTcgxP-s1ybh4zB1k/edit

xianjingfeng pushed a commit that referenced this issue Jun 28, 2023
### What changes were proposed in this pull request?
Make it possible to extend uniffle in spark3.
Optimize the shuffleMetric when use columnar shuffle
### Why are the changes needed?
#940

### Does this PR introduce any user-facing change?
No.

### How was this patch tested?
Unit test
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants