Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feat][Spark] Improve the EdgeWriter's efficiency with parallel process #328

Closed
acezen opened this issue Jan 17, 2024 · 0 comments · Fixed by #329
Closed

[Feat][Spark] Improve the EdgeWriter's efficiency with parallel process #328

acezen opened this issue Jan 17, 2024 · 0 comments · Fixed by #329
Assignees
@acezen
Labels
enhancement New feature or request

Comments

@acezen
Copy link
Contributor

acezen commented Jan 17, 2024 •
edited

Is your feature request related to a problem? Please describe.
Currently the EdgeWriter generate and output the offset chunks by sequence way1 2, we can improve the efficiency by converting to parallel process

Describe the solution you'd like
use scala's parallel process like

Array.par.foreach {}

Describe alternatives you've considered
NA
@acezen acezen added the enhancement New feature or request label Jan 17, 2024
@acezen acezen changed the title [Feat][Spark] Improve the writer effeciency with parallel process [Feat][Spark] Improve the EdgeWriter's efficiency with parallel process Jan 18, 2024
@acezen acezen self-assigned this Jan 18, 2024
This was referenced Jan 18, 2024
Merged
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@acezen acezen

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Improvement][Spark] Improve the writer effeciency with parallel process acezen/GraphAr
1 participant
@acezen