Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimize performance of arrow fragment to dynamic fragment with multi-threading #1458

Merged
merged 34 commits into from
Apr 24, 2022
Merged

Optimize performance of arrow fragment to dynamic fragment with multi-threading #1458

merged 34 commits into from
Apr 24, 2022

Conversation

acezen
Copy link
Collaborator

@acezen acezen commented Apr 14, 2022 •
edited
Loading

What do these changes do?

  • Parallel support on arrow fragment to dynamic fragment process.
  • Revise dynamic::Value implementation to avoid unnecessary copy.

Related issue number

Related #1300
Related alibaba/libgrape-lite#105

@codecov-commenter
Copy link

codecov-commenter commented Apr 14, 2022 •
edited
Loading

Codecov Report

Merging #1458 (74df3f0) into main (daa8649) will not change coverage.
The diff coverage is 100.00%.

Impacted file tree graph

@@           Coverage Diff           @@
##             main    #1458   +/-   ##
=======================================
  Coverage   73.60%   73.60%           
=======================================
  Files          88       88           
  Lines        9719     9719           
=======================================
  Hits         7154     7154           
  Misses       2565     2565
Impacted Files Coverage Δ
python/graphscope/framework/dag_utils.py 63.49% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update daa8649...74df3f0. Read the comment docs.

@acezen acezen force-pushed the acezen/optimize_convert branch 2 times, most recently from 1d550a5 to b2aaeca Compare April 19, 2022 02:56
@acezen
Copy link
Collaborator Author

acezen commented Apr 20, 2022 •
edited
Loading

Performance comparison

Graph no-optimization 12 threads
datagen-7_5_fb (0.6M, 34M) 172.218s 25.742s
datagen-7_8_zf (16M, 41M) > 250s (OOM) 37.307s

@acezen acezen changed the title [WIP] Optimize performance of arrow fragment to dynamic fragment Optimize performance of arrow fragment to dynamic fragment Apr 20, 2022
@acezen acezen changed the title Optimize performance of arrow fragment to dynamic fragment Optimize performance of arrow fragment to dynamic fragment with multi-threading Apr 21, 2022
@acezen acezen marked this pull request as ready for review April 21, 2022 02:03
@acezen acezen requested review from sighingnow and yecol April 21, 2022 02:03
@acezen acezen merged commit 8d5a029 into alibaba:main Apr 24, 2022
@acezen acezen deleted the acezen/optimize_convert branch April 24, 2022 06:38
@acezen acezen mentioned this pull request May 7, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sighingnow sighingnow sighingnow approved these changes

@yecol yecol Awaiting requested review from yecol

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@acezen @codecov-commenter @sighingnow