Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reduce fragment size #2628

Open
3 of 4 tasks
yecol opened this issue Apr 20, 2023 · 3 comments
Open
3 of 4 tasks

Reduce fragment size #2628

yecol opened this issue Apr 20, 2023 · 3 comments
Assignees
@sighingnow @luoxiaojian @vegetableysm
Labels
performance Performance related issues
Projects
GraphScope Sprint April-May

Comments

@yecol
Copy link
Collaborator

yecol commented Apr 20, 2023 •
edited by sighingnow

as titled, reduce the fragment size with potential optimization (e.g., varint)

  • baseline test, compare with Plato, GraphX and other systems....
  • optimization-1: varint, considering compatibility, co-exists with current fragments.
  • optimization-2: perfect hashing, (advise @luoxiaojian)
  • test and report effectiveness.
@yecol yecol added the performance Performance related issues label Apr 20, 2023
@sighingnow sighingnow added this to To do in GraphScope Sprint April-May via automation Apr 26, 2023
@lidongze0629 lidongze0629 moved this from To do to In progress in GraphScope Sprint April-May May 10, 2023
This was referenced May 12, 2023
Closed
Closed
@sighingnow sighingnow mentioned this issue May 26, 2023
Merged
sighingnow added a commit to v6d-io/v6d that referenced this issue May 26, 2023
@vegetableysm @sighingnow
Implements varint + delta encoding on CSR to optimize the fragment me…
0c9c16e 
…mory usage (#1372)

Related issue number
--------------------

- Fixes #1373
- Part of alibaba/GraphScope#2628

---------

Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>
Signed-off-by: Tao He <linzhu.ht@alibaba-inc.com>
Co-authored-by: Tao He <linzhu.ht@alibaba-inc.com>
@sighingnow sighingnow mentioned this issue Jun 2, 2023
Merged
sighingnow added a commit that referenced this issue Jun 5, 2023
@sighingnow
Adapt to latest vineyard for varint encoding in CSR (#2791)
278a24c 
## Related issue number

Part of #2628

Signed-off-by: Tao He <linzhu.ht@alibaba-inc.com>
@sighingnow
Copy link
Collaborator

baseline test, compare with Plato, GraphX and other systems....

Systems:

  • GraphScope
  • Plato
  • GraphX
  • Giraph
  • Gemini

Datasets:

  • soc-LiveJournal1
  • soc-twitter-2010
  • soc-sinaweibo
  • soc-orkut

Algorithms:

  • pagerank
  • sssp

Measurements:

  • Peek memory
    • during running graph
    • during running application
  • Memory after loading graph
  • Running time
    • loading graph
    • running application

@yecol @vegetableysm Any comments on the plan above?

@sighingnow
Copy link
Collaborator

Refer to https://github.com/alibaba/libgrape-lite/blob/master/Performance.md

@yecol
Copy link
Collaborator Author

yecol commented Jun 7, 2023

Sounds like a good plan!

@sighingnow sighingnow mentioned this issue Jul 24, 2023
Merged
sighingnow added a commit that referenced this issue Jul 24, 2023
@sighingnow
Add the performance & tunning page for GAE (#3046)
cd7cbf2 
Part of #2628

Signed-off-by: Tao He <linzhu.ht@alibaba-inc.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sighingnow sighingnow

@luoxiaojian luoxiaojian

@vegetableysm vegetableysm

Labels
performance Performance related issues
Projects
No open projects
GraphScope Sprint April-May
  
In progress
Milestone
No milestone
Development

No branches or pull requests
4 participants
@yecol @sighingnow @luoxiaojian @vegetableysm