Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support duplicated mode in DynamicFragment #325

Merged
merged 24 commits into from
May 28, 2021

Conversation

acezen
Copy link
Collaborator

@acezen acezen commented May 18, 2021

What do these changes do?

  • add a new loading graph mode which every worker duplicated load the whole graph.
  • make duplicated load graph mode as default of networkx.
  • use num_workers=2 in nx test

DynamicFragment now support two type of graph storage mode.

  • distributed mode: the storage is like other Fragments that every fragment holds a part of graph.
  • duplicated mode: every fragment holds the whole graph which duplicated in workers (the mode is designed for APSP-like algorithms and default in networkx).

if users want to use distributed mode to load graph, just init graph by

G = nx.Graph(dist=True)

Related issue number

#317

TODO:

@yecol
Copy link
Collaborator

yecol commented May 19, 2021

maybe revise the parameter name.

G = nx.Graph(duplicated_load=True)

@acezen acezen changed the title support load full graph in dynamic_fragment support duplicated load graph in dynamic_fragment May 20, 2021
@acezen acezen force-pushed the acezen/apsp-adaption branch 2 times, most recently from 186a880 to 5c7e275 Compare May 21, 2021 11:46
@acezen acezen marked this pull request as ready for review May 23, 2021 08:29
@codecov-commenter
Copy link

codecov-commenter commented May 23, 2021

Codecov Report

Merging #325 (88c903a) into main (1e6427e) will decrease coverage by 0.01%.
The diff coverage is 0.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #325      +/-   ##
==========================================
- Coverage   77.37%   77.36%   -0.02%     
==========================================
  Files          57       57              
  Lines        5300     5301       +1     
==========================================
  Hits         4101     4101              
- Misses       1199     1200       +1     
Impacted Files Coverage Δ
python/graphscope/framework/dag_utils.py 59.56% <0.00%> (-0.27%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b424da7...88c903a. Read the comment docs.

@acezen acezen changed the title support duplicated load graph in dynamic_fragment Support duplicated load whole graph every workers in dynamic_fragment for APSP algorithms May 24, 2021
@yecol
Copy link
Collaborator

yecol commented May 24, 2021

according to our discussion:

# by default, duplicated load
G = nx.Graph()

# or load a fragment, rather than a whole graph.
G = nx.Graph(dist=True)

@acezen acezen changed the title Support duplicated load whole graph every workers in dynamic_fragment for APSP algorithms Support duplicated mode in DynamicFragment May 28, 2021
@acezen
Copy link
Collaborator Author

acezen commented May 28, 2021

hi, @yecol @sighingnow @pwrliang the refactoring of duplicated load of DynamicFragment is done. the pr needs you to review again. Thanks a lot.

@acezen acezen merged commit 3e55fce into alibaba:main May 28, 2021
@acezen acezen deleted the acezen/apsp-adaption branch May 28, 2021 06:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants