[RFC] Implement distributed graph store to train giant graph on machines with limited memory

[futurely]

🚀 Feature

Implement distributed graph store to train giant graph on machines with limited memory.

Motivation

Many graphs need hundreds of GB or more memory to load into graph store. It's common that a single machine cannot load such graphs. So this problem is a serious blocker for large scale applications of DGL. Some other frameworks such as PyTorch-BigGraph and Euler partition the graph into smaller files for fully distributed training.

Pitch

Efficient distributed training with arbitrarily large graphs. Wider adoption of DGL especially in the industry.

[jermainewang]

Hey @futurely , thanks for your interests in the system optimization. If you have no objection I'd like to re-organize some of your feature requests into few RFCs so we could have more focused discussion.

[futurely]

It's okay. Thanks!

[zheng-da]

Thanks for your request. In the next release, we will have distributed training capability. Please stay tuned.

[ChenAris]

We are also interested in distributed training across multiple machines and want to profile the communication cost. Is there a rough tentative date for the new release supporting this feature? Thanks.

[futurely]

Baidu's Paddle Graph Learning (PGL) supports distributed graph store with Redis.

https://github.com/PaddlePaddle/PGL#large-scale-support-distributed-graph-storage-and-distributed-training-algorithms

https://github.com/PaddlePaddle/PGL/blob/master/pgl/redis_graph.py

https://github.com/PaddlePaddle/PGL/blob/master/examples/distribute_graphsage/reader.py

[github-actions]

This issue has been automatically marked as stale due to lack of activity. It will be closed if no further activity occurs. Thank you

[github-actions]

This issue is closed due to lack of activity. Feel free to reopen it if you still have questions.