Add base class DistLoader
#8079
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
JakubPietrakIntel
requested review from
mananshah99,
a team,
rusty1s and
wsad1
as code owners
JakubPietrakIntel
changed the title
DistLoader and utilsDistLoader
Codecov Report
@@ Coverage Diff @@
## master #8079 +/- ##
==========================================
- Coverage 89.42% 88.28% -1.14%
==========================================
Files 467 471 +4
Lines 27913 28151 +238
==========================================
- Hits 24962 24854 -108
- Misses 2951 3297 +346
... and 30 files with indirect coverage changes 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
This was referenced Sep 27, 2023
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
rusty1s
approved these changes
Oct 2, 2023
rusty1s
added a commit
that referenced
this pull request
Oct 9, 2023
This code belongs to the part of the whole distributed training for PyG. `DistNeighborSampler` leverages the `NeighborSampler` class from `pytorch_geometric` and the `neighbor_sample` function from `pyg-lib`. However, due to the fact that in case of distributed training it is required to synchronise the results between machines after each layer, the part of the code responsible for sampling was implemented in python. Added suport for the following sampling methods: - node, edge, negative, disjoint, temporal **TODOs:** - [x] finish hetero part - [x] subgraph sampling **This PR should be merged together with other distributed PRs:** pyg-lib: [#246](pyg-team/pyg-lib#246), [#252](pyg-team/pyg-lib#252) GraphStore\FeatureStore: #8083 DistLoaders: 1. #8079 2. #8080 3. #8085 --------- Co-authored-by: JakubPietrakIntel <jakub.pietrak@intel.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: ZhengHongming888 <hongming.zheng@intel.com> Co-authored-by: Jakub Pietrak <97102979+JakubPietrakIntel@users.noreply.github.com> Co-authored-by: Matthias Fey <matthias.fey@tu-dortmund.de>
rusty1s
added a commit
that referenced
this pull request
Nov 6, 2023
This code belongs to the part of the whole distributed training for PyG. Please be aware that this PR should be merged before Loaders package! - @JakubPietrakIntel Loaders: 1. #8079 2. #8080 3. #8085 Other PRs related to this module: DistSampler: #7974 --------- Co-authored-by: JakubPietrakIntel <jakub.pietrak@intel.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Matthias Fey <matthias.fey@tu-dortmund.de>
rusty1s
added a commit
that referenced
this pull request
Nov 10, 2023
**[2/3] Distributed Loaders PRs** This PR includes`DistributedNeighborLoader` used for processing node sampler output in distributed training setup. 1. #8079 2. #8080 3. #8085 Other PRs related to this module: DistSampler: #7974 GraphStore\FeatureStore: #8083 --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Matthias Fey <matthias.fey@tu-dortmund.de>
rusty1s
added a commit
that referenced
this pull request
Nov 10, 2023
**[3/3] Distributed Loaders PRs** This PR includes `DistributedLinkNeighborLoader` used for processing edge sampler output in distributed training setup. 1. #8079 2. #8080 3. #8085 Other PRs related to this module: DistSampler: #7974 GraphStore\FeatureStore: #8083 --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Matthias Fey <matthias.fey@tu-dortmund.de>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
[1/3] Distributed Loaders PRs
This PR includes base class of
DistributedLoaderthat handles RPC connection and handling requests fromDistributedNeighborSamplerprocesses.It includes basic
DistNeighborSamplerfunctions used by the loader.DistLoader#8079DistributedNeighborLoader[3/6] #8080DistributedLinkNeighborLoader[4/6] #8085Other PRs related to this module:
DistSampler: #7974
GraphStore\FeatureStore: #8083