Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactored E2E example for Distributed CPU solution #8713

Merged
merged 44 commits into from
Feb 4, 2024
Merged

Refactored E2E example for Distributed CPU solution #8713

merged 44 commits into from
Feb 4, 2024

Conversation

JakubPietrakIntel
Copy link
Contributor

@JakubPietrakIntel JakubPietrakIntel commented Jan 3, 2024 •
edited

This PR adds an improved and refactored E2E example using GraphSAGE and OGB datasets for both homogenous (ogbn-products) and heterogenous (ogbn-mag) data.

Changes wrt #8029:

  • Added heterogenous example
  • Merged homo & hetero into one script
  • Aligned with partitioning changes Rearrange partitions file structure #8638
  • Simplified user input
  • Improved display & logging
  • Enabled multithreading by default - hotfix for slow hetero sampling
  • Enabled 'persistent_workers` by default - hotfix for breaking RPC connection between train & test stages
  • Updated README
    Review:
  • Moved attribute assignment from load_partition_info() to LGS/LFS from_partition() and simplified Stores initiation from partition files.
  • Adjusted the tests.

@JakubPietrakIntel JakubPietrakIntel marked this pull request as ready for review January 3, 2024 14:50
kgajdamo
kgajdamo reviewed Jan 4, 2024
View reviewed changes
examples/distributed/pyg/README.md Outdated Show resolved Hide resolved
examples/distributed/pyg/README.md Outdated Show resolved Hide resolved
examples/distributed/pyg/README.md Outdated Show resolved Hide resolved
examples/distributed/pyg/distributed_cpu.py Outdated Show resolved Hide resolved
examples/distributed/pyg/distributed_cpu.py Show resolved Hide resolved
examples/distributed/pyg/distributed_cpu.py Outdated Show resolved Hide resolved
examples/distributed/pyg/distributed_cpu.py Outdated Show resolved Hide resolved
examples/distributed/pyg/distributed_cpu.py Outdated Show resolved Hide resolved
examples/distributed/pyg/distributed_cpu.py Outdated Show resolved Hide resolved
@JakubPietrakIntel JakubPietrakIntel mentioned this pull request Jan 4, 2024
Closed
@JakubPietrakIntel
Copy link
Contributor Author

#8029 has been closed and can be replaced by this PR

@rusty1s rusty1s added the feature label Jan 5, 2024
rusty1s
rusty1s reviewed Jan 22, 2024
View reviewed changes
examples/distributed/pyg/distributed_cpu.py Outdated Show resolved Hide resolved
examples/distributed/pyg/distributed_cpu.py Outdated Show resolved Hide resolved
examples/distributed/pyg/distributed_cpu.py Outdated Show resolved Hide resolved
examples/distributed/pyg/distributed_cpu.py Outdated Show resolved Hide resolved
examples/distributed/pyg/distributed_cpu.py Outdated Show resolved Hide resolved
examples/distributed/pyg/distributed_cpu.py Show resolved Hide resolved
examples/distributed/pyg/distributed_cpu.py Outdated Show resolved Hide resolved
examples/distributed/pyg/distributed_cpu.py Outdated Show resolved Hide resolved
rusty1s
rusty1s approved these changes Feb 4, 2024
View reviewed changes
Copy link
Member

@rusty1s rusty1s left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very clean, thanks for the clean-up.

@rusty1s rusty1s enabled auto-merge (squash) February 4, 2024 10:37
@rusty1s rusty1s merged commit 4984687 into master Feb 4, 2024
14 checks passed
@rusty1s rusty1s deleted the intel/dist_e2e_example branch February 4, 2024 10:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kgajdamo kgajdamo kgajdamo left review comments

@rusty1s rusty1s rusty1s approved these changes

@ZhengHongming888 ZhengHongming888 Awaiting requested review from ZhengHongming888

@DamianSzwichtenberg DamianSzwichtenberg Awaiting requested review from DamianSzwichtenberg

@wsad1 wsad1 Awaiting requested review from wsad1 wsad1 is a code owner

@mananshah99 mananshah99 Awaiting requested review from mananshah99 mananshah99 is a code owner

@EdisonLeeeee EdisonLeeeee Awaiting requested review from EdisonLeeeee

Assignees

@JakubPietrakIntel JakubPietrakIntel

Labels
0 - Priority P0 distributed example feature loader
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@JakubPietrakIntel @ZhengHongming888 @rusty1s @kgajdamo