Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

AxoNN's implementation of mixed precision for hybrid parallelism #7

Merged
merged 10 commits into from
Feb 6, 2022
Merged

AxoNN's implementation of mixed precision for hybrid parallelism #7

merged 10 commits into from
Feb 6, 2022

Conversation

siddharth9820
Copy link
Collaborator

@siddharth9820 siddharth9820 commented Feb 6, 2022 •
edited
Loading

  • p2p send-recv now happens in fp16 tensors for mixed precision
  • Tested on Vit+Mnist
  • One bulk all-reduce call instead of individual calls on each parameter
  • Supports all-reduce in both fp-16 and fp-32 for mixed precision
  • also compatible with full precicion (one flag to toggle between mixed precision and full precision)

@siddharth9820 siddharth9820 merged commit 3d486ea into develop Feb 6, 2022
@siddharth9820 siddharth9820 deleted the mixed_precision branch February 6, 2022 06:27
@siddharth9820 siddharth9820 restored the mixed_precision branch February 8, 2022 03:09
@siddharth9820 siddharth9820 deleted the mixed_precision branch February 8, 2022 05:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@siddharth9820