Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add mixed precision #74

Merged
merged 6 commits into from
Jun 20, 2021
Merged

Add mixed precision #74

merged 6 commits into from
Jun 20, 2021

Conversation

seanmor5
Copy link
Contributor

WIP.

Need to think of a better way to test

@seanmor5
Copy link
Contributor Author

For reference, this is based off of: https://github.com/deepmind/jmp

@josevalim
Copy link
Contributor

LGTM! Although I would consider working on some lightweight tests although I am not quite sure how to test this... maybe by testing directly on the Nx expression?

@seanmor5
Copy link
Contributor Author

@josevalim I think that might be the best way also, the expression might end up being really large. There is also a problem right now where I don't think it is respecting the policy in all cases. I can guarantee the policy is correct for compute and output because it's just a simple cast; however, the params policy doesn't seem to stay consistent even though they're initialized to {:f, 16} or {:bf, 16}

@seanmor5 seanmor5 mentioned this pull request Jun 19, 2021
@seanmor5 seanmor5 marked this pull request as ready for review June 20, 2021 18:06
@seanmor5 seanmor5 merged commit 103a1f3 into main Jun 20, 2021
@seanmor5 seanmor5 deleted the sm-mixed-precision branch June 20, 2021 18:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants