-
Notifications
You must be signed in to change notification settings - Fork 510
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Finetuning with weights in bfloat16 #100
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A few suggested changes
Co-authored-by: Luca Antiga <luca@lightning.ai>
Does memory fit a 3090 even in the LoRA case? |
oh ahah, that's tight! |
better check IMO |
Attempted to test it but got hold up by this problem: #101 |
The finetuning fits into the 3090: I had to install pytorch nightly to get around the issue #101 though. Perhaps we should hold off merging this? Or we could say this requires pytorch nightly. Or we could investigate whether changing the implementation can avoid the error. |
Great, I would merge and say that 3090 requires nightly then we’ll investigate complex again |
Amazing, let’s merge! |
Co-authored-by: Luca Antiga <luca@lightning.ai>
The memory consumption is now ~20GB compared to before at ~38GB.
The training iteration speed-up ~1.5x-2x.