-
Notifications
You must be signed in to change notification settings - Fork 516
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Crash when saving MPT checkpoints trained via FSDP #221
Comments
@jquesnelle Thank you for the report. Could you confirm if you are using our Docker image, and if not, what version of |
This is a pytorch 2 bug. We are implementing a workaround in composer (mosaicml/composer#2237). Your options in the meantime are:
|
@sashaDoubov I installed Edit: Yeah, I think |
I see... As you mentioned, |
Closing this as it seems like we have a workaround, but @jquesnelle please feel free to reopen if you run into further issues. |
Meant to circle back and confirm that this does work! |
this was fixed in last composer version? |
I'm using this docker |
Hi @germanjke, the |
* enabling logit scaling * abhi review cmt * adding warning * add tests to exercise code path * enable str logit_scale options (samhavens review)
hi @sashaDoubov @abhi-mosaic @vchiley have you fixed this issue and now |
When finetuning
mpt-7B-storywriter
with FSDP,llm-foundry
crashes when saving the actual checkpoint. This was using latest master 4c94c20. This running on a single 4x A100 40GB node.The text was updated successfully, but these errors were encountered: