Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[deepspeed] fix --load_best_model_at_end #14652

Merged
merged 8 commits into from
Dec 7, 2021
Merged

[deepspeed] fix --load_best_model_at_end #14652

merged 8 commits into from
Dec 7, 2021

Conversation

stas00
Copy link
Contributor

@stas00 stas00 commented Dec 6, 2021 •
edited
Loading

This PR is making --load_best_model_at_end reported in #14628 work.

There is a problem with resuming from a checkpoint after deepspeed engine was used microsoft/DeepSpeed#1394 - I made a new Issue there to stress out that this is a problem. microsoft/DeepSpeed#1612

This PR:

  • adds a workaround that re-creates the deepspeed engine when resuming from --load_best_model_at_end
  • fixes the timing when deepspeed checkpoint loading kicks in
  • adds a test
  • adds a commented out test that requires fixing on the deepspeed side

Additionally:

  • I added a saving of the DS checkpoint on model_save - I think that would be useful to some users - it's a very quick operation.

Fixes: #14628

@sgugger

@stas00 stas00 changed the title [deepspeed] fix load_best_model_at_end [deepspeed] fix --load_best_model_at_end Dec 6, 2021
@stas00 stas00 mentioned this pull request Dec 6, 2021
Closed
@stas00 stas00 mentioned this pull request Dec 6, 2021
Open
sgugger
sgugger approved these changes Dec 7, 2021
View reviewed changes
Copy link
Collaborator

@sgugger sgugger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing!

@stas00 stas00 marked this pull request as ready for review December 7, 2021 01:00
@stas00 stas00 merged commit b66c5ab into huggingface:master Dec 7, 2021
@stas00 stas00 deleted the ds-load-best-model branch December 7, 2021 05:57
Albertobegue pushed a commit to Albertobegue/transformers that referenced this pull request Jan 27, 2022
@stas00
[deepspeed] fix --load_best_model_at_end (huggingface#14652)
ecb9ab7 
* [deepspeed] fix load_best_model_at_end

* try with pull_request_target

* revert: try with pull_request_target

* style

* add test

* cleanup
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sgugger sgugger sgugger approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

load_best_model_at_end failed due to "size mismatch" when DeepSpeed is used
2 participants
@stas00 @sgugger