[WIP] [checkpoint v2] Use remote uploader v2 for checkpointing #3320
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…2_for_checkpointing
bigning
changed the title
…2_for_checkpointing
bigning
force-pushed
the
add_remote_file_uploader
branch
from
78e90f5
to
99bd9b7
Compare
…2_for_checkpointing
bigning
force-pushed
the
use_remote_uploader_v2_for_checkpointing
branch
from
6e9612e
to
deaec50
Compare
mvpatel2000
reviewed
May 30, 2024
composer/checkpoint/state_dict.py
Outdated
Comment on lines
159
to
164
| def get_metadata_state_dict( | ||
| model: Optional[Union[ComposerModel, nn.Module]] = None, | ||
| sharded_state_dict: Optional[bool] = None, | ||
| precision: Optional[Union[str, torch.dtype]] = None, | ||
| device: Optional[Device] = None, | ||
| device_train_microbatch_size: Optional[int] = None, |
There was a problem hiding this comment.
Is this copied from Evan's PR?
There was a problem hiding this comment.
oh it got messed up. will clean up
bigning
changed the title
eracah
reviewed
May 30, 2024
Comment on lines
+552
to
+556
| if self.use_remote_uploader_v2 and self.remote_uploader is not None: | ||
| self.remote_uploader.upload_file_async( | ||
| remote_file_name=symlink_name, | ||
| file_path=symlink_filename, | ||
| overwrite=True, |
There was a problem hiding this comment.
ok this makes sense for a PR, where we just simplify the remoteuploader and don't change any functionality, but eventually we want to have the remoteuploader manage symlinks. That way we can actually avoid race conditions
|
Discussed with @eracah , we're going to add a new checkpoint saver, instead of changing current one. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
what?
Composer change to use the new RemoteUploader for saving checkpoing
Test
llm-foundry PR: mosaicml/llm-foundry#1237
1-node small model OCI test
test-uploader-wfA8Vz
1-node small model S3 test
save: test-uploader-10udLt e2e time: 3 min 40s
resume: test-uploader-TDGgTs
use old remote_uploader_downloader
test-uploader-racGuCe2e time: 3 min 42s