Ensure EMA checkpoints are also deleted when normal checkpoints are #5724
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: SeanNaren <snarenthiran@nvidia.com>
SeanNaren
commented
Jan 3, 2023
| @@ -1,6 +1,6 @@ | |||
| hydra-core>=1.2.0,<1.3 | |||
| omegaconf>=2.2,<2.3 | |||
| pytorch-lightning>=1.8.3 | |||
| pytorch-lightning>=1.8.6 | |||
There was a problem hiding this comment.
Hope its ok to upgrade the version here @titu1994
There was a problem hiding this comment.
Should be fine, r1.15 has been cut already so it affects only main.
Signed-off-by: SeanNaren <snarenthiran@nvidia.com>
SeanNaren
commented
Jan 3, 2023
| @@ -248,6 +248,27 @@ def test_exp_manager_ema_weights(self, tmpdir): | |||
| for saved_weight, ema_weight in zip(duplicate_model.state_dict().values(), ema_weights): | |||
| assert torch.allclose(saved_weight.cpu(), ema_weight.cpu()) | |||
|
|
|||
| @pytest.mark.unit | |||
| def test_exp_manager_ema_weights_topk(self, tmpdir): | |||
There was a problem hiding this comment.
Confirmed this test fails when using pytorch-lightning==1.8.3
Signed-off-by: SeanNaren <snarenthiran@nvidia.com>
Signed-off-by: SeanNaren <snarenthiran@nvidia.com>
Signed-off-by: SeanNaren <snarenthiran@nvidia.com>
titu1994
approved these changes
Jan 5, 2023
| @@ -1,6 +1,6 @@ | |||
| hydra-core>=1.2.0,<1.3 | |||
| omegaconf>=2.2,<2.3 | |||
| pytorch-lightning>=1.8.3 | |||
| pytorch-lightning>=1.8.6 | |||
There was a problem hiding this comment.
Should be fine, r1.15 has been cut already so it affects only main.
erastorgueva-nv
pushed a commit
that referenced
this pull request
Jan 12, 2023
…5724) * Ensure EMA checkpoints are also deleted when normal checkpoints are Signed-off-by: SeanNaren <snarenthiran@nvidia.com> * Simplify test Signed-off-by: SeanNaren <snarenthiran@nvidia.com> * Remove comment Signed-off-by: SeanNaren <snarenthiran@nvidia.com> * Fix bug where `save_best_model` caused a crash Signed-off-by: SeanNaren <snarenthiran@nvidia.com> * Swap to logging only on rank 0 Signed-off-by: SeanNaren <snarenthiran@nvidia.com> Signed-off-by: SeanNaren <snarenthiran@nvidia.com> Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>
titu1994
pushed a commit
to titu1994/NeMo
that referenced
this pull request
Mar 24, 2023
…VIDIA#5724) * Ensure EMA checkpoints are also deleted when normal checkpoints are Signed-off-by: SeanNaren <snarenthiran@nvidia.com> * Simplify test Signed-off-by: SeanNaren <snarenthiran@nvidia.com> * Remove comment Signed-off-by: SeanNaren <snarenthiran@nvidia.com> * Fix bug where `save_best_model` caused a crash Signed-off-by: SeanNaren <snarenthiran@nvidia.com> * Swap to logging only on rank 0 Signed-off-by: SeanNaren <snarenthiran@nvidia.com> Signed-off-by: SeanNaren <snarenthiran@nvidia.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do ?
Closes #5631
Collection: Common
Changelog
Before your PR is "Ready for review"
Pre checks:
PR Type: