change checkpoint name pattern#786
Merged
Merged
Conversation
Signed-off-by: Farhad Ramezanghorbani <farhadr@nvidia.com>
farhadrgh
requested review from
cspades,
dorotat-nv,
jomitchellnv,
jstjohn,
jwilber,
malcolmgreaves,
pstjohn,
sichu2023,
skothenhill-nv and
trvachov
as code owners
Collaborator
Author
|
/ok to test |
pstjohn
approved these changes
Mar 31, 2025
sichu2023
reviewed
Mar 31, 2025
Codecov ReportAll modified and coverable lines are covered by tests ✅
✅ All tests successful. No failed tests found. Additional details and impacted files@@ Coverage Diff @@
## main #786 +/- ##
=======================================
Coverage ? 84.30%
=======================================
Files ? 137
Lines ? 8635
Branches ? 0
=======================================
Hits ? 7280
Misses ? 1355
Partials ? 0 ☔ View full report in Codecov by Sentry. |
sichu2023
approved these changes
Apr 1, 2025
cspades
pushed a commit
that referenced
this pull request
May 4, 2025
### Description
We have the following pattern in the callback of train.py:
```python
filename="{epoch}-{val_loss:.2f}-{step}-{consumed_samples}"
```
The issue raises is due to {val_loss:.2f} - Different ranks have
slightly different validation loss values due to distributed training,
leading to different directories per rank. We can avoid this by
switching to the following. This aggregates the rank specific data into
the same dir which we use as the aggregated checkpoint.
```python
filename="checkpoint-{step}-{consumed_samples}"
```
This works because both `step` and `consumed_samples` are global
counters that stay synchronized across all ranks during distributed
training.
### Type of changes
<!-- Mark the relevant option with an [x] -->
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Refactor
- [ ] Documentation update
- [ ] Other (please describe):
### CI Pipeline Configuration
Configure CI behavior by applying the relevant labels:
-
[SKIP_CI](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#skip_ci)
- Skip all continuous integration tests
-
[INCLUDE_NOTEBOOKS_TESTS](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#include_notebooks_tests)
- Execute notebook validation tests in pytest
-
[INCLUDE_SLOW_TESTS](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#include_slow_tests)
- Execute tests labelled as slow in pytest for extensive testing
> [!NOTE]
> By default, the notebooks validation tests are skipped unless
explicitly enabled.
#### Authorizing CI Runs
We use
[copy-pr-bot](https://docs.gha-runners.nvidia.com/apps/copy-pr-bot/#automation)
to manage authorization of CI
runs on NVIDIA's compute resources.
* If a pull request is opened by a trusted user and contains only
trusted changes, the pull request's code will
automatically be copied to a pull-request/ prefixed branch in the source
repository (e.g. pull-request/123)
* If a pull request is opened by an untrusted user or contains untrusted
changes, an NVIDIA org member must leave an
`/ok to test` comment on the pull request to trigger CI. This will need
to be done for each new commit.
### Usage
<!--- How does a user interact with the changed code -->
```python
TODO: Add code snippet
```
### Pre-submit Checklist
<!--- Ensure all items are completed before submitting -->
- [ ] I have tested these changes locally
- [ ] I have updated the documentation accordingly
- [ ] I have added/updated tests as needed
- [ ] All existing tests pass successfully
Signed-off-by: Farhad Ramezanghorbani <farhadr@nvidia.com>
Signed-off-by: Cory Ye <cye@nvidia.com>
farhadrgh
added a commit
that referenced
this pull request
May 5, 2025
### Description
We have the following pattern in the callback of train.py:
```python
filename="{epoch}-{val_loss:.2f}-{step}-{consumed_samples}"
```
The issue raises is due to {val_loss:.2f} - Different ranks have
slightly different validation loss values due to distributed training,
leading to different directories per rank. We can avoid this by
switching to the following. This aggregates the rank specific data into
the same dir which we use as the aggregated checkpoint.
```python
filename="checkpoint-{step}-{consumed_samples}"
```
This works because both `step` and `consumed_samples` are global
counters that stay synchronized across all ranks during distributed
training.
### Type of changes
<!-- Mark the relevant option with an [x] -->
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Refactor
- [ ] Documentation update
- [ ] Other (please describe):
### CI Pipeline Configuration
Configure CI behavior by applying the relevant labels:
-
[SKIP_CI](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#skip_ci)
- Skip all continuous integration tests
-
[INCLUDE_NOTEBOOKS_TESTS](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#include_notebooks_tests)
- Execute notebook validation tests in pytest
-
[INCLUDE_SLOW_TESTS](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#include_slow_tests)
- Execute tests labelled as slow in pytest for extensive testing
> [!NOTE]
> By default, the notebooks validation tests are skipped unless
explicitly enabled.
#### Authorizing CI Runs
We use
[copy-pr-bot](https://docs.gha-runners.nvidia.com/apps/copy-pr-bot/#automation)
to manage authorization of CI
runs on NVIDIA's compute resources.
* If a pull request is opened by a trusted user and contains only
trusted changes, the pull request's code will
automatically be copied to a pull-request/ prefixed branch in the source
repository (e.g. pull-request/123)
* If a pull request is opened by an untrusted user or contains untrusted
changes, an NVIDIA org member must leave an
`/ok to test` comment on the pull request to trigger CI. This will need
to be done for each new commit.
### Usage
<!--- How does a user interact with the changed code -->
```python
TODO: Add code snippet
```
### Pre-submit Checklist
<!--- Ensure all items are completed before submitting -->
- [ ] I have tested these changes locally
- [ ] I have updated the documentation accordingly
- [ ] I have added/updated tests as needed
- [ ] All existing tests pass successfully
Signed-off-by: Farhad Ramezanghorbani <farhadr@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
We have the following pattern in the callback of train.py:
The issue raises is due to {val_loss:.2f} - Different ranks have slightly different validation loss values due to distributed training, leading to different directories per rank. We can avoid this by switching to the following. This aggregates the rank specific data into the same dir which we use as the aggregated checkpoint.
This works because both
stepandconsumed_samplesare global counters that stay synchronized across all ranks during distributed training.Type of changes
CI Pipeline Configuration
Configure CI behavior by applying the relevant labels:
Note
By default, the notebooks validation tests are skipped unless explicitly enabled.
Authorizing CI Runs
We use copy-pr-bot to manage authorization of CI
runs on NVIDIA's compute resources.
automatically be copied to a pull-request/ prefixed branch in the source repository (e.g. pull-request/123)
/ok to testcomment on the pull request to trigger CI. This will need to be done for each new commit.Usage
Pre-submit Checklist