Skip to content

If desired, training can be stopped on a specific step without impacting the LR curve.#739

Merged
jstjohn merged 4 commits into
mainfrom
jstjohn/stop-at-step-callback
Mar 12, 2025
Merged

If desired, training can be stopped on a specific step without impacting the LR curve.#739
jstjohn merged 4 commits into
mainfrom
jstjohn/stop-at-step-callback

Conversation

@jstjohn
Copy link
Copy Markdown
Collaborator

@jstjohn jstjohn commented Mar 10, 2025

See new option --early-stop-on-step which may be used in JET tests to stop on a specific step without impacting the LR curve by changing --max-steps

…ing the LR curve by setting --early-stop-on-step

Signed-off-by: John St John <jstjohn@nvidia.com>
Copy link
Copy Markdown
Collaborator

@dorotat-nv dorotat-nv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still, would be nice to add a unit tests with label slow

Signed-off-by: John St John <jstjohn@nvidia.com>
@jstjohn jstjohn enabled auto-merge March 10, 2025 16:33
Comment thread sub-packages/bionemo-evo2/tests/bionemo/evo2/run/test_train.py
jstjohn added 2 commits March 10, 2025 16:46
Signed-off-by: John St John <jstjohn@nvidia.com>
Signed-off-by: John St John <jstjohn@nvidia.com>
@jstjohn jstjohn added this pull request to the merge queue Mar 11, 2025
@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Mar 11, 2025

Codecov Report

❌ Patch coverage is 53.84615% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.15%. Comparing base (588fb75) to head (785aaf4).
⚠️ Report is 515 commits behind head on main.

Files with missing lines Patch % Lines
...ackages/bionemo-evo2/src/bionemo/evo2/run/train.py 20.00% 4 Missing ⚠️
...o-testing/src/bionemo/testing/testing_callbacks.py 75.00% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #739      +/-   ##
==========================================
- Coverage   85.93%   84.15%   -1.78%     
==========================================
  Files         120      128       +8     
  Lines        7310     8029     +719     
==========================================
+ Hits         6282     6757     +475     
- Misses       1028     1272     +244     
Files with missing lines Coverage Δ
...o-testing/src/bionemo/testing/testing_callbacks.py 96.11% <75.00%> (-1.85%) ⬇️
...ackages/bionemo-evo2/src/bionemo/evo2/run/train.py 17.12% <20.00%> (ø)

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Merged via the queue into main with commit 7428f5f Mar 12, 2025
@jstjohn jstjohn deleted the jstjohn/stop-at-step-callback branch March 12, 2025 00:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants