Add validation tests for ESM2 fine-tuning benchmark partial-conv #1010
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pstjohn
approved these changes
Jul 30, 2025
nvmvle
force-pushed
the
mvle/jet-test-esm2-finetune
branch
from
54f15a1 to
589324c
Compare
jwilber
approved these changes
Aug 1, 2025
Collaborator
|
/ok to test 8e0042b |
Collaborator
Author
|
/ok to test dfd6913 |
trvachov
approved these changes
Aug 6, 2025
Collaborator
|
/ok to test e734275 |
Signed-off-by: My Le <mvle@nvidia.com>
Collaborator
Author
|
/ok to test 42918f6 |
Collaborator
|
/ok to test 42918f6 |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1010 +/- ##
=======================================
Coverage 81.29% 81.29%
=======================================
Files 152 152
Lines 10271 10271
=======================================
Hits 8350 8350
Misses 1921 1921 |
pstjohn
pushed a commit
to pstjohn/bionemo-framework
that referenced
this pull request
Aug 19, 2025
…DIA#1010) ### Description This PR adds a comprehensive test validation section to the ESM2 fine-tuning benchmark partial-conv configuration. The tests define expected baseline metrics and ranges for model performance validation during CI runs. The validation includes: - Exit code validation (expecting successful completion with code 0) - Performance baselines for key metrics: - `consumed_samples` - `val_loss` - `val_acc` - `TFLOPS per GPU` - `reduced_train_loss` ### Type of changes - [ ] Bug fix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Refactor - [ ] Documentation update - [ ] Other (please describe): ### CI Pipeline Configuration Configure CI behavior by applying the relevant labels: - [SKIP_CI](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#skip_ci) - Skip all continuous integration tests - [INCLUDE_NOTEBOOKS_TESTS](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#include_notebooks_tests) - Execute notebook validation tests in pytest - [INCLUDE_SLOW_TESTS](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#include_slow_tests) - Execute tests labelled as slow in pytest for extensive testing > [!NOTE] > By default, the notebooks validation tests are skipped unless explicitly enabled. #### Authorizing CI Runs We use [copy-pr-bot](https://docs.gha-runners.nvidia.com/apps/copy-pr-bot/#automation) to manage authorization of CI runs on NVIDIA's compute resources. * If a pull request is opened by a trusted user and contains only trusted changes, the pull request's code will automatically be copied to a pull-request/ prefixed branch in the source repository (e.g. pull-request/123) * If a pull request is opened by an untrusted user or contains untrusted changes, an NVIDIA org member must leave an `/ok to test` comment on the pull request to trigger CI. This will need to be done for each new commit. ### Pre-submit Checklist - [x] I have tested these changes locally - [ ] I have updated the documentation accordingly - [ ] I have added/updated tests as needed - [x] All existing tests pass successfully Signed-off-by: My Le <mvle@nvidia.com> --------- Signed-off-by: My Le <mvle@nvidia.com> Co-authored-by: Jared Wilber <jwilber@nvidia.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR adds a comprehensive test validation section to the ESM2 fine-tuning benchmark partial-conv configuration. The tests define expected baseline metrics and ranges for model performance validation during CI runs.
The validation includes:
consumed_samplesval_lossval_accTFLOPS per GPUreduced_train_lossType of changes
CI Pipeline Configuration
Configure CI behavior by applying the relevant labels:
Note
By default, the notebooks validation tests are skipped unless explicitly enabled.
Authorizing CI Runs
We use copy-pr-bot to manage authorization of CI
runs on NVIDIA's compute resources.
automatically be copied to a pull-request/ prefixed branch in the source repository (e.g. pull-request/123)
/ok to testcomment on the pull request to trigger CI. This will need to be done for each new commit.Pre-submit Checklist
Signed-off-by: My Le mvle@nvidia.com