evo2 stop and go test by yzhang123 · Pull Request #903 · NVIDIA-BioNeMo/bionemo-framework

yzhang123 · 2025-05-30T16:27:47Z

Description

Type of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Refactor
Documentation update
Other (please describe):

CI Pipeline Configuration

Configure CI behavior by applying the relevant labels:

SKIP_CI - Skip all continuous integration tests
INCLUDE_NOTEBOOKS_TESTS - Execute notebook validation tests in pytest
INCLUDE_SLOW_TESTS - Execute tests labelled as slow in pytest for extensive testing

Note

By default, the notebooks validation tests are skipped unless explicitly enabled.

Authorizing CI Runs

We use copy-pr-bot to manage authorization of CI
runs on NVIDIA's compute resources.

If a pull request is opened by a trusted user and contains only trusted changes, the pull request's code will
automatically be copied to a pull-request/ prefixed branch in the source repository (e.g. pull-request/123)
If a pull request is opened by an untrusted user or contains untrusted changes, an NVIDIA org member must leave an
/ok to test comment on the pull request to trigger CI. This will need to be done for each new commit.

Usage

TODO: Add code snippet

Pre-submit Checklist

I have tested these changes locally
I have updated the documentation accordingly
I have added/updated tests as needed
All existing tests pass successfully

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

copy-pr-bot · 2025-05-30T16:27:51Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

yzhang123 · 2025-05-30T16:28:44Z

/ok to test 3d8b9c0

codecov-commenter · 2025-05-30T17:33:29Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 84.21%. Comparing base (5aa0d3b) to head (e891fce).

✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #903   +/-   ##
=======================================
  Coverage   84.21%   84.21%           
=======================================
  Files         143      143           
  Lines        9061     9061           
=======================================
  Hits         7631     7631           
  Misses       1430     1430

see 1 file with indirect coverage changes

jstjohn

Looks good, left one comment related to a smaller model config.

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

…nemo-framework into evo2_stop_and_go_test

yzhang123 · 2025-06-04T14:23:04Z

/ok to test 73c8003

copy-pr-bot · 2025-06-04T14:23:07Z

/ok to test 73c8003

@yzhang123, there was an error processing your request: E2

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/

yzhang123 · 2025-06-04T14:24:30Z

/ok to test

copy-pr-bot · 2025-06-04T14:24:33Z

/ok to test

@yzhang123, there was an error processing your request: E1

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/1/

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

yzhang123 · 2025-06-04T14:34:09Z

/ok test ccf92c2

yzhang123 · 2025-06-04T14:36:09Z

/ok test ccf92c2

yzhang123 · 2025-06-04T14:36:24Z

/ok test ccf92c2

yzhang123 · 2025-06-04T14:53:02Z

/ok test 3d8b9c0

trvachov · 2025-06-04T15:41:47Z

@dorotat-nv can I ask for you review here?

dorotat-nv · 2025-06-04T15:49:10Z

great work, LGTM

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

jwilber · 2025-06-05T17:01:35Z

/ok to test c46b81b

jwilber · 2025-06-10T18:03:57Z

/ok to test e891fce

### Description  ### Type of changes  - [ ] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Refactor - [ ] Documentation update - [ ] Other (please describe): ### CI Pipeline Configuration Configure CI behavior by applying the relevant labels: - [SKIP_CI](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#skip_ci) - Skip all continuous integration tests - [INCLUDE_NOTEBOOKS_TESTS](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#include_notebooks_tests) - Execute notebook validation tests in pytest - [INCLUDE_SLOW_TESTS](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#include_slow_tests) - Execute tests labelled as slow in pytest for extensive testing > [!NOTE] > By default, the notebooks validation tests are skipped unless explicitly enabled. #### Authorizing CI Runs We use [copy-pr-bot](https://docs.gha-runners.nvidia.com/apps/copy-pr-bot/#automation) to manage authorization of CI runs on NVIDIA's compute resources. * If a pull request is opened by a trusted user and contains only trusted changes, the pull request's code will automatically be copied to a pull-request/ prefixed branch in the source repository (e.g. pull-request/123) * If a pull request is opened by an untrusted user or contains untrusted changes, an NVIDIA org member must leave an `/ok to test` comment on the pull request to trigger CI. This will need to be done for each new commit. ### Usage  ```python TODO: Add code snippet ``` ### Pre-submit Checklist  - [x] I have tested these changes locally - [ ] I have updated the documentation accordingly - [x] I have added/updated tests as needed - [x] All existing tests pass successfully --------- Signed-off-by: Yang Zhang <yangzhang@nvidia.com> Co-authored-by: Jared Wilber <jwilber@nvidia.com> Signed-off-by: Ubuntu <camirr@nvidia.com>

evo2 stop and go test

3d8b9c0

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

yzhang123 requested review from cspades, dorotat-nv, jomitchellnv, jstjohn, jwilber, malcolmgreaves, pstjohn, sichu2023, skothenhill-nv and trvachov as code owners May 30, 2025 16:27

jstjohn reviewed May 30, 2025

View reviewed changes

Comment thread sub-packages/bionemo-evo2/tests/bionemo/evo2/test_stop_and_go.py Outdated

jstjohn approved these changes May 30, 2025

View reviewed changes

yzhang123 added 7 commits June 2, 2025 05:18

disable TE and add init_global_step to 0

7d4eb2d

evo2 stop and go test

0c8c2e1

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

disable TE and add init_global_step to 0

aac20d8

save for now

c5a64a8

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

change layers to the same as in train.py

8d66548

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

working test

2e62eb9

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

Merge branch 'evo2_stop_and_go_test' of https://github.com/NVIDIA/bio…

c2ebb1f

…nemo-framework into evo2_stop_and_go_test

yzhang123 requested review from DejunL, farhadrgh, guoqing-zhou, nvdreidenbach, tshimko-nv, youhanl-nvidia and zcao0420 as code owners June 4, 2025 14:20

add transformer engine back

ccf92c2

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

Merge remote-tracking branch 'origin/main' into evo2_stop_and_go_test

b986324

dorotat-nv approved these changes Jun 4, 2025

View reviewed changes

yzhang123 enabled auto-merge June 4, 2025 16:03

yzhang123 and others added 3 commits June 4, 2025 09:18

forgot to add test

ad792c0

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

reduce test steps

50ac810

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

Merge branch 'main' into evo2_stop_and_go_test

c46b81b

yzhang123 added this pull request to the merge queue Jun 9, 2025

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jun 9, 2025

yzhang123 added this pull request to the merge queue Jun 10, 2025

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jun 10, 2025

Merge branch 'main' into evo2_stop_and_go_test

e891fce

jwilber enabled auto-merge June 10, 2025 18:04

jwilber added this pull request to the merge queue Jun 10, 2025

Merged via the queue into main with commit 9e34dd9 Jun 10, 2025
10 checks passed

jwilber deleted the evo2_stop_and_go_test branch June 10, 2025 20:56

Conversation

yzhang123 commented May 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of changes

CI Pipeline Configuration

Authorizing CI Runs

Usage

Pre-submit Checklist

Uh oh!

copy-pr-bot Bot commented May 30, 2025

Uh oh!

yzhang123 commented May 30, 2025

Uh oh!

codecov-commenter commented May 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

jstjohn left a comment

Choose a reason for hiding this comment

Uh oh!

yzhang123 commented Jun 4, 2025

Uh oh!

copy-pr-bot Bot commented Jun 4, 2025

Uh oh!

yzhang123 commented Jun 4, 2025

Uh oh!

copy-pr-bot Bot commented Jun 4, 2025

Uh oh!

yzhang123 commented Jun 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yzhang123 commented Jun 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yzhang123 commented Jun 4, 2025

Uh oh!

yzhang123 commented Jun 4, 2025

Uh oh!

trvachov commented Jun 4, 2025

Uh oh!

dorotat-nv commented Jun 4, 2025

Uh oh!

jwilber commented Jun 5, 2025

Uh oh!

Uh oh!

Uh oh!

jwilber commented Jun 10, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

yzhang123 commented May 30, 2025 •

edited

Loading

codecov-commenter commented May 30, 2025 •

edited

Loading

yzhang123 commented Jun 4, 2025 •

edited

Loading

yzhang123 commented Jun 4, 2025 •

edited

Loading