parallelize test stages in GitHub CI

[dorotat-nv]

Description

Parallelising testing stages in GitHub CI to run in the same time to speedup pipeline
Added verify-tests-status as a final job that checks the tests status and if success or skipped, then success

Checked that everything works

• when if-statement moved to the main body of the job, then there is no image squashing if the label is not selected, simply
  test status job collects test status so we will be able to add it as a check for pipeline success
• Here is pipeline with skipped notebooks (Verify All Tests Status passes)
  https://github.com/NVIDIA/bionemo-framework/actions/runs/13976419866/job/39135551206
• Here is a pipeline with failed notebooks (Verify All Tests Status fails since notebooks fail)
  https://github.com/NVIDIA/bionemo-framework/actions/runs/13991973092/job/39177843004?pr=768

EDIT:
Removed flag INCLUDE_NOTEBOOKS_TESTS since some of the notebooks do not pass CI pipeline, see
failed job logs.

Fix is being merged in #743

Current workflow in CI

• run-test jobs run sequentially on one runner
• image squashing need to run before the first (unit tests) job only
  ** duration: 18 min, see log
• the pipeline duration if not SKIP_CI
  **minimum: image_squashing_time + duration_L0 ~ 50mins,
  **maximum: image_squashing_time + duration_L0, + duration_L1 + duration_docs ~ 2h

Proposed parallelization:

• run-test jobs run in parallel since we have access to multiple runners
• image squashing needs to run before each of the test job
  duration: 18 min, see log
• the pipeline duration if not SKIP_CI
  **minimum: image_squashing_time + duration_L0 ~ 50mins,
  **maximum: image_squashing_time + max(duration_L0, duration_L1, duration_docs) ~1h10mins
• no duration change to the default PR pipeline, other pipelines have smaller duration

Type of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Refactor
• Documentation update
• Other (please describe):

CI Pipeline Configuration

Configure CI behavior by applying the relevant labels:

• SKIP_CI - Skip all continuous integration tests
• INCLUDE_NOTEBOOKS_TESTS - Execute notebook validation tests in pytest
• INCLUDE_SLOW_TESTS - Execute tests labelled as slow in pytest for extensive testing

Note

By default, the notebooks validation tests are skipped unless explicitly enabled.

Authorizing CI Runs

We use copy-pr-bot to manage authorization of CI
runs on NVIDIA's compute resources.

• If a pull request is opened by a trusted user and contains only trusted changes, the pull request's code will
  automatically be copied to a pull-request/ prefixed branch in the source repository (e.g. pull-request/123)
• If a pull request is opened by an untrusted user or contains untrusted changes, an NVIDIA org member must leave an
  /ok to test comment on the pull request to trigger CI. This will need to be done for each new commit.

Usage

TODO: Add code snippet

Pre-submit Checklist

• I have tested these changes locally
• I have updated the documentation accordingly
• I have added/updated tests as needed
• All existing tests pass successfully

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[dorotat-nv]

/ok to test

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 83.58%. Comparing base (72f9c93) to head (54afe3b).

✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #768   +/-   ##
=======================================
  Coverage   83.58%   83.58%           
=======================================
  Files         129      129           
  Lines        8116     8116           
=======================================
  Hits         6784     6784           
  Misses       1332     1332           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[dorotat-nv]

/ok to test

[trvachov]

LGTM but @dorotat-nv can you please draw out what kinds of tests we have in CI now (slow/notslow, notebooks), and when each one runs? in NVKS? In Jenkins? etc? Just a single slide.

[dorotat-nv]

LGTM but @dorotat-nv can you please draw out what kinds of tests we have in CI now (slow/notslow, notebooks), and when each one runs? in NVKS? In Jenkins? etc? Just a single slide.

@trvachov , added charts with current workflow in CI and a workflow after parallelization

[dorotat-nv]

/ok to test

[dorotat-nv]

/ok to test

[dorotat-nv]

/ok to test

[dorotat-nv]

/ok to test