Replace shell commands with subprocess.run by balvisio · Pull Request #941 · NVIDIA-BioNeMo/bionemo-framework

balvisio · 2025-06-13T23:17:59Z

Description

Instead of using shell commands in Jupyter Notebooks we replace them with calls to subprocess.run so that in case of failure we have better logging of the error.

Type of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Refactor
Documentation update
Other (please describe): Improvement to logging

CI Pipeline Configuration

Configure CI behavior by applying the relevant labels:

SKIP_CI - Skip all continuous integration tests
INCLUDE_NOTEBOOKS_TESTS - Execute notebook validation tests in pytest
INCLUDE_SLOW_TESTS - Execute tests labelled as slow in pytest for extensive testing

Note

By default, the notebooks validation tests are skipped unless explicitly enabled.

Authorizing CI Runs

We use copy-pr-bot to manage authorization of CI
runs on NVIDIA's compute resources.

If a pull request is opened by a trusted user and contains only trusted changes, the pull request's code will
automatically be copied to a pull-request/ prefixed branch in the source repository (e.g. pull-request/123)
If a pull request is opened by an untrusted user or contains untrusted changes, an NVIDIA org member must leave an
/ok to test comment on the pull request to trigger CI. This will need to be done for each new commit.

Usage

TODO: Add code snippet

Pre-submit Checklist

I have tested these changes locally
I have updated the documentation accordingly
I have added/updated tests as needed
All existing tests pass successfully

copy-pr-bot · 2025-06-13T23:18:03Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

balvisio · 2025-06-13T23:18:59Z

/ok to test 5622869

balvisio · 2025-06-23T15:30:38Z

/ok to test f099aec

codecov-commenter · 2025-06-23T16:45:59Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.19%. Comparing base (2d73084) to head (9a4c548).
⚠️ Report is 377 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #941      +/-   ##
==========================================
+ Coverage   84.18%   84.19%   +0.01%     
==========================================
  Files         144      144              
  Lines        9227     9227              
==========================================
+ Hits         7768     7769       +1     
+ Misses       1459     1458       -1

see 1 file with indirect coverage changes

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

dorotat-nv · 2025-06-24T17:42:27Z

/ok to test f099aec

balvisio · 2025-06-24T21:33:04Z

/ok to test 4321f12

balvisio · 2025-06-24T23:07:53Z

/ok to test e4e11f5

jstjohn

Added a suggestion about using something in the middle, shlex.split() on the string so it's copy/pastable to make the string command usable by subprocess.run, rather than specifying the command as a list of strings to subprocess.run, which is harder to copy/paste into the CLI.

jstjohn · 2025-06-24T23:56:40Z

It should handle all of the multi-new-line stuff that comes from making escaped strings (which are nice to copy/paste). For example:

In [4]: import shlex

In [5]: test_cmd = """
   ...: echo \
   ...:   "hi"
   ...: """

In [6]: test_cmd
Out[6]: '\necho   "hi"\n'

In [7]: shlex.split(test_cmd)
Out[7]: ['echo', 'hi']

Signed-off-by: Bruno Alvisio <balvisio@nvidia.com>

balvisio · 2025-06-25T01:41:23Z

/ok to test 9a4c548

### Description  Instead of using shell commands in Jupyter Notebooks we replace them with calls to `subprocess.run` so that in case of failure we have better logging of the error. ### Type of changes  - [ ] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Refactor - [ ] Documentation update - [x] Other (please describe): Improvement to logging ### CI Pipeline Configuration Configure CI behavior by applying the relevant labels: - [SKIP_CI](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#skip_ci) - Skip all continuous integration tests - [INCLUDE_NOTEBOOKS_TESTS](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#include_notebooks_tests) - Execute notebook validation tests in pytest - [INCLUDE_SLOW_TESTS](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#include_slow_tests) - Execute tests labelled as slow in pytest for extensive testing > [!NOTE] > By default, the notebooks validation tests are skipped unless explicitly enabled. #### Authorizing CI Runs We use [copy-pr-bot](https://docs.gha-runners.nvidia.com/apps/copy-pr-bot/#automation) to manage authorization of CI runs on NVIDIA's compute resources. * If a pull request is opened by a trusted user and contains only trusted changes, the pull request's code will automatically be copied to a pull-request/ prefixed branch in the source repository (e.g. pull-request/123) * If a pull request is opened by an untrusted user or contains untrusted changes, an NVIDIA org member must leave an `/ok to test` comment on the pull request to trigger CI. This will need to be done for each new commit. ### Usage  ```python TODO: Add code snippet ``` ### Pre-submit Checklist  - [x] I have tested these changes locally - [x] I have updated the documentation accordingly - [x] I have added/updated tests as needed - [x] All existing tests pass successfully Signed-off-by: Bruno Alvisio <balvisio@nvidia.com> Signed-off-by: Ubuntu <camirr@nvidia.com>

balvisio requested review from DejunL, cspades, dorotat-nv, farhadrgh, guoqing-zhou, jomitchellnv, jstjohn, jwilber, malcolmgreaves, pstjohn, sichu2023, skothenhill-nv and trvachov as code owners June 13, 2025 23:18

balvisio force-pushed the ba/BIONEMO-2086-replace-shell-commands-in-jupyter-nbs branch from 2b90947 to 5622869 Compare June 13, 2025 23:18

balvisio force-pushed the ba/BIONEMO-2086-replace-shell-commands-in-jupyter-nbs branch 10 times, most recently from 5919932 to d106dca Compare June 20, 2025 05:16

balvisio force-pushed the ba/BIONEMO-2086-replace-shell-commands-in-jupyter-nbs branch from d106dca to f099aec Compare June 23, 2025 15:29

dorotat-nv added the INCLUDE_NOTEBOOKS_TESTS label Jun 24, 2025

dorotat-nv approved these changes Jun 24, 2025

View reviewed changes

pstjohn reviewed Jun 24, 2025

View reviewed changes

Comment thread ci/scripts/run_pytest_notebooks.sh

pstjohn reviewed Jun 24, 2025

View reviewed changes

Comment thread sub-packages/bionemo-core/src/bionemo/core/utils/subprocess_utils.py

pstjohn approved these changes Jun 24, 2025

View reviewed changes

balvisio force-pushed the ba/BIONEMO-2086-replace-shell-commands-in-jupyter-nbs branch from f099aec to 4321f12 Compare June 24, 2025 21:32

balvisio enabled auto-merge June 24, 2025 22:58

balvisio disabled auto-merge June 24, 2025 22:58

balvisio requested a review from ohadmo as a code owner June 24, 2025 23:06

balvisio force-pushed the ba/BIONEMO-2086-replace-shell-commands-in-jupyter-nbs branch 2 times, most recently from db8a3aa to 5401a5e Compare June 24, 2025 23:27

jstjohn reviewed Jun 24, 2025

View reviewed changes

Comment thread Dockerfile

jstjohn reviewed Jun 24, 2025

View reviewed changes

Comment thread sub-packages/bionemo-evo2/examples/fine-tuning-tutorial.ipynb Outdated

jstjohn approved these changes Jun 24, 2025

View reviewed changes

balvisio force-pushed the ba/BIONEMO-2086-replace-shell-commands-in-jupyter-nbs branch 2 times, most recently from 4324f93 to d2d2f2c Compare June 25, 2025 01:37

Replace shell commands with subprocess.run

9a4c548

Signed-off-by: Bruno Alvisio <balvisio@nvidia.com>

balvisio force-pushed the ba/BIONEMO-2086-replace-shell-commands-in-jupyter-nbs branch from d2d2f2c to 9a4c548 Compare June 25, 2025 01:40

balvisio added this pull request to the merge queue Jun 25, 2025

Merged via the queue into main with commit ffdadb5 Jun 25, 2025
10 checks passed

balvisio deleted the ba/BIONEMO-2086-replace-shell-commands-in-jupyter-nbs branch June 25, 2025 21:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace shell commands with subprocess.run#941

Replace shell commands with subprocess.run#941
balvisio merged 1 commit into
mainfrom
ba/BIONEMO-2086-replace-shell-commands-in-jupyter-nbs

balvisio commented Jun 13, 2025

Uh oh!

copy-pr-bot Bot commented Jun 13, 2025

Uh oh!

balvisio commented Jun 13, 2025

Uh oh!

balvisio commented Jun 23, 2025

Uh oh!

codecov-commenter commented Jun 23, 2025 •

edited by codecov Bot

Loading

Uh oh!

dorotat-nv commented Jun 24, 2025

Uh oh!

Uh oh!

Uh oh!

balvisio commented Jun 24, 2025

Uh oh!

balvisio commented Jun 24, 2025

Uh oh!

Uh oh!

Uh oh!

jstjohn left a comment

Uh oh!

jstjohn commented Jun 24, 2025

Uh oh!

balvisio commented Jun 25, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

balvisio commented Jun 13, 2025

Description

Type of changes

CI Pipeline Configuration

Authorizing CI Runs

Usage

Pre-submit Checklist

Uh oh!

copy-pr-bot Bot commented Jun 13, 2025

Uh oh!

balvisio commented Jun 13, 2025

Uh oh!

balvisio commented Jun 23, 2025

Uh oh!

codecov-commenter commented Jun 23, 2025 • edited by codecov Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

dorotat-nv commented Jun 24, 2025

Uh oh!

Uh oh!

Uh oh!

balvisio commented Jun 24, 2025

Uh oh!

balvisio commented Jun 24, 2025

Uh oh!

Uh oh!

Uh oh!

jstjohn left a comment

Choose a reason for hiding this comment

Uh oh!

jstjohn commented Jun 24, 2025

Uh oh!

balvisio commented Jun 25, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov-commenter commented Jun 23, 2025 •

edited by codecov Bot

Loading