4095 Add bundle download #4114

yiheng-wang-nv · 2022-04-12T15:08:36Z

Fixes #4095 .

This PR is used to add download and load features for monai.bundle, and users can download bundle zip file from github releases or url. The load function is able to load model weights, or TorchScript module from bundle.

Status

Ready

Types of changes

Non-breaking change (fix or new feature that would not break existing functionality).
Breaking change (fix or new feature that would cause existing functionality to change).
New tests added to cover the changes.
Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
Quick tests passed locally by running ./runtests.sh --quick --unittests --disttests.
In-line docstrings updated.
Documentation updated, tested make html command in the docs/ folder.

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

for more information, see https://pre-commit.ci

monai/bundle/scripts.py

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

… into add-bundle-download Signed-off-by: Yiheng Wang <vennw@nvidia.com>

for more information, see https://pre-commit.ci

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

… into add-bundle-download

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

yiheng-wang-nv · 2022-04-19T09:33:27Z

Hi @Nic-Ma @ericspod @wyli , to do the unit test, I added a new release in: https://github.com/Project-MONAI/MONAI-extra-test-data/releases/tag/test_bundle
I'm not sure if it is a good way, and do you have any suggestions? Thanks!

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

ericspod · 2022-04-19T16:10:31Z

If we do it's fine but doesn't really illustrate the correct configuration, you could do the release a second time with just the zip file with things in the right places.

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

Nic-Ma · 2022-04-20T12:05:12Z

Hi @ericspod ,

@yiheng-wang-nv and I tried to revisit this "subdirectories" issue today, how about let's simply the first version API:

Put the bundle in a zip file with well folder structures, then upload it to the Github release of some repo. For the CI test bundle, we can just put it to: https://github.com/Project-MONAI/MONAI-extra-test-data/releases/tag/0.8.1.
The monai.bundle download API only supports downloading the whole bundle zip directly, similar to our existing MMAR API: https://github.com/Project-MONAI/MONAI/blob/dev/monai/apps/mmars/mmars.py#L97.
I imagine we will show a list of bundles and the corresponding description in https://monai.io/, so we don't need to consider the files browser of a bundle in Github release directly.
The load API still supports to automatically download a bundle and load weights or torchscript model from it.

What do you think?

Thanks in advance.

ericspod · 2022-04-20T13:37:02Z

Hi @ericspod ,

@yiheng-wang-nv and I tried to revisit this "subdirectories" issue today, how about let's simply the first version API:

1. Put the bundle in a `zip` file with well folder structures, then upload it to the Github release of some repo. For the CI test bundle, we can just put it to: https://github.com/Project-MONAI/MONAI-extra-test-data/releases/tag/0.8.1.

2. The `monai.bundle download` API only supports downloading the whole bundle zip directly, similar to our existing MMAR API: https://github.com/Project-MONAI/MONAI/blob/dev/monai/apps/mmars/mmars.py#L97.

3. I imagine we will show a list of bundles and the corresponding description in https://monai.io/, so we don't need to consider the files browser of a bundle in Github release directly.

4. The `load` API still supports to automatically download a bundle and load weights or torchscript model from it.

What do you think?

Thanks in advance.

OK sounds good let's go with this and address any issues we might come up with in subsequent revisions. This looks good for the first version.

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

Nic-Ma

Thanks for the quick update.
Looks good to me except for some minor suggestions.

Thanks.

monai/bundle/scripts.py

Signed-off-by: Nic Ma <nma@nvidia.com>

monai/bundle/scripts.py

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

… into add-bundle-download

Nic-Ma

Thanks for the quick update.
Looks good to me except for some minor comments inline.

Thanks.

monai/bundle/scripts.py

tests/test_bundle_download.py

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

… into add-bundle-download

monai/bundle/scripts.py

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

Nic-Ma · 2022-04-22T14:56:00Z

/black

Nic-Ma · 2022-04-22T14:56:07Z

/build

* 4095 Add bundle download (#4114) * draft download Signed-off-by: Yiheng Wang <vennw@nvidia.com> * update bundle download Signed-off-by: Yiheng Wang <vennw@nvidia.com> * add url and load Signed-off-by: Yiheng Wang <vennw@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * rename args and remove a few places Signed-off-by: Yiheng Wang <vennw@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix flake8 issue Signed-off-by: Yiheng Wang <vennw@nvidia.com> * enhance with reviews Signed-off-by: Yiheng Wang <vennw@nvidia.com> * add instantiate for load Signed-off-by: Yiheng Wang <vennw@nvidia.com> * fix black error Signed-off-by: Yiheng Wang <vennw@nvidia.com> * add unittest Signed-off-by: Yiheng Wang <vennw@nvidia.com> * add load to docs Signed-off-by: Yiheng Wang <vennw@nvidia.com> * add skip Signed-off-by: Yiheng Wang <vennw@nvidia.com> * add schemaerror Signed-off-by: Yiheng Wang <vennw@nvidia.com> * fix partial places Signed-off-by: Yiheng Wang <vennw@nvidia.com> * download zip bundle Signed-off-by: Yiheng Wang <vennw@nvidia.com> * [DLMED] restore Exception for test Signed-off-by: Nic Ma <nma@nvidia.com> * update ts features Signed-off-by: Yiheng Wang <vennw@nvidia.com> * add config_files test case Signed-off-by: Yiheng Wang <vennw@nvidia.com> * enhance docstring example for args_file Signed-off-by: Yiheng Wang <vennw@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Nic Ma <nma@nvidia.com> * Disable pylint error and fix CI tests of new tifffile (#4162) * workaround Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] fix tifffile issue Signed-off-by: Nic Ma <nma@nvidia.com> * Fixed an error in DiNTS model implementation and enabled act and norm layer options (#4157) * fixed a bug Signed-off-by: dongy <dongy@nvidia.com> * autofix Signed-off-by: dongy <dongy@nvidia.com> * update test case Signed-off-by: dongy <dongy@nvidia.com> Co-authored-by: dongy <dongy@nvidia.com> * Split transform (#4153) * Redesign whole slide image reading (#4107) * Redesign BaseWSIReader, WSIReader, CuCIMWSIReader Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * Add unittests for WSIReader Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * Add image mode for output validation Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * Update docs Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * Update references to new WSIReader Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * Remove legacy WSIReader Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * Update unittests Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * Update docs Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * sort imports Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * Clean up imports Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * Update docstrings Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * Update docs and docstrings Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix a typo Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * Remove redundant checking Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * Update read and other methods Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * Update wsireader to support multi image and update docstrings Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * Make workaround for CuImage objects Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * Add unittests for multi image reading Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * Update a note about cucim Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * Update type hints and docstrings Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * Implement Split transform Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * Add unittests Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * Update formatting Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * Implement SplitDict Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * Add unittests for SplitDict Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * Add docs Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * Remove images from docs Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * Address all comments Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * Add example and size check Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * Update docs Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * Revert references to new wsireader Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * Add missing comma Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * fix bundle download test issue (#4169) Signed-off-by: Yiheng Wang <vennw@nvidia.com> * 4094 Enhance `ckpt_export` to save config files (#4159) * [DLMED] enhance checkpoint export Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] update according to comments Signed-off-by: Nic Ma <nma@nvidia.com> * Move RGB/RGBA checks to base class (#4171) Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> Co-authored-by: Nic Ma <nma@nvidia.com> * [CICD] To support temp dgx runner (#4175) * Support new temp dgx runner Signed-off-by: Peixin Li <pxli@nyu.edu> * atol 1e-5 Signed-off-by: Wenqi Li <wenqil@nvidia.com> Co-authored-by: Wenqi Li <wenqil@nvidia.com> * Test fix for AMP kwargs (#4178) Signed-off-by: Eric Kerfoot <eric.kerfoot@kcl.ac.uk> Co-authored-by: Yiheng Wang <68361391+yiheng-wang-nv@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Nic Ma <nma@nvidia.com> Co-authored-by: dongyang0122 <don.yang.mech@gmail.com> Co-authored-by: dongy <dongy@nvidia.com> Co-authored-by: Behrooz <3968947+drbeh@users.noreply.github.com> Co-authored-by: Peixin <pxli@nyu.edu> Co-authored-by: Eric Kerfoot <17726042+ericspod@users.noreply.github.com>

* draft download Signed-off-by: Yiheng Wang <vennw@nvidia.com> * update bundle download Signed-off-by: Yiheng Wang <vennw@nvidia.com> * add url and load Signed-off-by: Yiheng Wang <vennw@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * rename args and remove a few places Signed-off-by: Yiheng Wang <vennw@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix flake8 issue Signed-off-by: Yiheng Wang <vennw@nvidia.com> * enhance with reviews Signed-off-by: Yiheng Wang <vennw@nvidia.com> * add instantiate for load Signed-off-by: Yiheng Wang <vennw@nvidia.com> * fix black error Signed-off-by: Yiheng Wang <vennw@nvidia.com> * add unittest Signed-off-by: Yiheng Wang <vennw@nvidia.com> * add load to docs Signed-off-by: Yiheng Wang <vennw@nvidia.com> * add skip Signed-off-by: Yiheng Wang <vennw@nvidia.com> * add schemaerror Signed-off-by: Yiheng Wang <vennw@nvidia.com> * fix partial places Signed-off-by: Yiheng Wang <vennw@nvidia.com> * download zip bundle Signed-off-by: Yiheng Wang <vennw@nvidia.com> * [DLMED] restore Exception for test Signed-off-by: Nic Ma <nma@nvidia.com> * update ts features Signed-off-by: Yiheng Wang <vennw@nvidia.com> * add config_files test case Signed-off-by: Yiheng Wang <vennw@nvidia.com> * enhance docstring example for args_file Signed-off-by: Yiheng Wang <vennw@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Nic Ma <nma@nvidia.com>

yiheng-wang-nv and others added 4 commits April 8, 2022 23:49

draft download

9a6a80a

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

update bundle download

3b6d2cd

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

add url and load

899dbd6

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

b9fc7dc

for more information, see https://pre-commit.ci

Nic-Ma reviewed Apr 14, 2022

View reviewed changes

monai/bundle/scripts.py Outdated Show resolved Hide resolved

Nic-Ma reviewed Apr 14, 2022

View reviewed changes

monai/bundle/scripts.py Outdated Show resolved Hide resolved

Nic-Ma reviewed Apr 14, 2022

View reviewed changes

monai/bundle/scripts.py Outdated Show resolved Hide resolved

Nic-Ma reviewed Apr 14, 2022

View reviewed changes

monai/bundle/scripts.py Show resolved Hide resolved

Nic-Ma reviewed Apr 14, 2022

View reviewed changes

monai/bundle/scripts.py Outdated Show resolved Hide resolved

Nic-Ma reviewed Apr 14, 2022

View reviewed changes

monai/bundle/scripts.py Show resolved Hide resolved

Nic-Ma reviewed Apr 14, 2022

View reviewed changes

monai/bundle/scripts.py Outdated Show resolved Hide resolved

yiheng-wang-nv and others added 10 commits April 14, 2022 22:37

rename args and remove a few places

981a990

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

Merge branch 'add-bundle-download' of github.com:yiheng-wang-nv/MONAI…

24ea321

… into add-bundle-download Signed-off-by: Yiheng Wang <vennw@nvidia.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

2769ad2

for more information, see https://pre-commit.ci

fix flake8 issue

31e9280

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

Merge branch 'dev' into add-bundle-download

0d11dc9

Merge branch 'add-bundle-download' of github.com:yiheng-wang-nv/MONAI…

d37279a

… into add-bundle-download

enhance with reviews

95227b1

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

Merge branch 'dev' into add-bundle-download

adf187a

add instantiate for load

0e53035

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

fix black error

5cad064

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

yiheng-wang-nv requested review from Nic-Ma, ericspod and wyli April 18, 2022 14:46

add unittest

3ece567

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

yiheng-wang-nv marked this pull request as ready for review April 19, 2022 09:31

yiheng-wang-nv changed the title ~~[WIP] 4095 Add bundle download~~ 4095 Add bundle download Apr 19, 2022

yiheng-wang-nv added 2 commits April 19, 2022 17:34

add load to docs

15dc4b6

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

add skip

12d400a

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

yiheng-wang-nv and others added 2 commits April 20, 2022 19:23

fix partial places

780ecb7

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

Merge branch 'dev' into add-bundle-download

2b936d6

yiheng-wang-nv and others added 2 commits April 21, 2022 14:48

download zip bundle

627ae48

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

Merge branch 'dev' into add-bundle-download

0e6062d

yiheng-wang-nv requested review from Nic-Ma and ericspod April 21, 2022 14:32

Nic-Ma approved these changes Apr 22, 2022

View reviewed changes

monai/bundle/scripts.py Show resolved Hide resolved

monai/bundle/scripts.py Outdated Show resolved Hide resolved

[DLMED] restore Exception for test

415f529

Signed-off-by: Nic Ma <nma@nvidia.com>

Nic-Ma reviewed Apr 22, 2022

View reviewed changes

monai/bundle/scripts.py Outdated Show resolved Hide resolved

yiheng-wang-nv and others added 3 commits April 22, 2022 17:37

update ts features

dc5d7b4

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

Merge branch 'add-bundle-download' of github.com:yiheng-wang-nv/MONAI…

cb906fd

… into add-bundle-download

Merge branch 'dev' into add-bundle-download

8927d2b

Nic-Ma approved these changes Apr 22, 2022

View reviewed changes

monai/bundle/scripts.py Outdated Show resolved Hide resolved

monai/bundle/scripts.py Outdated Show resolved Hide resolved

tests/test_bundle_download.py Show resolved Hide resolved

yiheng-wang-nv added 2 commits April 22, 2022 19:23

add config_files test case

c083b48

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

Merge branch 'add-bundle-download' of github.com:yiheng-wang-nv/MONAI…

284be7e

… into add-bundle-download

ericspod reviewed Apr 22, 2022

View reviewed changes

monai/bundle/scripts.py Outdated Show resolved Hide resolved

ericspod reviewed Apr 22, 2022

View reviewed changes

monai/bundle/scripts.py Show resolved Hide resolved

ericspod approved these changes Apr 22, 2022

View reviewed changes

yiheng-wang-nv and others added 2 commits April 22, 2022 22:53

enhance docstring example for args_file

062a800

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

Merge branch 'dev' into add-bundle-download

c8cc140

Nic-Ma enabled auto-merge (squash) April 22, 2022 14:56

Nic-Ma merged commit e0e2e65 into Project-MONAI:dev Apr 22, 2022

Nic-Ma mentioned this pull request May 11, 2022

Update to Bundle Specifiation #4250

Merged

7 tasks

4095 Add bundle download #4114

4095 Add bundle download #4114

Uh oh!

Conversation

yiheng-wang-nv commented Apr 12, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Status

Types of changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yiheng-wang-nv commented Apr 19, 2022

Uh oh!

ericspod commented Apr 19, 2022

Uh oh!

Nic-Ma commented Apr 20, 2022

Uh oh!

ericspod commented Apr 20, 2022

Uh oh!

Nic-Ma left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Nic-Ma left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Nic-Ma commented Apr 22, 2022

Uh oh!

Nic-Ma commented Apr 22, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

yiheng-wang-nv commented Apr 12, 2022 •

edited

Loading