6834 Add `MetaProterties` and update `load` API #6835

KumoLiu · 2023-08-08T06:43:48Z

Fixes #6834.

Description

See #6835 (comment).

Types of changes

Non-breaking change (fix or new feature that would not break existing functionality).
Breaking change (fix or new feature that would cause existing functionality to change).
New tests added to cover the changes.
Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
Quick tests passed locally by running ./runtests.sh --quick --unittests --disttests.
In-line docstrings updated.
Documentation updated, tested make html command in the docs/ folder.

Signed-off-by: KumoLiu <yunl@nvidia.com>

…rom-bundle

Signed-off-by: KumoLiu <yunl@nvidia.com>

to `bundle_name` and `config_path` Signed-off-by: KumoLiu <yunl@nvidia.com>

KumoLiu · 2023-08-11T07:09:35Z

After all the reviews, I'll add the relevant unittests.

monai/bundle/scripts.py

Signed-off-by: KumoLiu <yunl@nvidia.com>

wyli

thanks, please see some comments inline..

monai/bundle/scripts.py

Signed-off-by: KumoLiu <yunl@nvidia.com>

monai/bundle/scripts.py

Signed-off-by: KumoLiu <yunl@nvidia.com>

Nic-Ma · 2023-08-29T11:20:11Z

I will review this PR tonight.

Thanks.

monai/bundle/scripts.py

monai/bundle/workflows.py

ericspod · 2023-08-29T17:45:51Z

looks good to me, it seems monai.bundle.scripts module now has too many functions, perhaps it could be refactored into multiple modules/files..

Yes maybe we should turn this into a subdirectory now and have separate files for each script. I would leave that to a later PR however and get this one completed sooner as I want to align my bundle tutorials with this.

Signed-off-by: KumoLiu <yunl@nvidia.com>

…rom-bundle

Signed-off-by: KumoLiu <yunl@nvidia.com>

Nic-Ma

Put some minor comments inline, others look good to me.

Thanks.

Signed-off-by: KumoLiu <yunl@nvidia.com>

wyli · 2023-08-30T12:49:05Z

/build

Fixes Project-MONAI#6834. ### Description This PR introduces the idea of `BundleManager`, with `get` as one of the methods, to avoid repeatedly instantiating `ConfigWorkflow`. **usage** ``` bundle = BundleManager("spleen_ct_segmentation") bundle.get("network_data_format") or bundle.get(property="train_preprocessing") ``` ### Types of changes  - [x] Non-breaking change (fix or new feature that would not break existing functionality). - [ ] Breaking change (fix or new feature that would cause existing functionality to change). - [ ] New tests added to cover the changes. - [ ] Integration tests passed locally by running `./runtests.sh -f -u --net --coverage`. - [ ] Quick tests passed locally by running `./runtests.sh --quick --unittests --disttests`. - [ ] In-line docstrings updated. - [ ] Documentation updated, tested `make html` command in the `docs/` folder. --------- Signed-off-by: KumoLiu <yunl@nvidia.com>

Fixes #1503. ### Description After Project-MONAI/MONAI#6835, `load` can directly return an instantiated network that loaded the weights. ### Checks  - [ ] Avoid including large-size files in the PR. - [ ] Clean up long text outputs from code cells in the notebook. - [ ] For security purposes, please check the contents and remove any sensitive info such as user names and private key. - [ ] Ensure (1) hyperlinks and markdown anchors are working (2) use relative paths for tutorial repo files (3) put figure and graphs in the `./figure` folder - [ ] Notebook runs automatically `./runner.sh -t <path to .ipynb file>` --------- Signed-off-by: KumoLiu <yunl@nvidia.com>

Fixes # . ### Description Avoid breaking changes introduced by #6835 - when creating `BundleWorkflow` - when using `load` API, add `return_state_dict` when `model` and `net_name` are both `None`. ### Types of changes  - [x] Non-breaking change (fix or new feature that would not break existing functionality). - [ ] Breaking change (fix or new feature that would cause existing functionality to change). - [ ] New tests added to cover the changes. - [ ] Integration tests passed locally by running `./runtests.sh -f -u --net --coverage`. - [ ] Quick tests passed locally by running `./runtests.sh --quick --unittests --disttests`. - [ ] In-line docstrings updated. - [ ] Documentation updated, tested `make html` command in the `docs/` folder. --------- Signed-off-by: KumoLiu <yunl@nvidia.com> Signed-off-by: YunLiu <55491388+KumoLiu@users.noreply.github.com> Co-authored-by: Eric Kerfoot <17726042+ericspod@users.noreply.github.com>

@ericspod

…6917) Part of #6552. ### Description After PR #6835, we have added `copy_model_args` in the `load` API which can help us update the state_dict flexibly. https://github.com/KumoLiu/MONAI/blob/93a149a611b66153cf804b31a7b36a939e2e593a/monai/bundle/scripts.py#L397 Given this [issue](#6552), we need to be able to filter the model's weights flexibly. In `copy_model_state`, we already have a "mapping" arg, the filter will be more flexible if we can support regular expression in the mapping. This PR mainly added the support for regular expression for "mapping" arg. In the [example](#6552 (comment)) in this [issue](#6552), after this PR, we can do something like: ``` exclude_vars = "encoder.mask_token|encoder.norm.weight|encoder.norm.bias|out.conv.conv.weight|out.conv.conv.bias" mapping={"encoder.layers(.*).0.0.": "swinViT.layers(.*).0."} dst_dict, updated_keys, unchanged_keys = copy_model_state( model, ssl_weights, exclude_vars=exclude_vars, mapping=mapping ) ``` Additionally, based on the comments of Eric [here](#6552 (comment)), I totally agree, we could add a handler to make the pipeline easier to implement, but perhaps this task is no need to set as a "BundleTodo" for MONAIv1.3 but as an enhancement for MONAI near future. What do you think? @ericspod @wyli @Nic-Ma ### Types of changes  - [x] Non-breaking change (fix or new feature that would not break existing functionality). - [ ] Breaking change (fix or new feature that would cause existing functionality to change). - [ ] New tests added to cover the changes. - [ ] Integration tests passed locally by running `./runtests.sh -f -u --net --coverage`. - [ ] Quick tests passed locally by running `./runtests.sh --quick --unittests --disttests`. - [ ] In-line docstrings updated. - [ ] Documentation updated, tested `make html` command in the `docs/` folder. --------- Signed-off-by: KumoLiu <yunl@nvidia.com>

Fixes Project-MONAI#1503. ### Description After Project-MONAI/MONAI#6835, `load` can directly return an instantiated network that loaded the weights. ### Checks  - [ ] Avoid including large-size files in the PR. - [ ] Clean up long text outputs from code cells in the notebook. - [ ] For security purposes, please check the contents and remove any sensitive info such as user names and private key. - [ ] Ensure (1) hyperlinks and markdown anchors are working (2) use relative paths for tutorial repo files (3) put figure and graphs in the `./figure` folder - [ ] Notebook runs automatically `./runner.sh -t <path to .ipynb file>` --------- Signed-off-by: KumoLiu <yunl@nvidia.com>

KumoLiu and others added 8 commits August 8, 2023 14:41

add AutoBundle and from_bundle

faa8dc6

Signed-off-by: KumoLiu <yunl@nvidia.com>

Merge remote-tracking branch 'origin/dev' into from-bundle

efcb485

minor fix

be7aa45

Signed-off-by: KumoLiu <yunl@nvidia.com>

Merge branch 'dev' into from-bundle

e83478a

Merge branch 'dev' into from-bundle

7c26f75

rename to BundleManager

a7bc4e8

Signed-off-by: KumoLiu <yunl@nvidia.com>

Merge branch 'from-bundle' of https://github.com/KumoLiu/MONAI into f…

5dc2f4e

…rom-bundle

support multi configs

b8bc55a

Signed-off-by: KumoLiu <yunl@nvidia.com>

KumoLiu changed the title ~~6834 from_bundle API: support get properties and metadata from bundle~~ 6834 get API: support get properties and metadata from bundle Aug 10, 2023

KumoLiu and others added 3 commits August 10, 2023 23:42

add docstring

eb6a9d4

Signed-off-by: KumoLiu <yunl@nvidia.com>

change bundle_name_or_path

fc677b5

to `bundle_name` and `config_path` Signed-off-by: KumoLiu <yunl@nvidia.com>

Merge branch 'dev' into from-bundle

e312186

KumoLiu marked this pull request as ready for review August 11, 2023 07:08

KumoLiu requested review from ericspod, Nic-Ma, yiheng-wang-nv and vikashg August 11, 2023 07:08

yiheng-wang-nv reviewed Aug 11, 2023

View reviewed changes

monai/bundle/scripts.py Outdated Show resolved Hide resolved

support flexible postfix

8fb5076

Signed-off-by: KumoLiu <yunl@nvidia.com>

KumoLiu requested a review from wyli August 11, 2023 07:52

KumoLiu added 2 commits August 11, 2023 16:00

minor fix

e4d061f

Signed-off-by: KumoLiu <yunl@nvidia.com>

minor fix

d49f3d0

Signed-off-by: KumoLiu <yunl@nvidia.com>

wyli reviewed Aug 11, 2023

View reviewed changes

KumoLiu added 3 commits August 11, 2023 19:20

update based on comments

dc8858b

Signed-off-by: KumoLiu <yunl@nvidia.com>

fix flake8

cc5c2c7

Signed-off-by: KumoLiu <yunl@nvidia.com>

update docstring

52c9f52

Signed-off-by: KumoLiu <yunl@nvidia.com>

ericspod reviewed Aug 11, 2023

View reviewed changes

monai/bundle/scripts.py Outdated Show resolved Hide resolved

ericspod reviewed Aug 11, 2023

View reviewed changes

monai/bundle/scripts.py Outdated Show resolved Hide resolved

update based on comments

5eec914

Signed-off-by: KumoLiu <yunl@nvidia.com>

fix flake8

2789c5f

Signed-off-by: KumoLiu <yunl@nvidia.com>

KumoLiu mentioned this pull request Aug 29, 2023

Add bundle root directory to Python search directories automatically #6910

Merged

7 tasks

KumoLiu requested review from ericspod and Nic-Ma August 29, 2023 10:41

Nic-Ma reviewed Aug 29, 2023

View reviewed changes

monai/bundle/scripts.py Outdated Show resolved Hide resolved

monai/bundle/scripts.py Show resolved Hide resolved

monai/bundle/scripts.py Show resolved Hide resolved

monai/bundle/workflows.py Outdated Show resolved Hide resolved

ericspod reviewed Aug 29, 2023

View reviewed changes

monai/bundle/workflows.py Outdated Show resolved Hide resolved

KumoLiu and others added 5 commits August 30, 2023 10:56

add deprecated_arg for net_name

561a68e

Signed-off-by: KumoLiu <yunl@nvidia.com>

Merge branch 'dev' into from-bundle

9b6bc08

update worklfow to workflow_type

0039860

Signed-off-by: KumoLiu <yunl@nvidia.com>

Merge branch 'from-bundle' of https://github.com/KumoLiu/MONAI into f…

32de5aa

…rom-bundle

fix flake8

134d1ea

Signed-off-by: KumoLiu <yunl@nvidia.com>

Nic-Ma approved these changes Aug 30, 2023

View reviewed changes

KumoLiu and others added 3 commits August 30, 2023 13:24

fix ci

e93309e

Signed-off-by: KumoLiu <yunl@nvidia.com>

address comments

edb8768

Signed-off-by: KumoLiu <yunl@nvidia.com>

Merge branch 'dev' into from-bundle

93a149a

wyli enabled auto-merge (squash) August 30, 2023 12:49

wyli merged commit a4e4894 into Project-MONAI:dev Aug 30, 2023
32 of 37 checks passed

KumoLiu deleted the from-bundle branch August 30, 2023 14:18

This was referenced Aug 31, 2023

Support regular expression in the mapping arg of copy_model_state #6917

Merged

Update "app_integrate_bundle" Project-MONAI/tutorials#1499

Closed

Fix error in bundle notebooks test Project-MONAI/tutorials#1504

Merged

KumoLiu mentioned this pull request Sep 6, 2023

Avoid breaking change in creating BundleWorkflow #6950

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

6834 Add `MetaProterties` and update `load` API #6835

6834 Add `MetaProterties` and update `load` API #6835

KumoLiu commented Aug 8, 2023 •

edited

KumoLiu commented Aug 11, 2023

wyli left a comment

Nic-Ma commented Aug 29, 2023

ericspod commented Aug 29, 2023

Nic-Ma left a comment

wyli commented Aug 30, 2023

6834 Add MetaProterties and update load API #6835

6834 Add MetaProterties and update load API #6835

Conversation

KumoLiu commented Aug 8, 2023 • edited

Description

Types of changes

KumoLiu commented Aug 11, 2023

wyli left a comment

Choose a reason for hiding this comment

Nic-Ma commented Aug 29, 2023

ericspod commented Aug 29, 2023

Nic-Ma left a comment

Choose a reason for hiding this comment

wyli commented Aug 30, 2023

6834 Add `MetaProterties` and update `load` API #6835

6834 Add `MetaProterties` and update `load` API #6835

KumoLiu commented Aug 8, 2023 •

edited