Add InstanceNorm3dNVFuser support #4194

yiheng-wang-nv · 2022-04-28T07:29:11Z

Signed-off-by: Yiheng Wang vennw@nvidia.com

Fixes #4085 .

Description

This PR enhances the layer factory via adding the InstanceNorm3dNVFuser, a faster version of nn.InstanceNorm3d implemented by apex.

The consistency between InstanceNorm3dNVFuser and nn.InstanceNorm3d has been validated via using DynUNet.

Status

Ready

Types of changes

Non-breaking change (fix or new feature that would not break existing functionality).
Breaking change (fix or new feature that would cause existing functionality to change).
New tests added to cover the changes.
Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
Quick tests passed locally by running ./runtests.sh --quick --unittests --disttests.
In-line docstrings updated.
Documentation updated, tested make html command in the docs/ folder.

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

for more information, see https://pre-commit.ci

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

…wang-nv/MONAI into 4085-add-instance_norm_3dnvfuser

for more information, see https://pre-commit.ci

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

…wang-nv/MONAI into 4085-add-instance_norm_3dnvfuser

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

monai/networks/layers/factories.py

tests/test_dynunet.py

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

wyli

Thanks, it looks good to me.

Nic-Ma

Hi @yiheng-wang-nv ,

Thanks for the quick update, looks good to me now.
Could you please help add some unit tests to ensure the InstanceNorm3dNVFuser is compatible with TorchScript exporting?

Thanks in advance.

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

yiheng-wang-nv · 2022-05-05T03:10:14Z

Hi @yiheng-wang-nv ,

Thanks for the quick update, looks good to me now. Could you please help add some unit tests to ensure the InstanceNorm3dNVFuser is compatible with TorchScript exporting?

Thanks in advance.

Hi @Nic-Ma and @wyli , unfortunately, this layer uses the customized autograd function, which is not supported in TorchScript.
The error is like:

Python builtin <built-in method apply of FunctionMeta object at 0x558ca33eca90> is currently not supported in Torchscript

(please click this link for more details: pytorch/pytorch#22329)
I updated the doc-strings about it, what do you think? Thanks!

Nic-Ma · 2022-05-05T03:14:17Z

/build

wyli · 2022-05-05T04:37:54Z

/build

* add swin_unetr model (#4074) * add swin_unetr model * 4217 Update PyTorch docker to 22.04 (#4218) * [DLMED] update to 22.04 Signed-off-by: Nic Ma <nma@nvidia.com> * fixes unit test tests.test_lr_finder Signed-off-by: Wenqi Li <wenqil@nvidia.com> * test new_empty Signed-off-by: Wenqi Li <wenqil@nvidia.com> Co-authored-by: Wenqi Li <wenqil@nvidia.com> Co-authored-by: Wenqi Li <831580+wyli@users.noreply.github.com> * Add InstanceNorm3dNVFuser support (#4194) * implement the base class Signed-off-by: Yiheng Wang <vennw@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add unittest Signed-off-by: Yiheng Wang <vennw@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * autofix Signed-off-by: Yiheng Wang <vennw@nvidia.com> * switch to call apex directly Signed-off-by: Yiheng Wang <vennw@nvidia.com> * uncomment unittest Signed-off-by: Yiheng Wang <vennw@nvidia.com> * add apex install link in docstring Signed-off-by: Yiheng Wang <vennw@nvidia.com> * add channels_last_3d test case Signed-off-by: Yiheng Wang <vennw@nvidia.com> * rewrite types Signed-off-by: Yiheng Wang <vennw@nvidia.com> * change types Signed-off-by: Yiheng Wang <vennw@nvidia.com> * add docstrings Signed-off-by: Yiheng Wang <vennw@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Wenqi Li <831580+wyli@users.noreply.github.com> Co-authored-by: Ali Hatamizadeh <ahatamizadeh@nvidia.com> Co-authored-by: Nic Ma <nma@nvidia.com> Co-authored-by: Yiheng Wang <68361391+yiheng-wang-nv@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* implement the base class Signed-off-by: Yiheng Wang <vennw@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add unittest Signed-off-by: Yiheng Wang <vennw@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * autofix Signed-off-by: Yiheng Wang <vennw@nvidia.com> * switch to call apex directly Signed-off-by: Yiheng Wang <vennw@nvidia.com> * uncomment unittest Signed-off-by: Yiheng Wang <vennw@nvidia.com> * add apex install link in docstring Signed-off-by: Yiheng Wang <vennw@nvidia.com> * add channels_last_3d test case Signed-off-by: Yiheng Wang <vennw@nvidia.com> * rewrite types Signed-off-by: Yiheng Wang <vennw@nvidia.com> * change types Signed-off-by: Yiheng Wang <vennw@nvidia.com> * add docstrings Signed-off-by: Yiheng Wang <vennw@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Wenqi Li <831580+wyli@users.noreply.github.com>

* add swin_unetr model (#4074) * add swin_unetr model Signed-off-by: kbressem <kenobressem@gmail.com> * 4217 Update PyTorch docker to 22.04 (#4218) * [DLMED] update to 22.04 Signed-off-by: Nic Ma <nma@nvidia.com> * fixes unit test tests.test_lr_finder Signed-off-by: Wenqi Li <wenqil@nvidia.com> * test new_empty Signed-off-by: Wenqi Li <wenqil@nvidia.com> Co-authored-by: Wenqi Li <wenqil@nvidia.com> Co-authored-by: Wenqi Li <831580+wyli@users.noreply.github.com> Signed-off-by: kbressem <kenobressem@gmail.com> * Add InstanceNorm3dNVFuser support (#4194) * implement the base class Signed-off-by: Yiheng Wang <vennw@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add unittest Signed-off-by: Yiheng Wang <vennw@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * autofix Signed-off-by: Yiheng Wang <vennw@nvidia.com> * switch to call apex directly Signed-off-by: Yiheng Wang <vennw@nvidia.com> * uncomment unittest Signed-off-by: Yiheng Wang <vennw@nvidia.com> * add apex install link in docstring Signed-off-by: Yiheng Wang <vennw@nvidia.com> * add channels_last_3d test case Signed-off-by: Yiheng Wang <vennw@nvidia.com> * rewrite types Signed-off-by: Yiheng Wang <vennw@nvidia.com> * change types Signed-off-by: Yiheng Wang <vennw@nvidia.com> * add docstrings Signed-off-by: Yiheng Wang <vennw@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Wenqi Li <831580+wyli@users.noreply.github.com> Signed-off-by: kbressem <kenobressem@gmail.com> * Update dice.py (#4234) * Update dice.py reduce redundant operations in DiceFocalLoss, initially caused oom Signed-off-by: Ryan Clanton <55164720+ryancinsight@users.noreply.github.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: Ryan Clanton <55164720+ryancinsight@users.noreply.github.com> * [MONAI] python code formatting Signed-off-by: monai-bot <monai.miccai2019@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: monai-bot <monai.miccai2019@gmail.com> Signed-off-by: kbressem <kenobressem@gmail.com> * Bug fix and improvement in WSI (#4216) * Make all transforms optional Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * Update wsireader tests Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * Remove optional from PersistentDataset and its derivatives Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * Add unittests for cache without transform Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * Add default replace_rate Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * Add default value Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * Set default replace_rate to 0.1 Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * Update metadata to include path Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * Adds SmartCachePatchWSIDataset Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * Add unittests for SmartCachePatchWSIDataset Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * Update references Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * Update docs Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * Remove smart cache Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * Remove unused imports Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * Add path metadata for OpenSlide Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * Update metadata to be unified across different backends Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * Update wsi metadata for multi wsi objects Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> * Add unittests for wsi metadata Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com> Signed-off-by: kbressem <kenobressem@gmail.com> * Replace module (#4245) * replace modules Signed-off-by: Richard Brown <33289025+rijobro@users.noreply.github.com> * fix Signed-off-by: Richard Brown <33289025+rijobro@users.noreply.github.com> * replace_module -> replace_modules Signed-off-by: Richard Brown <33289025+rijobro@users.noreply.github.com> * fix Signed-off-by: Richard Brown <33289025+rijobro@users.noreply.github.com> Signed-off-by: kbressem <kenobressem@gmail.com> * Add GaussianSmooth as antialiasing filter in Resize (#4249) Signed-off-by: Can Zhao <canz@nvidia.com> Signed-off-by: kbressem <kenobressem@gmail.com> * 4235 fix 2204 nvfuser issue (#4241) * reproduce issue Signed-off-by: Yiheng Wang <vennw@nvidia.com> * remove 22.01 02 Signed-off-by: Yiheng Wang <vennw@nvidia.com> * remove other workflows Signed-off-by: Yiheng Wang <vennw@nvidia.com> * run on pull request Signed-off-by: Yiheng Wang <vennw@nvidia.com> * remove sleep Signed-off-by: Yiheng Wang <vennw@nvidia.com> * test single layer forward Signed-off-by: Yiheng Wang <vennw@nvidia.com> * add has_nvfuser Signed-off-by: Yiheng Wang <vennw@nvidia.com> * add check within factory Signed-off-by: Yiheng Wang <vennw@nvidia.com> * revert to original cron.yml Signed-off-by: Yiheng Wang <vennw@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix old pt issue Signed-off-by: Yiheng Wang <vennw@nvidia.com> * change to return directly if no cuda Signed-off-by: Yiheng Wang <vennw@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: kbressem <kenobressem@gmail.com> * Update to Bundle Specifiation (#4250) * Update to bundle specifiation Signed-off-by: Eric Kerfoot <eric.kerfoot@kcl.ac.uk> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Adding description in spec discussing the saved Torchscript object's file storage behaviour, and tweaking ckpt_export to add .json extension Signed-off-by: Eric Kerfoot <eric.kerfoot@kcl.ac.uk> * Annotating optional bundle files Signed-off-by: Eric Kerfoot <eric.kerfoot@kcl.ac.uk> * Adjusted ckpt_export test Signed-off-by: Eric Kerfoot <eric.kerfoot@kcl.ac.uk> * Fix Signed-off-by: Eric Kerfoot <eric.kerfoot@kcl.ac.uk> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: kbressem <kenobressem@gmail.com> * Implement NrrdReader and NrrdImage classes Signed-off-by: kbressem <kenobressem@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: kbressem <kenobressem@gmail.com> * run auto style fixes on image_reader.py Signed-off-by: kbressem <kenobressem@gmail.com> * add NrrdReader to monai/data/__init__.py Signed-off-by: kbressem <kenobressem@gmail.com> * Change the way spatial information is handled in NrrdReader Signed-off-by: kbressem <kenobressem@gmail.com> * add tests for NrrdReader Signed-off-by: kbressem <kenobressem@gmail.com> * Add NrrdReader to list of possible readers for LoadImage Signed-off-by: kbressem <kenobressem@gmail.com> * autofix formating Signed-off-by: kbressem <kenobressem@gmail.com> * autofix formating Signed-off-by: kbressem <kenobressem@gmail.com> * change NrrdImage class to namedtuple and make flake8 happy Signed-off-by: kbressem <kenobressem@gmail.com> * Add pynrrd to requirements Signed-off-by: kbressem <kenobressem@gmail.com> * correct typing for namedtumple make flake8 happy Signed-off-by: kbressem <kenobressem@gmail.com> * Add pynrrd info to `get_optional_config_values` Changed NrrdImage to dataclass Signed-off-by: kbressem <kenobressem@gmail.com> * exclude test_nrrd_reader.py from min tests Signed-off-by: kbressem <kenobressem@gmail.com> * add pynrrd to config files Signed-off-by: kbressem <kenobressem@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Change the way space is handled in the header. Now, if space is not in header, it is assumed to be LPS and converted to RAS. If space is defined and not LPS, nothing is done to prevent wrong conversions. Signed-off-by: kbressem <kenobressem@gmail.com> * add `TestLoadSaveNrrd` where it is tested if a nrrd file, created by ITKWriter can be loaded again. 2D and 3D files with no channels are tested Signed-off-by: kbressem <kenobressem@gmail.com> * autofix format Co-authored-by: kbressem <kenobressem@charite.de>

yiheng-wang-nv and others added 9 commits April 28, 2022 15:27

implement the base class

dbb5eef

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

0163f39

for more information, see https://pre-commit.ci

add unittest

09e8559

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

Merge branch '4085-add-instance_norm_3dnvfuser' of github.com:yiheng-…

fdec987

…wang-nv/MONAI into 4085-add-instance_norm_3dnvfuser

[pre-commit.ci] auto fixes from pre-commit.com hooks

9443d86

for more information, see https://pre-commit.ci

autofix

caade5e

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

Merge branch 'dev' into 4085-add-instance_norm_3dnvfuser

d8aa39d

Merge branch '4085-add-instance_norm_3dnvfuser' of github.com:yiheng-…

91a89bb

…wang-nv/MONAI into 4085-add-instance_norm_3dnvfuser

switch to call apex directly

8d69e51

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

yiheng-wang-nv marked this pull request as ready for review April 28, 2022 13:32

yiheng-wang-nv changed the title ~~[WIP] Implement InstanceNorm3dNVFuser~~ Add InstanceNorm3dNVFuser support Apr 28, 2022

yiheng-wang-nv requested review from Nic-Ma and wyli April 28, 2022 13:33

uncomment unittest

332eb6a

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

Nic-Ma reviewed Apr 29, 2022

View reviewed changes

monai/networks/layers/factories.py Show resolved Hide resolved

monai/networks/layers/factories.py Show resolved Hide resolved

monai/networks/layers/factories.py Show resolved Hide resolved

tests/test_dynunet.py Outdated Show resolved Hide resolved

yiheng-wang-nv and others added 5 commits April 29, 2022 15:37

add apex install link in docstring

b83b453

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

add channels_last_3d test case

d693049

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

rewrite types

e33a757

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

change types

3595cb1

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

Merge branch 'dev' into 4085-add-instance_norm_3dnvfuser

763cce3

wyli approved these changes May 2, 2022

View reviewed changes

Nic-Ma approved these changes May 2, 2022

View reviewed changes

yiheng-wang-nv and others added 2 commits May 5, 2022 10:43

Merge branch 'dev' into 4085-add-instance_norm_3dnvfuser

8238fed

add docstrings

b985cb8

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

Nic-Ma approved these changes May 5, 2022

View reviewed changes

Nic-Ma enabled auto-merge (squash) May 5, 2022 03:13

Nic-Ma disabled auto-merge May 5, 2022 03:35

wyli enabled auto-merge (squash) May 5, 2022 04:35

Merge branch 'dev' into 4085-add-instance_norm_3dnvfuser

cd8f34a

wyli merged commit 66f3280 into Project-MONAI:dev May 5, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add InstanceNorm3dNVFuser support #4194

Add InstanceNorm3dNVFuser support #4194

Uh oh!

yiheng-wang-nv commented Apr 28, 2022 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wyli left a comment

Uh oh!

Nic-Ma left a comment

Uh oh!

yiheng-wang-nv commented May 5, 2022

Uh oh!

Nic-Ma commented May 5, 2022

Uh oh!

wyli commented May 5, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Add InstanceNorm3dNVFuser support #4194

Add InstanceNorm3dNVFuser support #4194

Uh oh!

Conversation

yiheng-wang-nv commented Apr 28, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Status

Types of changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wyli left a comment

Choose a reason for hiding this comment

Uh oh!

Nic-Ma left a comment

Choose a reason for hiding this comment

Uh oh!

yiheng-wang-nv commented May 5, 2022

Uh oh!

Nic-Ma commented May 5, 2022

Uh oh!

wyli commented May 5, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

yiheng-wang-nv commented Apr 28, 2022 •

edited

Loading