670 add bundle example for multi-gpu training #673

Nic-Ma · 2022-04-25T08:34:45Z

Fixes #670 .

Description

This PR added multi-gpu training example in the spleen bundle.

Status

Ready

Checks

Notebook runs automatically ./runner [-p <regex_pattern>]

Signed-off-by: Nic Ma <nma@nvidia.com>

review-notebook-app · 2022-04-26T16:35:01Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

Nic-Ma · 2022-04-26T16:35:28Z

Depends on Project-MONAI/MONAI#4168.

Thanks.

Signed-off-by: Nic Ma <nma@nvidia.com>

Nic-Ma · 2022-04-27T11:45:38Z

Hi @wyli @ericspod ,

I added 2 methods to support multi-gpu training in this PR:
(1) Newly defined config train_multi_gpu.json which contains all the logic.
(2) A much simpler config multi_gpu_train.json which only overrides the existing train.json config.

Which method do you prefer?

Thanks in advance.

ericspod · 2022-04-27T20:47:40Z

I think providing the override and an example of its usage would be a better idea. The override mechanism is quite powerful but I imagine for beginners it's not clear how it's meant to be used.

Signed-off-by: Nic Ma <nma@nvidia.com>

Nic-Ma · 2022-04-28T03:05:52Z

Depends on: Project-MONAI/MONAI#4192.

Thanks.

Nic-Ma · 2022-04-28T03:10:26Z

Hi @ericspod @wyli ,

Thanks for your review and comments.
I updated the PR to make it clearer with the multiple configs method instead of the "implicit" args_file method:
--config_file [configs/train.json, configs/multi_gpu_train.json]
It depends on this minor PR: Project-MONAI/MONAI#4192.
Could you please help review it again?

Thanks in advance.

Nic-Ma · 2022-04-28T11:19:20Z

I will also add the evaluation config in this PR soon.

Thanks.

Signed-off-by: Nic Ma <nma@nvidia.com>

for more information, see https://pre-commit.ci

ericspod · 2022-04-28T11:37:35Z

I approved and committed Project-MONAI/MONAI#4192

Signed-off-by: Nic Ma <nma@nvidia.com>

Nic-Ma · 2022-04-28T11:43:15Z

I approved and committed Project-MONAI/MONAI#4192

Thanks @ericspod ,

@wyli @ericspod Then this PR is ready for review now.
I added multi-gpu training, evaluate configs based on override, and also simplified the inference config as we have completed the configs of this bundle now.

Thanks.

modules/bundles/get_started.ipynb

ericspod · 2022-04-28T12:15:55Z

I tried reviewing through ReviewNB, I'm not sure it's as coherent as I'd hoped.

When you have a structure like:

"train":{
...some defs...
}

The "train" value is interpreted as a dict by the parser correct? So in places where you have "train#loss" or something like it that sets that entry in that dictionary. I think that needs to be explained in a bit more detail to make sense to the reader and to understand what to use it for.

ericspod

I had commented on a few changes to text and a question on references to members of dictionaries (I think) like "validate#postprocessing" that we should explain at little more.

Signed-off-by: Nic Ma <nma@nvidia.com>

Nic-Ma · 2022-04-28T14:52:11Z

I tried reviewing through ReviewNB, I'm not sure it's as coherent as I'd hoped.

When you have a structure like:
"train":{
...some defs...
}
The "train" value is interpreted as a dict by the parser correct? So in places where you have "train#loss" or something like it that sets that entry in that dictionary. I think that needs to be explained in a bit more detail to make sense to the reader and to understand what to use it for.

Good point, I added more description for it in the dataset and dataloader section.

Thanks.

Nic-Ma · 2022-04-28T14:53:18Z

Hi @ericspod ,

Thanks for your review and comments, I updated the PR according to them.
Could you please help review again?

Thanks in advance.

ericspod

Looks good!

* [DLMED] draft config Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] update for test Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] update based on enhancement Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] update tutorial Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] simplify to override Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] update according to comments Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] remove test file Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] add evaluation config Signed-off-by: Nic Ma <nma@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [DLMED] simplify inference Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] update according to comments Signed-off-by: Nic Ma <nma@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

Nic-Ma added 4 commits April 25, 2022 16:30

[DLMED] draft config

e8f8b26

Signed-off-by: Nic Ma <nma@nvidia.com>

[DLMED] update for test

43e9c9f

Signed-off-by: Nic Ma <nma@nvidia.com>

[DLMED] update based on enhancement

0b30d31

Signed-off-by: Nic Ma <nma@nvidia.com>

[DLMED] update tutorial

d80a4da

Signed-off-by: Nic Ma <nma@nvidia.com>

Nic-Ma changed the title ~~[WIP] 670 add bundle example for multi-gpu training~~ 670 add bundle example for multi-gpu training Apr 26, 2022

Nic-Ma marked this pull request as ready for review April 26, 2022 16:35

Nic-Ma requested review from ericspod, rijobro and wyli April 26, 2022 16:36

Nic-Ma changed the title ~~670 add bundle example for multi-gpu training~~ [WIP] 670 add bundle example for multi-gpu training Apr 27, 2022

[DLMED] simplify to override

a7e0ed1

Signed-off-by: Nic Ma <nma@nvidia.com>

Nic-Ma force-pushed the 670-support-multigpu branch from 67d1d33 to a7e0ed1 Compare April 27, 2022 11:40

Merge branch 'master' into 670-support-multigpu

0efa61d

Nic-Ma added 2 commits April 28, 2022 10:56

[DLMED] update according to comments

f9781b2

Signed-off-by: Nic Ma <nma@nvidia.com>

[DLMED] remove test file

7734d1a

Signed-off-by: Nic Ma <nma@nvidia.com>

Nic-Ma changed the title ~~[WIP] 670 add bundle example for multi-gpu training~~ 670 add bundle example for multi-gpu training Apr 28, 2022

Nic-Ma changed the title ~~670 add bundle example for multi-gpu training~~ [WIP] 670 add bundle example for multi-gpu training Apr 28, 2022

Nic-Ma and others added 2 commits April 28, 2022 19:26

[DLMED] add evaluation config

de93f47

Signed-off-by: Nic Ma <nma@nvidia.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

3c7a0a5

for more information, see https://pre-commit.ci

[DLMED] simplify inference

69fd508

Signed-off-by: Nic Ma <nma@nvidia.com>

Nic-Ma changed the title ~~[WIP] 670 add bundle example for multi-gpu training~~ 670 add bundle example for multi-gpu training Apr 28, 2022

ericspod reviewed Apr 28, 2022

View reviewed changes

modules/bundles/get_started.ipynb Show resolved Hide resolved

modules/bundles/get_started.ipynb Show resolved Hide resolved

modules/bundles/get_started.ipynb Show resolved Hide resolved

modules/bundles/get_started.ipynb Show resolved Hide resolved

ericspod reviewed Apr 28, 2022

View reviewed changes

[DLMED] update according to comments

453aca9

Signed-off-by: Nic Ma <nma@nvidia.com>

ericspod approved these changes Apr 29, 2022

View reviewed changes

wyli merged commit b66354a into master Apr 29, 2022

wyli deleted the 670-support-multigpu branch April 29, 2022 11:52

670 add bundle example for multi-gpu training #673

670 add bundle example for multi-gpu training #673

Uh oh!

Conversation

Nic-Ma commented Apr 25, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Status

Checks

Uh oh!

review-notebook-app bot commented Apr 26, 2022

Uh oh!

Nic-Ma commented Apr 26, 2022

Uh oh!

Nic-Ma commented Apr 27, 2022

Uh oh!

ericspod commented Apr 27, 2022

Uh oh!

Nic-Ma commented Apr 28, 2022

Uh oh!

Nic-Ma commented Apr 28, 2022

Uh oh!

Nic-Ma commented Apr 28, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ericspod commented Apr 28, 2022

Uh oh!

Nic-Ma commented Apr 28, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ericspod commented Apr 28, 2022

Uh oh!

ericspod left a comment

Choose a reason for hiding this comment

Uh oh!

Nic-Ma commented Apr 28, 2022

Uh oh!

Nic-Ma commented Apr 28, 2022

Uh oh!

ericspod left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Nic-Ma commented Apr 25, 2022 •

edited

Loading

Nic-Ma commented Apr 28, 2022 •

edited

Loading

Nic-Ma commented Apr 28, 2022 •

edited

Loading