Improve Compose encapsulation #6224

atbenmurray · 2023-03-22T13:57:08Z

Fixes #6223 .

Description

A few sentences describing the changes proposed in this pull request.

Types of changes

Non-breaking change (fix or new feature that would not break existing functionality).
Breaking change (fix or new feature that would cause existing functionality to change).
New tests added to cover the changes.
Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
Quick tests passed locally by running ./runtests.sh --quick --unittests --disttests.
In-line docstrings updated.
Documentation updated, tested make html command in the docs/ folder.

for more information, see https://pre-commit.ci

wyli · 2023-03-22T14:00:55Z

/black

atbenmurray · 2023-03-22T14:01:03Z

@ericspod, @Nic-Ma, @rijobro, @wyli
I'm writing up the test extensions for this, but please take a look at the code changes when you have a moment. I'm essentially trying to eliminate the need for dataset.py to have to be updated by the lazy resampling changes coming up in #5860.

Signed-off-by: monai-bot <monai.miccai2019@gmail.com>

Signed-off-by: Ben Murray <ben.murray@gmail.com>

monai/data/dataset.py

I, Ben Murray <ben.murray@gmail.com>, hereby add my Signed-off-by to this commit: 1759f96 Signed-off-by: Ben Murray <ben.murray@gmail.com>q:

I, Ben Murray <ben.murray@gmail.com>, hereby add my Signed-off-by to this commit: 1759f96 Signed-off-by: Ben Murray <ben.murray@gmail.com>

…o compose_refactor

I, Ben Murray <ben.murray@gmail.com>, hereby add my Signed-off-by to this commit: 10edabd Signed-off-by: Ben Murray <ben.murray@gmail.com>

…edataset_persistent_workers pass Signed-off-by: Ben Murray <ben.murray@gmail.com>

atbenmurray · 2023-03-22T15:22:18Z

Taking the opportunity boost the docstrings a little. It looks as if logging only does anything when the pipeline raises an exception of some kind, so the comment for that parameter should be updated, yes?

Signed-off-by: Ben Murray <ben.murray@gmail.com>

wyli · 2023-03-22T17:00:06Z

I think the refactoring makes sense, let me trigger some integration tests about multiprocessing

Signed-off-by: Wenqi Li <wenqil@nvidia.com>

wyli

the integration tests work fine, could you please revise the docstrings, and in my opinion we can merge this. more tests with Compose.execute would be great as well.

Nic-Ma

The change logic looks good to me, put minor comments inline.
Please complete the doc-strings and double-confirm the threading arg in the self.transform() function of every dataset class.

Thanks.

monai/transforms/compose.py

monai/data/dataset.py

atbenmurray · 2023-03-24T10:22:13Z

@wyli @Nic-Ma what is the issue with the failure? Not sure what is causing it

wyli · 2023-03-24T10:41:36Z

@wyli @Nic-Ma what is the issue with the failure? Not sure what is causing it

those tests are optional and deprecating... please ignore

…o compose_refactor

I, Ben Murray <ben.murray@gmail.com>, hereby add my Signed-off-by to this commit: 61f67e4 Signed-off-by: Ben Murray <ben.murray@gmail.com>

Signed-off-by: Ben Murray <ben.murray@gmail.com>

monai/data/dataset.py

atbenmurray · 2023-03-28T15:14:34Z

~~If everyone is happy review-wise, I'll sort out the remaining reds in anticipation of a merge~~
@ericspod suggested that I make Compose.execute a function rather than having it as a class method, so I'll do that first before we review for merge.

wyli · 2023-03-28T15:16:16Z

If everyone is happy review-wise, I'll sort out the remaining reds in anticipation of a merge

please update the docstring of the datasets to clarify the implicit execution of pending operations during caching. otherwise this looks good to me.

monai/transforms/compose.py

@wyli

… Documenting lazy resampling functionality and updating dataset docs to refer to it as per @wyli's request Signed-off-by: Ben Murray <ben.murray@gmail.com>

Signed-off-by: Ben Murray <ben.murray@gmail.com>

I, Ben Murray <ben.murray@gmail.com>, hereby add my Signed-off-by to this commit: 6bdedac Signed-off-by: Ben Murray <ben.murray@gmail.com>

through list comprehension Signed-off-by: Ben Murray <ben.murray@gmail.com>

Signed-off-by: Ben Murray <ben.murray@gmail.com>

execute_compose. Signed-off-by: Ben Murray <ben.murray@gmail.com>

Signed-off-by: Ben Murray <ben.murray@gmail.com>

critical design flaw that needs fixing Signed-off-by: Ben Murray <ben.murray@gmail.com>

code with an incorrect type Signed-off-by: Ben Murray <ben.murray@gmail.com>

Signed-off-by: Ben Murray <ben.murray@gmail.com>

wyli · 2023-03-30T09:57:43Z

/build

Fixes Project-MONAI#6223 . ### Types of changes  - [x] Non-breaking change (fix or new feature that would not break existing functionality). - [x] Breaking change (fix or new feature that would cause existing functionality to change). - [ ] New tests added to cover the changes. - [ ] Integration tests passed locally by running `./runtests.sh -f -u --net --coverage`. - [ ] Quick tests passed locally by running `./runtests.sh --quick --unittests --disttests`. - [ ] In-line docstrings updated. - [ ] Documentation updated, tested `make html` command in the `docs/` folder. --------- Signed-off-by: Ben Murray <ben.murray@gmail.com> Signed-off-by: a-parida12 <abhijeet.parida@tum.de>

atbenmurray and others added 2 commits March 22, 2023 13:55

Initial commit to resolve #6223

1759f96

[pre-commit.ci] auto fixes from pre-commit.com hooks

ba4115d

for more information, see https://pre-commit.ci

atbenmurray requested review from wyli, ericspod, Nic-Ma and rijobro March 22, 2023 13:58

monai-bot and others added 3 commits March 22, 2023 14:06

[MONAI] code formatting

632119e

Signed-off-by: monai-bot <monai.miccai2019@gmail.com>

Initial commit to resolve #6223

50a9d06

Signed-off-by: Ben Murray <ben.murray@gmail.com>

Merging into autoformat fixes

dd5ad30

Signed-off-by: Ben Murray <ben.murray@gmail.com>

wyli reviewed Mar 22, 2023

View reviewed changes

monai/data/dataset.py Show resolved Hide resolved

atbenmurray added 5 commits March 22, 2023 14:16

DCO Remediation Commit for Ben Murray <ben.murray@gmail.com>

10edabd

I, Ben Murray <ben.murray@gmail.com>, hereby add my Signed-off-by to this commit: 1759f96 Signed-off-by: Ben Murray <ben.murray@gmail.com>q:

DCO Remediation Commit for Ben Murray <ben.murray@gmail.com>

5a8ccaf

I, Ben Murray <ben.murray@gmail.com>, hereby add my Signed-off-by to this commit: 1759f96 Signed-off-by: Ben Murray <ben.murray@gmail.com>

Merge branch 'compose_refactor' of github.com:project-monai/monai int…

918cbf6

…o compose_refactor

DCO Remediation Commit for Ben Murray <ben.murray@gmail.com>

c3bd8c0

I, Ben Murray <ben.murray@gmail.com>, hereby add my Signed-off-by to this commit: 10edabd Signed-off-by: Ben Murray <ben.murray@gmail.com>

Fixes to make test_cachedataset, test_persistentdataset and test_cach…

0c0baa4

…edataset_persistent_workers pass Signed-off-by: Ben Murray <ben.murray@gmail.com>

Documentation for Compose.execute

c5a73f6

Signed-off-by: Ben Murray <ben.murray@gmail.com>

wyli added 2 commits March 22, 2023 17:13

style/docs

41a156a

Signed-off-by: Wenqi Li <wenqil@nvidia.com>

Merge remote-tracking branch 'upstream/dev' into compose_refactor

eda9b4a

Signed-off-by: Wenqi Li <wenqil@nvidia.com>

wyli approved these changes Mar 23, 2023

View reviewed changes

Nic-Ma approved these changes Mar 23, 2023

View reviewed changes

monai/transforms/compose.py Outdated Show resolved Hide resolved

monai/data/dataset.py Outdated Show resolved Hide resolved

wyli added this to the Lazy resampling (+ cache-based dataset refactoring) [P1 v1.2] milestone Mar 23, 2023

atbenmurray added 3 commits March 24, 2023 17:03

Added tests; updated documentation

61f67e4

Merge branch 'compose_refactor' of github.com:project-monai/monai int…

bd591b7

…o compose_refactor

DCO Remediation Commit for Ben Murray <ben.murray@gmail.com>

7f37392

I, Ben Murray <ben.murray@gmail.com>, hereby add my Signed-off-by to this commit: 61f67e4 Signed-off-by: Ben Murray <ben.murray@gmail.com>

atbenmurray and others added 5 commits March 24, 2023 17:36

Honoring the self.copy_cache flag

5301af1

Signed-off-by: Ben Murray <ben.murray@gmail.com>

Merge branch 'dev' into compose_refactor

5bab340

Signed-off-by: Ben Murray <ben.murray@gmail.com>

Updating for lazy resampling

6bdedac

Autoformatting

d99b2b9

Signed-off-by: Ben Murray <ben.murray@gmail.com>

Merge branch 'dev' into compose_refactor

0223e62

wyli reviewed Mar 27, 2023

View reviewed changes

monai/data/dataset.py Show resolved Hide resolved

wyli mentioned this pull request Mar 28, 2023

update docstring and eval with overrides #6241

Closed

7 tasks

KumoLiu reviewed Mar 29, 2023

View reviewed changes

monai/transforms/compose.py Show resolved Hide resolved

atbenmurray added 11 commits March 29, 2023 10:31

Moving Compose.execute to execute_compose as per @ericspod's request.…

d939395

… Documenting lazy resampling functionality and updating dataset docs to refer to it as per @wyli's request Signed-off-by: Ben Murray <ben.murray@gmail.com>

Test fix: missed Compose.execute to execute_compose changes

4ecf2c3

Signed-off-by: Ben Murray <ben.murray@gmail.com>

DCO Remediation Commit for Ben Murray <ben.murray@gmail.com>

948444f

I, Ben Murray <ben.murray@gmail.com>, hereby add my Signed-off-by to this commit: 6bdedac Signed-off-by: Ben Murray <ben.murray@gmail.com>

Bug fix for SomeOff; generate list of transforms in execution order

9097e07

through list comprehension Signed-off-by: Ben Murray <ben.murray@gmail.com>

Documentation for Compose.get_index_of_first

d701642

Signed-off-by: Ben Murray <ben.murray@gmail.com>

Slight documentation reformatting for Compose.get_index_of_first

77e465d

Signed-off-by: Ben Murray <ben.murray@gmail.com>

Updated docstrings for execute_compose. Renamed input_ to data for

5605e71

execute_compose. Signed-off-by: Ben Murray <ben.murray@gmail.com>

Fixing errors reported by flake8-py3 (mypy) output

c06e014

Signed-off-by: Ben Murray <ben.murray@gmail.com>

Had to go back to lazy_evaluation default of None for now but this is a

191506d

critical design flaw that needs fixing Signed-off-by: Ben Murray <ben.murray@gmail.com>

execute_compose type ignore as it can't be fixed without polluting more

ec71402

code with an incorrect type Signed-off-by: Ben Murray <ben.murray@gmail.com>

type: ignore suppression as this is being addressed separately

85b18de

Signed-off-by: Ben Murray <ben.murray@gmail.com>

atbenmurray marked this pull request as ready for review March 30, 2023 09:35

Merge branch 'dev' into compose_refactor

7684edf

wyli enabled auto-merge (squash) March 30, 2023 13:30

wyli merged commit e4d48f0 into dev Mar 30, 2023
27 of 32 checks passed

wyli deleted the compose_refactor branch March 30, 2023 13:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve Compose encapsulation #6224

Improve Compose encapsulation #6224

atbenmurray commented Mar 22, 2023

wyli commented Mar 22, 2023 •

edited

atbenmurray commented Mar 22, 2023

atbenmurray commented Mar 22, 2023

wyli commented Mar 22, 2023

wyli left a comment

Nic-Ma left a comment

atbenmurray commented Mar 24, 2023

wyli commented Mar 24, 2023

atbenmurray commented Mar 28, 2023 •

edited

wyli commented Mar 28, 2023

wyli commented Mar 30, 2023

Improve Compose encapsulation #6224

Improve Compose encapsulation #6224

Conversation

atbenmurray commented Mar 22, 2023

Description

Types of changes

wyli commented Mar 22, 2023 • edited

atbenmurray commented Mar 22, 2023

atbenmurray commented Mar 22, 2023

wyli commented Mar 22, 2023

wyli left a comment

Choose a reason for hiding this comment

Nic-Ma left a comment

Choose a reason for hiding this comment

atbenmurray commented Mar 24, 2023

wyli commented Mar 24, 2023

atbenmurray commented Mar 28, 2023 • edited

wyli commented Mar 28, 2023

wyli commented Mar 30, 2023

wyli commented Mar 22, 2023 •

edited

atbenmurray commented Mar 28, 2023 •

edited