813 ah net #815
Merged
813 ah net #815
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The groups parameter is also a parameter in Conv1d (and Conv2d, Conv3d, ...) class in torch.nn.modules.conv.py. In the original code, the value is set as 1 and cannot be modified directly, which is inconvenient for some networks such as seresnet and seresnext. Mention parameters when initialize Convolution Since one more parameter is added in Cnvolution, the parameter names should be mentioned (rather than input magic numbers directly) when initialize this class, otherwise errors (values are input into wrong parameters) may happen.
The project variable is used to match the number of channels before and after the sequeeze_and_excitation manipulation. In original code, the variable may be defined as a Conv layer with kernel size = 1, however, it can also be defined in other formats, such as a block that is consisted with a Conv layer and a Norm layer (see downsample part in github.com/Cadene/pretrained-models.pytorch/blob/master/pretrainedmodels/models/senet.py). Therefore, a more flexiable parameter could be more convenient.
Type hint losses/ and networks/ forward (Project-MONAI#774)
In a standard senet block and/or resnet block, after the features add the residuals, one relu layer is also utilized at the end of the block. Examples can be found in 1) senet implementations (https://github.com/Cadene/pretrained-models.pytorch/blob/master/pretrainedmodels/models/senet.py), 2) resnet implementations (https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py). Add Bottlenecks for SENet, SEResnet and SEResnext Bottlenecks are necessary in SE related networks.
SENet, SEResnet and SEResnext networks are implemented. Unittest for SENet shapes is added Add typing hints Some typing hints are added in convolutions.py and squeeze_and_excitation.py
Change the fixed relu function into an optional activation layer. Re-write test cases for senet Correct errors for docs and typing hints
For SENet, all relu layers have the parameters inplace equals to true, thus re-write the corresponding parts.
The differences between nn.Dropout, Dropout2d and Dropout3d are on the scopes of input tensors. 1D dropout can zero out some elements for each channel, 2D dropout can zero out some of the entire 2d channels and 3D dropout can zero out some of the entire 3d channels. Normally, as for SENet (no matter for 2D or 3D), only 1D dropout is needed. Therefore, the original implementations which keep the dims of dropout layers the same as other layers such as conv and norm layers may not be very reasonable. Here a new parameter named dropout_dim with default value 1 is added, and users do not have to set it unless they need 2d or 3d dropout layers.
FCN is a part of AHNet and helped to train 2D slices first. The code and the corresponding unittest are added. Fix errors automatically by isort
Add some typing hints for senet
Pytorch 1.4 corresponds to torchvision==0.5
wyli
reviewed
Jul 24, 2020
|
Hi @wyli , As we don't use |
looks like the current code requires both a |
|
OK, sounds good. Thanks. |
wyli
reviewed
Aug 3, 2020
yiheng-wang-nv
force-pushed
the
813-AHNet
branch
from
46902bd
to
45f0b26
Compare
wyli
approved these changes
Aug 4, 2020
|
/black |
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Implemented #813 .
Description
AHNet and its necessary block FCN have been implemented. Based on the author's original implementations ( https://github.com/lsqshr/AH-Net ), one more deterministic version has been added. Specifically, in this version, nn.ConvTranspose3d is used to replace all interpolate and up-sampling manipulations. Moreover, the kernel size for the maxpool layer has been changed to meet the consistency with stride value which is necessary to avoid utilizing atomicAdd operation (an non-deterministic operation).
Several control tests and ablation tests have been done to figure out the non-deterministic places, thus I think my conclusions are reliable. In addition, there are some evidences to support them:
Pytorch's official explanation:
https://pytorch.org/docs/stable/notes/randomness.html#pytorch
Similar findings in pooling layer's non-deterministic issue:
https://discuss.pytorch.org/t/inconsistent-results-with-3d-maxpool-on-gpu/38558/2
Since 2D model (FCN) uses resnet50 as its backbone and we do not wanna re-implement it, torchvision package is necessary. Whereas, the version of torchvision and pytorch has to be consistent. In MONAI, we use both Pytorch 1.4 (which needs torchvision 0.5) and the latest version (like Pytorch 1.6 which needs torchvision 0.7), thus my solution to maintain the compatibility is to modify several yml files in .github/workflows (please see the changed files for details). If there are some other more suitable strategies, feel free to mention here and I am able to change it.
What's more, this PR includes some changes for existing codes, including:
All modifications can be traced in the corresponding commits with comments.
As for the performance, I used AHNet (interpolate mode) and follow the notebook
examples/notebooks/spleen_segmentation_3d.ipynbto test the performance, the network achieves mean dice > 0.9470 in validation set after 300 epochs. @Nic-MaStatus
Ready
Types of changes
./runtests.sh --codeformat --coverage.make htmlcommand in thedocs/folder.