Add GroupNorm #5175
Add GroupNorm #5175
Conversation
| if __name__ == "__main__": | ||
| import doctest | ||
|
|
||
| doctest.testmod() |
| from oneflow.python.nn.module import Module | ||
|
|
||
|
|
||
| @oneflow_export("nn.GroupNorm") |
There was a problem hiding this comment.
需要在 oneflow/python/test/modules 目录添加对应的 test_xxx 文件，测试
| The documentation is referenced from: | ||
| https://pytorch.org/docs/stable/generated/torch.nn.GroupNorm.html | ||
|
|
||
| Applies Group Normalization over a mini-batch of inputs as described in |
| https://pytorch.org/docs/stable/generated/torch.nn.GroupNorm.html | ||
|
|
||
| Applies Group Normalization over a mini-batch of inputs as described in | ||
| the paper `Group Normalization <https://arxiv.org/abs/1803.08494>`__ |
There was a problem hiding this comment.
这种超链接形式可能在我们的编译过程中不能很好支持，请留意编译后的结果，并且确保编译过程中没有 warning（否则现在也通过不了CI）
| assert (input.shape[1] == self.num_channels), "The channels of input tensor must equal num_channels" | ||
| origin_shape = input.shape | ||
| reshape_to_1d = flow.experimental.reshape(input, shape=[origin_shape[0], self.num_groups, -1]) | ||
| (mean, variance) = flow.experimental.nn.moments(reshape_to_1d, [2], keepdims=True) |
There was a problem hiding this comment.
pytorch 没有 nn.momentd， OneFlow 也没有迁移 flow.experimental.nn.moments。
这里可以用已经迁移的 flow.experimental.mean 和还没有迁移的 flow.experimental.var（但是已经有对应的老接口 flow.reduce_variance 分别算出来。不使用 nn.moments。
所以这个PR里，建议同时把 flow.experimental.var 也给对齐、搬运了（这也是为什么我说真的需求会在搬运过程中发现）。
PS：像 flow.experimental.nn.moments 其实不存在这种问题，自己试一试就会发现，所以搬运过程中还是要把自己力所能及的test给跑通比较好，自己就会发现问题，这样可以省掉来回沟通的时间成本
zjlablichenyang
requested review from
Ancientshi and
oneflow-ci-bot
and removed request for
Ancientshi
…into lcy_groupnorm
| test_case.assertTrue(np.allclose(y.numpy(), output, 1e-5, 1e-5)) | ||
|
|
||
|
|
||
| def _test_groupnorm_v2(test_case, device): |
There was a problem hiding this comment.
这个名字可以用更有含义的代替么 v2 的信息太少了，比如 smaller_shape 什么的
| flow.nn.init.zeros_(self.bias) | ||
|
|
||
| def forward(self, input: Tensor) -> Tensor: | ||
| assert len(input.shape) >= 3 |
| and zeros (for biases). Default: ``True``. | ||
|
|
||
| Shape: | ||
| - Input: :math:`(N, C, *)` where :math:`C=\text{num_channels}` |
There was a problem hiding this comment.
这里已经修改过了，我更新一下截图
| >>> # Put all 6 channels into a single group (equivalent with LayerNorm) | ||
| >>> m = flow.nn.GroupNorm(1, 6) | ||
| >>> # Activating the module | ||
| >>> output = m(input) |
There was a problem hiding this comment.
每种情况输出的shape都一样
| from test_util import GenArgList | ||
|
|
||
|
|
||
| input_arr = np.array( |
| ], | ||
| dtype=np.float32, | ||
| ) | ||
| output = np.array( |
There was a problem hiding this comment.
别的instancenorm，batchnorm都是这样直接写的
oneflow-ci-bot
requested review from
oneflow-ci-bot
and removed request for
oneflow-ci-bot
oneflow-ci-bot
requested review from
oneflow-ci-bot
and removed request for
oneflow-ci-bot
Add GroupNorm Module
GroupNorm Doctest
