Add local_response_norm in nn.functional and nn.layer #27725
Conversation
|
Thanks for your contribution! |
|
✅ This PR's description meets the template requirements! |
huangjun12
changed the title
| data_format="NCHW", | ||
| name=None): | ||
| """ | ||
| Local Response Normalization performs a type of "lateral inhibition" by normalizing over local input regions. |
There was a problem hiding this comment.
refer to its own class doc
There was a problem hiding this comment.
Done, functional is referred to class
|
|
||
| input1 = fluid.data( | ||
| name="input1", shape=[3, 40, 40], dtype="float32") | ||
| input2 = fluid.data( |
There was a problem hiding this comment.
input1/2 shape should be [3, 3, 40] and [3, 40, 3]?
There was a problem hiding this comment.
[3, 40, 40] also works
| self.check_static_4d_input(place=place) | ||
| self.check_static_5d_input(place=place) | ||
|
|
||
| def check_dygraph_3d_input(self, place): |
There was a problem hiding this comment.
so it only check the data_format?
There was a problem hiding this comment.
Yeah, Acc is aligned with torch implementation for simplicity, here we just test data_format
| self.places.append(fluid.CUDAPlace(0)) | ||
|
|
||
| def test_dygraph(self): | ||
| for place in self.places: |
There was a problem hiding this comment.
why don't use the new api to test dygraph?
python/paddle/nn/functional/norm.py
Outdated
|
|
||
| channel_last = True if data_format[-1] == "C" else False | ||
|
|
||
| div = fluid.layers.unsqueeze(paddle.multiply(x, x), axes=[1]) |
There was a problem hiding this comment.
use functional instead of layers?
python/paddle/nn/functional/norm.py
Outdated
|
|
||
| div = fluid.layers.unsqueeze(paddle.multiply(x, x), axes=[1]) | ||
| if channel_last: | ||
| if dim == 3: |
There was a problem hiding this comment.
this part is a bit too repetitive, could use some consolidation.
benchmark测试: |
so the time cost of the 3d version is about the same as the 1d version, even though the data is 1120x larger? |
Yes... a little confused |
|
this is weird, looks like it is implemented with mostly elementwise and reshape operations , would expect it to be linear in complexity. |
the mean cost time of this api is obtained by removing the time of initialization and data copy, results as follows: | input tensor | shape | torch cost | paddle cost | which is faster than pytorch, and shows linear complexity as expected. |
…7725) * add local_response_norm in nn.functional and nn.layer, test=develop * update layers to functional api, test=develop * fix ci coverage, test=develop * fix unittests bug, test=develop
PR types
New features
PR changes
APIs
Describe
原API:
paddle.fluid.layers.lrn(input, n=5, k=1.0, alpha=0.0001, beta=0.75, name=None, data_format='NCHW')
存在如下问题:
(1) 精度与torch实现有较大diff，最大相对误差约为1e-4
(2) 仅支持4-D输入，不支持3D及5D输入
(3) api名称、参数名称、参数顺序与torch的接口不同
在functional和layer下新增如下两个API，修复了以上问题，实现和接口形式对齐torch:
(1) paddle.nn.functional.local_response_norm(x, size, alpha=1e-4, beta=0.75, k=1., data_format="NCHW", name=None)
(2) paddle.nn.LocalResponseNorm(size, alpha=0.0001, beta=0.75, k=1.0, data_format="NCHW", name=None)