Add RFC for batchnorm1d optim #135
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
|
zkh2016
reviewed
May 23, 2022
|
PR格式检查通过，你的PR将接受Paddle专家以及开源社区的review，请及时关注PR动态。 |
JamesLim-sy
reviewed
May 24, 2022
| batch_norm_out = batch_norm(x) | ||
| ``` | ||
|
|
||
| 简单的进行端到端实验测试，可以发现在数据规模较小时，torch和paddle都使用cudnn进行计算，而当数据规模大于一定阈值后，torch使用自己开发的算子进行计算使得计算时延小于paddle，进一步使用nsight compute查看profile结果： |
There was a problem hiding this comment.
请教下，这部分可以发现在数据规模较小时，torch和paddle都使用cudnn进行计算，此时Paddle和torch的性能是相同的吗，和后文中提到的oneflow相比呢？想确认下torch无论是调cudnn，还是启用自己手写的kernel，性能更好些
There was a problem hiding this comment.
这里我的测试shape是[200000, 1, 3]，在这个shape下paddle和torch都使用的是cudnn，性能也是相似的，后续我把shape改成[136000, 16]之后，torch在小数据规模下调用的自己的kernel，性能要比cudnn要好，oneflow的性能我还没测试过，之后测一下补充到这个文档里
zkh2016
reviewed
May 27, 2022
|
|
||
| ## 命名与参数设计 | ||
| 参考：[飞桨API 设计及命名规范](https://www.paddlepaddle.org.cn/documentation/docs/zh/develop/dev_guides/api_contributing_guides/api_design_guidelines_standard_cn.html) | ||
| ## 底层OP设计 |
There was a problem hiding this comment.
考虑到bn在许多模型中被使用，直接基于原算子修改需要验证大量的模型精度，所以建议
- 对当前的bn先解决不支持大batch_size的问题，
- 在paddle/phi/sparse目录下新增batch_norm，支持bn1d的情况，性能尽量打平或优于竞品
zkh2016
approved these changes
Jul 13, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.