【Hackathon 6th No.13】为 Paddle 新增 RAdam / NAdam API 之 NAdam #849
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
megemini
changed the title
cxxly
reviewed
Apr 3, 2024
| 输出数值结果的一致性和数据类型是否正确，使用 PyTorch 作为参考标准 | ||
|
|
||
| - **计算精度** | ||
| 需要保证 `前向/后向` 计算的精度正确性，使用 PyTorch 作为参考标准 |
cxxly
reviewed
Apr 3, 2024
| - $\mu _t$ 与 $\mu _ {t+1}$ | ||
|
|
||
| PyTorch 使用 `momentum_decay` 与一个系数 `0.96` 和 `step` 组合计算，TensorFlow 只与 `step` 相关 | ||
|
|
There was a problem hiding this comment.
详细对比下Paddle优化器参数与PyTorch区别，PyTorch涵盖的必要功能原则Paddle都能覆盖，且签名要和Paddle现有优化器风格保持一致。建议这里加个表格，详细对比Paddle已有Adam优化器、PyTorch优化器、拟设计的Paddle新增NAdam优化器
Update 20240409
@cxxly 请评审！ |
This was referenced Apr 17, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR types
Others
PR changes
Docs
Description
【Hackathon 6th No.13】为 Paddle 新增 RAdam / NAdam API
这里涉及两个算法，此 PR 为其中
NAdam算法。请评审！