【Hackathon 5th No.12】Add AdaptiveLogSoftmaxWithLoss API to Paddle #770
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
2 tasks
GGBond8488
reviewed
Dec 5, 2023
|
|
||
| adaptive_log_softmax_with_loss的计算分步骤如下 | ||
|
|
||
| $\text{head_output} = \text{linear}(\text{input}, \text{head_weight}, \text{head_bias})$ |
| $\text{output} += \text{take_along_axis}(\text{head_logprob}, \text{gather_inds.unsqueeze(1)}, \text{axis}=1).\text{squeeze()}$ | ||
|
|
||
| $\text{loss} = -\text{output.mean()}$ | ||
|
|
||
| ## 3、意义 | ||
| 在自然语言处理中，当字典维度过大时，embedding 将占据模型大部分参数量。 | ||
| 例如机器翻译任务中，词表维度大约是2^17，embedding维度取1024，那么就会产生将近1亿参数量， |
Patrick-Star125
force-pushed
the
softmax
branch
from
e5483e0
to
ec7a0af
Compare
GGBond8488
reviewed
Dec 7, 2023
|
|
||
| adaptive_log_softmax_with_loss的计算分步骤如下 | ||
|
|
||
|  |
There was a problem hiding this comment.
这个图片好像还是有点问题，那个=1应该是axis=1吧，还有，把每一层在做什么也说明一下
|
|
||
| layer层类API：`paddle.nn.AdaptiveLogSoftmaxWithLoss(in_features, n_classes, cutoffs, div_value=4.0, head_bias=False, name=None)`，包含两个主要方法： | ||
| - forward(self, input, label)，用于训练，返回为`output` 和 `loss` |
| # 六、测试和验收的考量 | ||
| 测试考虑的case如下： | ||
|
|
||
| - 数值正确性 | ||
| - 数值正确性（CPU、GPU、动态图、静态图） |
There was a problem hiding this comment.
和torch一样用计算等价的方式验证，numpy一部分缺失部分API，并且该API函数逻辑比较多，所以完全复现会比较繁琐
|
Done |
luotao1
approved these changes
Dec 11, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
修改 AdaptiveLogSoftmaxWithLoss API 设计文档