Transformer decoding support fuse qkv #1455
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
guoshengCS
previously approved these changes
Dec 15, 2021
Done. Thanks. |
guoshengCS
reviewed
Dec 15, 2021
| @@ -65,7 +65,8 @@ std::vector<paddle::Tensor> DecodingForward( | |||
| const int64_t& max_len, | |||
| const float& beam_search_diversity_rate, | |||
| const bool& rel_len, | |||
| const float& alpha) { | |||
| const float& alpha, | |||
| const bool& fuse_qkv) { | |||
There was a problem hiding this comment.
不另外加这个fuse_qkv的attr了吧，直接根据size来判断吧，也保证对之前模型的兼容性
guoshengCS
reviewed
Dec 16, 2021
| param_type = item.split(".")[-1] | ||
|
|
||
| model_dict[ | ||
| "decoding.slf_q_" + param_type + "_" + |
There was a problem hiding this comment.
load和export是否也判断下self._fuse_qkv呢
There was a problem hiding this comment.
目前不会报 warning 也不影响使用。
已优化代码。Thanks.
guoshengCS
previously approved these changes
Dec 16, 2021
ZeyuChen
requested changes
Dec 16, 2021
| float alpha); | ||
| const std::string& decoding_strategy, | ||
| const int& beam_size, | ||
| const int& topk, |
There was a problem hiding this comment.
理论上int值的参数，是不需要加const和引用的。因为没有任何加速意义。
string的话通过引用，是会将copy从string降低到指针（32bit）的拷贝。
但是int呢，你不管加不加引用，拷贝的指针地址和拷贝int值是一样的。
正因为如此，所以你加不加const都一样了。
There was a problem hiding this comment.
Truth. 这里加 const 单纯是不需要修改的形参都习惯性加上 const，加上引用是想让形参列表看起来统一。我新的修改删除了引用，不过对于这样不需要修改的形参，还是建议保留 const。
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR types
Performance optimization
PR changes
Models
Description
Transformer decoding support fuse qkv. Performance comes later.