Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix reduce op in nlp: text_matching/sentence_transformers when last dim is 1 and reduce mid dim #34941

Merged
merged 18 commits into from
Aug 17, 2021

Conversation

AnnaTrainingG
Copy link
Contributor

@AnnaTrainingG AnnaTrainingG commented Aug 16, 2021

PR types

Bug fixes

PR changes

OPs

Describe

fix a bug in nlp: text_matching/sentence_transformers when last dim is 1 and reduce mid dim

bug现象:

global step 10, epoch: 1, batch: 10, loss: nan, accu: 0.42500, speed: 12.92 step/s
global step 20, epoch: 1, batch: 20, loss: nan, accu: 0.42188, speed: 23.03 step/s
global step 30, epoch: 1, batch: 30, loss: nan, accu: 0.42292, speed: 23.87 step/s
global step 40, epoch: 1, batch: 40, loss: nan, accu: 0.43906, speed: 22.53 step/s
global step 50, epoch: 1, batch: 50, loss: nan, accu: 0.43562, speed: 22.56 step/s
global step 60, epoch: 1, batch: 60, loss: nan, accu: 0.43281, speed: 23.17 step/s
global step 70, epoch: 1, batch: 70, loss: nan, accu: 0.43661, speed: 21.93 step/s
global step 80, epoch: 1, batch: 80, loss: nan, accu: 0.43789, speed: 23.65 step/s
global step 90, epoch: 1, batch: 90, loss: nan, accu: 0.43368, speed: 23.62 step/s
global step 100, epoch: 1, batch: 100, loss: nan, accu: 0.43188, speed: 22.27 step/s

正常运行结果:
global step 10, epoch: 1, batch: 10, loss: 0.77060, accu: 0.56875, speed: 12.83 step/s
global step 20, epoch: 1, batch: 20, loss: 0.64212, accu: 0.58437, speed: 23.04 step/s
global step 30, epoch: 1, batch: 30, loss: 0.63177, accu: 0.60729, speed: 23.92 step/s
global step 40, epoch: 1, batch: 40, loss: 0.61330, accu: 0.60703, speed: 22.58 step/s
global step 50, epoch: 1, batch: 50, loss: 0.63015, accu: 0.62125, speed: 22.58 step/s
global step 60, epoch: 1, batch: 60, loss: 0.65486, accu: 0.63281, speed: 23.30 step/s
global step 70, epoch: 1, batch: 70, loss: 0.60717, accu: 0.63705, speed: 21.97 step/s
global step 80, epoch: 1, batch: 80, loss: 0.61400, accu: 0.63984, speed: 23.68 step/s
global step 90, epoch: 1, batch: 90, loss: 0.48885, accu: 0.64306, speed: 23.56 step/s
global step 100, epoch: 1, batch: 100, loss: 0.58543, accu: 0.64281, speed: 22.10 step/s

修复后:
global step 10, epoch: 1, batch: 10, loss: 0.77057, accu: 0.56875, speed: 12.79 step/s
global step 20, epoch: 1, batch: 20, loss: 0.64191, accu: 0.58437, speed: 22.83 step/s
global step 30, epoch: 1, batch: 30, loss: 0.63172, accu: 0.60729, speed: 23.88 step/s
global step 40, epoch: 1, batch: 40, loss: 0.61274, accu: 0.60703, speed: 22.50 step/s
global step 50, epoch: 1, batch: 50, loss: 0.63057, accu: 0.62062, speed: 22.54 step/s
global step 60, epoch: 1, batch: 60, loss: 0.65494, accu: 0.63229, speed: 23.25 step/s
global step 70, epoch: 1, batch: 70, loss: 0.60811, accu: 0.63661, speed: 21.94 step/s
global step 80, epoch: 1, batch: 80, loss: 0.61390, accu: 0.63984, speed: 23.60 step/s
global step 90, epoch: 1, batch: 90, loss: 0.48840, accu: 0.64306, speed: 23.63 step/s
global step 100, epoch: 1, batch: 100, loss: 0.58263, accu: 0.64250, speed: 22.28 step/s

现象说明: 模型reduce中间维度的时候如case:【32 33 1】 reduce_dim = 1 模型运行结果为nan
原因分析: 在进行代码整合的时候将kReduceALL 类型删除,并且增加了reduce.left_num == 1 调用cubReduce的判断,
但是在进行ReduceConfig中进行block、grid设置时,更新了reduce.left_num,导致reduce_left_num等于最低维1, 进入cu bReduce调用,使得最终结果计算错误。

@paddle-bot-old
Copy link

Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

@xingfeng01
Copy link
Contributor

LGTM

1 similar comment
@ZzSean
Copy link
Contributor

ZzSean commented Aug 16, 2021

LGTM

@Xreki Xreki merged commit 181f7ce into PaddlePaddle:develop Aug 17, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants