Merge pull request PaddlePaddle#20 from huangxu96/fix_bug_Bert_loss_n…

…oncoverage fix a bug in fp16 training of BERT that will cause non-covergence.
wangxicoding · Feb 20, 2021 · a4309db · a4309db
2 parents f7bd0ab + d709542
commit a4309db
Showing 1 changed file with 1 addition and 1 deletion.
diff --git a/benchmark/bert/run_pretrain_single.py b/benchmark/bert/run_pretrain_single.py
@@ -238,7 +238,7 @@ def do_train(args):
             p.name for n, p in model.named_parameters()
             if not any(nd in n for nd in ["bias", "norm"])
         ],
-        multi_precision=args.use_pure_fp16)
+        multi_precision=False)
     if args.use_amp:
         custom_black_list = (['lookup_table', 'lookup_table_v2']
                              if args.use_pure_fp16 else None)