New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add momentum optimizer and test FP32 AMP PureFP16 #4

Open

pangyoki wants to merge 1 commit into wawltor:master from pangyoki:add_pure_fp16_for_bert_finetune

pangyoki commented Nov 26, 2020 •

edited

Loading

In Bert SST-2 finetune task:

change AdamW optimizer to Momentum optimizer ( for using multi_precision in Pure Fp16 training)
use run_glue_fp32.sh, run_glue_amp.sh, run_glue_pure_fp16.sh to run FP32, AMP or Pure FP16 version.
you may need to use cast if it occurs LayerNorm or matmul dtype ERROR


          add momentum optimizer and test FP32 AMP PureFP16

e9a03c3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet