-
Notifications
You must be signed in to change notification settings - Fork 247
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
slow BERT MLM evaluation #375
Comments
Besides the GPU to CPU overhead there are list appending for loops in |
Additionally in Evaluator's For MLM, NSP and TextClassifiationHeads this is not necessary. Are there any cases, in which we need this? |
I quickly ran some tests regarding the two comments: performance is only affected if a debugger is attached. |
Thanks for spotting & fixing this @tstadel ! Highly appreciated :) |
Describe the bug
Evaluation of BERT MLM head takes a lot of time due to unnecessary GPU to CPU overhead. argmax on logits should be called before shifting the tensor to cpu.
Expected behavior
Evaluation of BERT MLM head should be as fast as possible as it delays training.
To Reproduce
run lm_finetuning.py from examplex
System:
The text was updated successfully, but these errors were encountered: