slow BERT MLM evaluation

[tstadel]

Describe the bug
Evaluation of BERT MLM head takes a lot of time due to unnecessary GPU to CPU overhead. argmax on logits should be called before shifting the tensor to cpu.

Expected behavior
Evaluation of BERT MLM head should be as fast as possible as it delays training.

To Reproduce
run lm_finetuning.py from examplex

System:

• OS: Ubuntu 18.04
• GPU/CPU: GPU
• FARM version: newest

[tstadel]

Besides the GPU to CPU overhead there are list appending for loops in logits_to_preds and prepare_labels. Converting them to list comprehensions also increases evaluation speed as the loop is hit 30k times in case of BERT. However, I assume, this might be a performance vs. understandability issue.

[tstadel]

Additionally in Evaluator's eval function there is a seemingly unnecessary casting to numpy and back to list (line 75 and 76):
preds_all[head_num] += list(to_numpy(preds[head_num]))
label_all[head_num] += list(to_numpy(labels[head_num]))

For MLM, NSP and TextClassifiationHeads this is not necessary. Are there any cases, in which we need this?

[tstadel]

I quickly ran some tests regarding the two comments: performance is only affected if a debugger is attached.

[tholor]

Thanks for spotting & fixing this @tstadel ! Highly appreciated :)