Reduce the number of concatenation for 10% inference time reduction

[pommedeterresautee]

Instead of concatenating token representations at the token level (row approach) and then perform the sentence tokenization, we make the process lazy by retrieving each token representation before concat, reorganize them in nested lists and perform the concat per column (all Word embeddings are concatenated in one operation, then all LM are concatenated in one op), then columns (each column being a kind of representation) are concatenated together.
The idea is because there are much more tokens per sentence than different kind of representation per token, there are less concatenation operations performed.

On Conll2003, 40 -> 36s.
GPU use whole time over 70% (when it reaches 100 there may be some improvements remaining, but the main bottleneck will be the model itself).

Let me know if your measures match :-)

FWIW, on French dataset, it s a 20% improvement, I have stopped to try to guess why.

Nb. : I downloaded Connl 2003 from https://github.com/synalp/NER/tree/master/corpus/CoNLL-2003

[alanakbik]

Wow this is a great idea to save cat operations! :) I get the same speed improvements, i.e. CoNLL-03 inference is down from ~34 to ~31 seconds with 'gpu' storage and from ~32.7 to ~29.8 seconds with 'none'. Awesome!

[adizdari]

👍

[alanakbik]

👍