Optimize pos-stats

[HLasse]

Calculating POS stats seems to slow things down significantly. TODO:

• Profile the package, what causes the slowdown?
• Calculate the sum of values once and then call in the dict comprehension in PosStatistics

Ideas for speedup:

• Identify which pos_tags the model can make and predefine the counter/dictionary with those keys (would also solve the issue of different numbers of keys across docs/sentences)
• Alternatives to Counter?

Other options:

• Remove posstats from default TextDescriptives and make it an optional component that takes in which specific POS tags the user is interested in and extracts those (+ 'others')

[HLasse]

A job for you, @martbern ?

[MartinBernstorff]

Very well might be! Have quite a few open loops atm – what's the priority on this?

[HLasse]

Profiling has fairly high priority - if pos-stats really slows things down as much as i fear, then it should be removed from the default TextDescriptives pipe pretty soon. Optimizing it is not a huge priority atm

[MartinBernstorff]

Okay, I've been doing some digging! I've been unable to find any component that dramatically improves runtime.

Attached is my profiling script and its profile from cProfile: https://www.dropbox.com/sh/s60k7894ufva5vf/AADSpcN5OaDkg5rFDIPAAdLwa?dl=0

I recommend snakeviz for visualisation 👍

Let me know if there's anything I can add!