How can I speed up Simplemma for Polish?

[rafaelsaback]

First of all, thank you for creating this great lemmatizer!

I've used it for English and it's blazing fast (in my trials, it found the lemma in less than 10 ms). For other languages, such as Portuguese and Spanish it's still reasonably fast, with lemmatization working under 50ms.

For Polish, however, lemmatization is taking over 2 seconds. I know Polish is a more complicated language for lemmatizing because it's inflected, but is there a way I can speed it up? Ideally, I'd like to have it below 100ms, but even under 500ms I think it would be good enough.

In my trials, I'm passing a single word and doing it simply like this:

import  simplemma
import time

start = time.time()
langdata = simplemma.load_data('pl')
print(simplemma.lemmatize('robi', langdata))
end = time.time()
print(end-start)

[1over137]

I believe much of this time is spent loading the data. langdata is essentially a dict that is loaded from a big file. You can try lemmatizing 100k words to see how long it takes. It is quite fast for most needs if you pre-load the necessary data.

[rafaelsaback]

Awesome, you're 100%! After loading the langData, getting the lemma is really fast. I'll preload them. Thanks for the insight.