Speed improvements and support for Python 3.10

[adbar]

• use lru_cache from functools to minimize stoplist processing time
• use sets instead of lists for string search
• take regex compilation out of loop
• support for Python 3.10 in tests and setup

[miso-belica]

Thanks for the changes. I added some notes. Please take a look. Also, it seems Python 3.10 has a problem with the old pytest. Maybe it's finally time to drop Python 2.7. But I will deal with it later to decide if it can be fixed somehow even with v2.7 support.

[adbar]

Hi @miso-belica, thanks for the feedback, I implemented the changes you requested.

[adbar]

Another question: some changes were made by the Lexical Computing team, did you think of including them in this version of jusText?
https://corpus.tools/wiki/Justext_changelog

[miso-belica]

Thank you @adbar. I will check also the changes made by the team.

I am going to fix CI in the main, but in the future also for other projects, it is great to allow maintainers to edit your PR. Just a tip ;)

[adbar]

@miso-belica sorry, my bad, I left it unchecked.

[miso-belica]

No worries but while I have your attention. Do you know the URL of the git repo for https://corpus.tools/wiki/Justext_changelog? It's hard to find changes without comparable git history.

[adbar]

No, I don't know if it's publicly available, I just found this commit mentioned in the changelog:

• https://github.com/msklvsk/justext/commits/master

[adbar]

@miso-belica All changes combined lead to a 2.5x speedup on my data!

The commit mentioned above didn't get strictly translated or at least it has unexpected effects: slightly more recall and less precision. So I don't know what to think, maybe we should revert it, amend it or investigate further.

Anyway, a complete test suite would also be nice to detect such changes...

[miso-belica]

Wow, didn't expect such a speedup on those micro-optimizations. Good work 👏

The commit you mentioned really just fixes the bug in the parsing data, so I would let it be. It also has a test that would fail otherwise. If the NLP score is a bit different maybe it's a task to improve the algorithm/implementation somewhere but I don't believe we should add that bug back just to have a better score. The score would be just synthetic probably, meaning it's a coincidence for some specific data.

[adbar]

Thanks! OK, let's keep it that way.