Memoize tokenisation in CountVectorizer

[jnothman]

IIRC, previous attempts to improve efficiency in CountVectorizer have found that tokenization is the primary bottleneck. In my private extensions, memoizing the mapping (s -> tokenize(preprocess(s))) can improve runtime greatly, especially when the vectorizer is in a cross-validation pipeline (which ColumnTransformer etc. helps encourage).

Challenges to memoization:

• it might makes sense to use in-memory caching rather than on-disk given the relatively small blobs being cached.
• it has to be conditioned on all relevant constructor parameters to the CountVectorizer (i.e. anything that goes into build_analyzer up to and including tokenization)

[rth]

Interesting! Possibly use functools.lru_cache for this once the code base becomes Python 3 only after the 0.20 release. There are existing benchmarks that could be used to evaluate the performance in bench_20newsgroups.py.

[mina1987]

Has anyone worked on this? I would like to work on this in Nov 2 WIML SF sprint.

[jnothman]

I think you could work on it @mina1987.

One tricky decision is how to handle the cache if the vectorizer is cloned

[mina1987]

I will start working on this!

[mina1987]

I am trying to understand this issue. The purpose is to store doc to token mapping to be used by possibly next similar documents, right?

[jnothman]

Yes it move help with repeated content, but more likely when it is fitting or transforming the data across multiple cross validation folds. Hence the need for the cache to be maintained when the estimator is cloned.

[mina1987]

How about having the cache as input/output of fit transform vectorizer?

[jnothman]

I'm not sure what you mean. A pull request or code snippet would make your proposal more concrete