Spacy as tokenizer and for stop words?

[MaximilianHess]

spaCy provides solid tokenization across many languages, along with built-in stop-word lists and lemmatisation. Would it make sense to integrate it as an optional tokenizer for users who prefer higher accuracy, even if it comes with a performance cost?

I would be happy to work on it

[xhluca]

Hi Maximilian!

A goal of the project for bm25s is to keep things as simple as possible, thus have as few dependencies as possible. For instance, we removed scipy from the dependencies and move to optional (although I would ideally remove it completley but want to keep it for backward compatibility, at least for v0.3.0). One issue with having spacy as a dependency, even optional, is that it will give the user the impression that it's the "official" way to do tokenize text, similar to how PyStemmer is the current "official" way to stem text. However, it may lead to a substantial slowdown (at least that was the case when i tried to use other tokenizer, haven't tested each and every tokenizer so can't tell the exact number). I want to avoid having slowdowns out-of-the-box, and i consider optional dependencies to be part of the "out-of-the-box" experience.

That said, I think it'd be great to have examples showing how to use bm25s with different tokenizers, such as spacy. For example, here's an example with nltk: https://github.com/xhluca/bm25s/blob/main/examples/nltk_stemmer.py

Since Spacy is already high level, it'd probably make it straightforward to create an example for that, which i'm happy to merge if it matches the format of other examples. I also think it'd be good to share the examples more prominently, maybe by refactoring the huggingface integration section into a broader integration section that also mentions spacy, nltk, etc.

Feel free to start a PR if that sounds interesting to you!

[MaximilianHess]

Thank you for the quick reply.

I completely understand the concern about adding another dependency.

I took a quick look at the source code, and it seems that integrating a spaCy-based tokenizer would require at least a partial rewrite of the tokenize function, as well as adding a language parameter. That feels more like a low-level change than a simple example. Still, if you think this would be in scope for an example, I’d be happy to hear your thoughts.

The reason I’m leaning toward spaCy is that I’m working on a project involving Ukrainian, and I’ve had trouble finding a tokenizer that works well for it. My impression is that the default tokenizer in bm25s may be a bit too simple for Ukrainian.

I’ve also been wondering whether a subword tokenizer, similar to those used in LLMs, could improve BM25 performance. Do you think that would make sense in practice?

[xhluca]

I agree the default tokenizer mainly works for english. That said, the tokenize function is completely optional and only provided as a convenience. You can check out the Tokenizer class as an example of a more elaborate tokenizer, which you can probably subclass for your usecase.

[xhluca]

For subword tokenizers, i found the performance to be worse on beir. if you have good performance feel free to share the numbers in https://github.com/xhluca/bm25-benchmarks/

[MaximilianHess]

Thank you for the insights, and for the great support.

I hadn’t realized there was already a Tokenizer class. I’ll take a closer look at it and, if I manage to get spaCy integrated cleanly, I may open a PR for an example.

While I’m here, I’d also be curious about your experience with lemmatization versus stemming. Have you found one to work better than the other in practice?

Thanks again for your help.

[xhluca]

I haven't explored lemmatization much but i think it's interesting to explore since I imagine it would have fewer collision than stemming, and the results are probably more interpretable if we are printing out the frequencies!