Language and Locale

[silvadenisaraujo]

Dear commiters,

I would like to use txtai for a search query purpose but currently my content is not in English, is there parameters that can be provided to improve the results based on language and locale ?

Thanks,

[davidmezzetti]

There are additional models available on Hugging Face's model hub.

There are also additional models provided by the Sentence Transformers team. #17 has an example of how to use one of these.

[silvadenisaraujo]

Thanks for the answer, I will try with a couple of different models from the hub !

Adding to that, do you have an example on how to pass a tokenizer, to don't use the default at:

txtai/src/python/txtai/extractor.py

Line 33 in c85256f

self.tokenizer = tokenizer if tokenizer else Tokenizer

[davidmezzetti]

For the tokenizer, you can pass any object that implements the method:

def tokenize(text)

In this method, you can split the text as you see fit. Could be as simple as text.split(). There is also a lower level abstraction called Pipeline, that you may find more useful if you want to skip the tokenization/content filtering process all together.

[silvadenisaraujo]

Thanks a lot 👍