[BM25 TEXT SEARCH] How to input the preprocessed Bag-of-Word instead of full text.

[Zrealshadow]

I have found the tutorial of full text search in Milvus in https://milvus.io/docs/full-text-search.md.

the question is as the title,
is there a interface where we can skip the built-in analyzer and directly save the preprocess bag-of-word ?

[yhmo]

Milvus analyzer could not be so flexible.

A possible workaround:

1. Use whitespace tokenizer

analyzer_params = {
    "tokenizer": "whitespace",
}

2. You scan your text content by yourself. For each text, extract the words that you are interested, and concatenate them with whitespace to be a text string.
3. Define two varchar fields, one to store the original text content and the other to store the concatenated string.
4. Set the concatenated string field as the input_field of BM25Function, set a sparse vector field as ourput_field

With this approach, the tokenizer will process the concatenated strings and the internal bag-of-word is almost equal to yours.

[Zrealshadow]

Hi, thx for your explain.
I have found this method in https://milvus.io/docs/analyzer-overview.md.
However, my preprocess Bag-of-Word involve phrases which contain whitespaces. So I can not use "whitespace" to split it.

I notice there is a description in tutorial that

Tokenization follows specific rules, such as splitting by whitespace or punctuation, depending on the tokenizer type.

So can I customize the split character ?

If it's ok, do you mind giving me an example, since there are no refereces
Thx sincerely.

[xiaofan-luan]

@Zrealshadow
maybe we need a special tokenzier and you can contribute one?
Shouldn't be too hard to do it

[Zrealshadow]

Glad to do it, but I am not sure I can handle it. Maybe need some guidances

[xiaofan-luan]

@zhengbuqian @aoiasd could you setup a meeting with @Zrealshadow on implement this specialized analyzer?

[yhmo]

This is the entry of tokenizer:

milvus/internal/core/thirdparty/tantivy/tantivy-binding/src/analyzer/tokenizers/tokenizer.rs

Line 60 in a56b240

match name {

The tantivy lib is written by RUST, which requires rust background to do this task.

[Zrealshadow]

Proposed a draft PR #40577
Need revise (QAQ. I'm not rust programmer. )