Designing Search index for general user search

[ibotty]

Hi!

I am investigating using Quickwit for our customer facing search. We do have a problem that our (source) data is not that great. It is full of spam and it's unfeasible to clean it up but it should still find the right things.

An example might illustrate our problem.

We do have many documents with the string chapter somewhere in the indexed field among many other strings. It cannot be a stopword though, because we also do have the field The Chapter which should be found.

How can this be achieved?

We also do have some fields with extra characters that are part of the input search. Think M*A*S*H which ideally should be found by the search string m*a*s*h as well as mash.

Is this possible? If so, how?

Thank you in advance!

[fulmicoton]

@ibotty what kind of search is this? It sounds like Quickwit is not the right the right product for your use case.

For user facing search, if you don't have terabytes of data, meilisearch should be a better fit for instance.

[ibotty]

Thank you for getting back!

It indeed is meant as user facing search in a web interface. Unfortunately we do have much data. To be specific: we have many documents but only relatively small string fields.

I was under the assumption that quickwit as OpenSearch replacement could also be used for user facing searches.

I asked a similar question on Meilisearch's discussion page. A problem with Meilisearch is its missing HA deployment story and having all data local (and not in object storage) which is harder to operate.

[PSeitz]

We do have many documents with the string chapter somewhere in the indexed field among many other strings. It cannot be a stopword though, because we also do have the field The Chapter which should be found.

As Phrase Search with "The Chapter" should work. For that you would need to configure to index with positions.

We also do have some fields with extra characters that are part of the input search. Think MASH which ideally should be found by the search string mash as well as mash.

We don't support regex search currently.