Relevancy customization prototype

[curquiza]

Hello everyone,

For more context, check out the previous discussions on this subject:

• Policy for matching words product#255

According to the user feedback we got, we want to extend the user's possibilities to customize the relevancy behavior during the search.

Currently, Meilisearch returns documents containing all the query words as the most relevant documents, so at the top of the results. Then, below, it "removes" the last query words to make it optional and retrieves documents containing all the query words except the last one it removed, etc...

For example, if someone were to search batman dark knight, Meilisearch would return documents containing all three terms first, documents containing only batman AND dark second, and documents containing only batman third.

This prototype introduces a new search parameter named optionalWords allowing the users to customize the word matching policy chosen between several modes.

✨ We welcome everyone to install the prototype and try it out for a bit. Making the prototype available before the official release means the feedback you give us will go a long way toward ensuring this new feature is helpful and easy to use. ✨

Installation of the prototype

You must use Docker to install this prototype:

docker run -it --rm \
    -p 7700:7700 \
    -v $(pwd)/meili_data:/meili_data \
    getmeili/meilisearch:v0.29.0-optional-words.beta.2

We have also created custom versions of meilisearch-js and instant-meilisearch compatible with this prototype. We recommend installing these as project dependencies:

npm install meilisearch@0.27.0-optional-words-beta.1
npm install @meilisearch/instant-meilisearch@0.8.1-optional-words-beta.0

API Reference

A new search parameter has been introduced: optionalWords. Its values are:

last:

The documents containing ALL the query words (i.e. in the q parameter) are returned first by Meilisearch.
If Meilisearch doesn't have enough documents to fit the requested limit, it iteratively ignores the query words from the last typed word to the first typed word until the limit of requested documents is reached.

This is the previous Meilisearch's behavior and the prototype's default behavior, if optionalWords is not present in the search request body, last is the activated relevancy mode.

none:

Only the documents containing ALL the query words (i.e. in the q parameter) are returned by Meilisearch.
If Meilisearch doesn't have enough documents to fit the requested limit, it returns the documents found without reaching the limit.

How to use

cURL

curl \
  -X POST 'http://localhost:7700/indexes/movies/search' \
  -H 'Content-Type: application/json' \
  --data-binary '{
    "q": "batman dark knight",
    "optionalWords": "none"
  }'

meilisearch-js

await index.search('hello world', { optionalWords: 'none' })

instant-meilisearch

const searchClient = instantMeiliSearch(host, apiKey, { optionalWords: 'none' })

Bonus

We also introduced another mode for optionalWords you might want to test: the any value. With this mode, all the query words are optional, no matter their order in the query (i.e. the q parameter).

⚠️ This mode is valid as a prototype but, due to the high number of use case for this mode, we will probably not release it at the same time as none if we go into delivery phase.

Feedback

We are actively looking for feedback from our users. Let us know about your experience using the prototype in this discussion—your suggestions help us make this new feature better and easier to use. ❤️

Core: @ManyTheFish
Integration: @bidoubiwa

[gmourier]

Hello, this is only my feedback as a proxy user, so take it with a grain of salt.

I find the name optionalWords not very appropriate.

It makes me think that I have to pass a boolean and so doesn't naturally make me guess that there is an enum behind it.

Even if at first there is only last and none, I would rename it to something like termMatchingStrategy; this way, we could elegantly introduce more strategies in the future.

If we use a name like that, we should probably rename none to all to make more sense.

---

Regarding the none usage, it allowed me to retrieve the documents with all the terms present in the query and thus fulfilled his mission. Well done!

[bidoubiwa]

I agree on the questionable naming, nonetheless, I prefer wordsMatchingStrategy; maybe @meilisearch/docs-team have a better suggestion?

[curquiza]

I don't have a better solution, sorry, but this appears a long name to me, don't know if we can do shorter and still clear

[bidoubiwa]

There is matchingStrategy that can be infered as word matching. Not sure though

[curquiza]

Hello 👋

We have just released v0.29.0rc1, which is a release candidate of v0.29.0 🔥

It introduces a new search parameter named matchingStrategy which allows you to change the default Meilisearch behavior about retrieving documents.

For now, it accepts 2 strategies:

• last: the default strategy. If no document contains all the given terms, the engine removes the last query term and so on.
• all: All the query terms must be present in a document to make it a search result.

How to use it:

curl \
  -X POST 'http://localhost:7700/indexes/movies/search' \
  -H 'Content-Type: application/json' \
  --data-binary '{ "q": "american ninja", "matchingStrategy": "all" }'

Binaries are attached to the release, or you can use the docker image:

docker run -it --rm \
    -p 7700:7700 \
    getmeili/meilisearch:v0.29.0rc1

Let us know about any bugs or feedback! 😄 It would be really helpful.

FYI, the official v0.29.0 release will be available on 3rd October.