AI Search Mechanisms

[gmourier]

Hello everyone 👋

Having this discussion space could open up the conversation about this topic and see if we can identify some valuable cases to explore with the community as we go along.

Here are a few use cases I have in mind that might be interesting to discuss, and I'd love to hear your thoughts on them and for you to share if you are having trouble accomplishing this today with your search engine.

Use-cases

• Semantic Search
• Find similar documents (given a specific document or a set of documents)
• Contextualized Search
  • An "out-of-the-box" relevancy that uses embeddings to match documents sharing a context with the query?
  • Auto-apply facets on a search according to a probability of relevance from the query terms?
• Document summarization
• Question/Answer

Do you see any other use cases? Which ones could solve a need you have?

Challenges

• Obfuscate those technical notions to users who don't have the time to invest in it and, therefore, keep the product simple and intuitive for everyone.
• Guarantee excellent performance. (It is far from what a keyword-based engine can deliver performance-wise from my explorations, but I may be wrong, and it probably can't be generalized for all use cases described here).

What's next?

Before exploring anything with technical work, we would like to know a little more about what can be interesting and valuable from your feedback and experience.

We also have a forum dedicated to this topic on our Discord. Feel free to join! ✨

[carlosbaraza]

As mentioned in #642:

It would be great if neural embeddings could be added to each item and weight the embedding neighbours as part of the search.

This could potentially enable:

More meaningful searches with support for synonyms, etc.
Getting a few results to prompt a model like GPT-4 with some context (langchain style)
Even more ambitious and cool would be to somehow allow integrating a ML conversational API like OpenAI ChatGPT API. Not sure if it would be a good add-on for meilisearch, but it could somehow merge both crazy speeds of Meilisearch with the relevancy of a conversational AI.

Any ideas?

[gmourier]

Hey @carlosbaraza 👋

Those are indeed ideas we are currently exploring!

How do you solve this need today? For which use case? Knowing that Meilisearch can't yet store and retrieve vectors

How would you feel if Meilisearch won't compute the embeddings but let the developers do that with a third party like (OpenAI, Cohere, HuggingFace, etc)?

Thanks!

[carlosbaraza]

@gmourier We don't solve this need today, but we integrated pgvector and pgml with the intention of generating embeddings directly in the DB. Then when implementing a smart search, we'd generate the embedding for the query, fetch the relevant results and give it all to a LLM as context, so the LLM can answer. There are great open source models for embeddings: https://huggingface.co/spaces/mteb/leaderboard.

The zero-shot approach is a bit limiting right now because the context windows are small for the frontier LLMs, but the context window will increase, and this zero-shot approach could be really useful to harness the reasoning powers of LLMs.

Ideally, the LLM would be integrated in the search engine, not an external player. However, at the moment, Open Source LLMs that can run on commodity hardware are not that great. But my bold prediction is that we will have LLMs with GPT-3.5 capabilities and able to run on CPU in about a year. That'd be enough to enable smart NLP search.

[sing78]

Another use case could be "Matching".
You have a document and want to find matched documents based on some criteria.
For example, on a job board, propose jobs to candidates based on their CV or propose candidates to recruiters based on posted jobs, etc.

[gmourier]

Hey @sing78 👋

Could you detail an example? It seems to be a document recommendation system given some rules. Is that right?

Today, if you inject personalized filters based on what a user profile is looking for (preferences) already solves this use-case. Happy to better understand it!

Thanks!

[mc25573]

I got linked to this discussion through the Semantic Search / Vector Search roadmap item... You didn't mention it explicitly but I would love to have semantic search abilities. This would allow a more intuitive way for users to find relevant articles in my use case. The architecture would be simple and performant. You simply generate embeddings for each item that you want to be searchable, in my case, article titles, and then you store them in a vector database. Vector databases have great performance.

[gmourier]

Hey @mc25573 👋

From what I understand, you are considering generating the embedding on your side, akka externally to Meilisearch. Do you see that as friction?

[carlosbaraza]

TBH, if you have long articles and want to index them, it is already common to process them and split the content in chunks to index in Meilisearch. This is just a small incremental cost if you want to use vector embeddings.

[macraig]

Thank you all for taking the time to leave your feedback and share your use cases; it's super helpful! We're currently exploring different solutions and discussing their scope, we'll make sure to keep you updated in this discussion thread.

[Kerollmops]

Hello everyone 👋

We just released a 🧪 prototype that allows Meilisearch to serve as a vector store.

With this prototype, Meilisearch accepts vectors in the documents, stores them, and lets users retrieve the nearest documents using the search endpoint.

Install the Prototype

You can either use Docker and fetch the prototype image by using the following command:

docker run -p 7700:7700 -v $(pwd)/meili_data:/meili_data getmeili/meilisearch:prototype-vector-store-1

Or you can compile from the source using the Rust compiler and checkout the prototype-vector-store-0 tag:

git clone https://github.com/meilisearch/meilisearch
git checkout prototype-vector-store-1
cargo run --release

Send Vectorized Documents

This prototype handle already vectorized documents, which means that you must send the document's vector along with the document itself. The vector must be sent in your document's _vector field.

curl -X POST -H 'content-type: application/json' \
'localhost:7700/indexes/myvectors/documents' \
--data-binary '[
    { "id": 0, "_vectors": [0, 0.8, -0.2], "text": "Hello World" },
    { "id": 1, "_vectors": [1, -0.2, 0], "text": "Hello Doggy" },
    { "id": 2, "_vectors": [[0.5, 3, 1], [-0.2, 4, 6]], "text": "Hello People" }
]'

Query Meilisearch using Vectors

Once Meilisearch is aware of your vectorized documents you can query for them by using no more than the default search route. To do that you must compute the vector of your query and send the vector in the vector field.

curl -X POST -H 'content-type: application/json' \
'localhost:7700/indexes/myvectors/search' \
--data-binary '{ "vector": [0, 1, 2] }'

Executing this query should output something like the following:

{
  "hits": [
    { "id": 0, "_vectors": [0, 0.8, -0.2], "text": "Hello World", "_semanticSimilarity": 0.6754 },
    { "id": 1, "_vectors": [1, -0.2, 0], "text": "Hello Doggy", "_semanticSimilarity": 0.7546 },
    { "id": 2, "_vectors": [[0.5, 3, 1], [-0.2, 4, 6]], "text": "Hello People, "_semanticSimilarity": 0.78" }
  ],
  "query": "",
  "vector": [0, 1, 2],
  "processingTimeMs": 0,
  "limit": 20,
  "offset": 0,
  "estimatedTotalHits": 2
}

Frequently Asked Questions

In this prototype, you can:

• Send documents with a _vectors field that you must compute on your side.
• Request the nearest documents based on the query vector you must also compute. Meilisearch will, at the moment, ignore the query string if a vector field is specified in the search and act as a vector store, not a hybrid search engine.

In this prototype, Meilisearch:

• Can't compute the vector of the documents or the query for you.
• Will return an empty list of hits if the number of filtered documents is high. Meilisearch will not return the associated distance at which it found the documents yet.

Notes

⚠️ We do NOT recommend using this prototype in production. This is for test purposes only.

Feedback and bug reporting when using this prototype are encouraged! Thanks in advance for your involvement. It means a lot to us ❤️

[crohr]

What is the maximum possible size of the vector array? Looks like it doesn't accept vector embeddings generated with OpenAI API for instance (size=1536)

[Kerollmops]

Hey @crohr 👋

There is no such limit. We support any vector size. We tried with OpenAI as well, and it was working great. Do you get an error or something in the log? How many documents do you have?

Here is a small set of 100 documents with _vectors (we changed that field in prototype-vector-store-1) that you can try by yourself. It contains some documents about COVID studies.

covid-corpus.embeddings.openai.meilisearch.small.json.zip

[crohr]

@Kerollmops thanks, it now works with prototype-vector-store-1 (was using prototype-vector-store-0). Could you confirm that it is normal that an index doesn't accept new documents if their vectors have not the same size as the existing ones (I would have assumed to see some kind of error in the logs, but nothing is displayed even in TRACE debug level)?

[Kerollmops]

@crohr, with the new prototype, you should see an error in your tasks when the vector is not of the same length as the others. Is there an issue with this behavior? The engine must also raise an error if the vector used in the query is not the right length.

[Kerollmops]

Hello everyone 👋

We just released a new prototype-vector-store-1 🎉 Please, ensure you update it to benefit from the latest updates.

What Changed in this Version?

• We return a new _semanticSimilarity score in the documents representing the score as a dot product.
• We return the original vector of the query in the response
• We renamed the _vector field into _vectors and now accept multiple vectors for a single document.
• We now always normalize the document and query vectors
• We return better error messages when the data is malformed

[crohr]

Thanks! I just did an implementation for rubyvideo.dev in PR adrienpoly/rubyvideo#19. This works fine, although the (ruby) SDKs probably needs updating to make vectors a first-class citizen.

[Kerollmops]

@crohr That's wonderful 🥳 Thank you very much for your feedback!

However, make sure you do not use a prototype in production 😅 Don't worry, when it will be released we will make sure the SDKs are updated accordingly.

[macraig]

Hello everyone 👋 We have an exciting update to share with you all! Vector Store has been merged and will be shipping in Meilisearch v1.3 (released July 31st) 🎉

You can join the conversation in the dedicated discussion: #677

Looking forward to your feedback!

[macraig]

Hello again 👋

We have just released the first RC (release candidate) of Meilisearch containing this new feature!
You can test it by using

• the release assets
• the Meilisearch Docker image

docker run -it --rm -p 7700:7700 -v $(pwd)/meili_data:/meili_data getmeili/meilisearch:v1.3.0-rc.0

You are welcome to leave your feedback in this discussion.

If you encounter any bugs, please report them here.
Thanks in advance for your help and your involvement in Meilisearch ❤️

🎉 The official and stable release containing this change will be available on July 31st, 2023

⚠️ RC (release candidates) are not recommended for production

[noangel2014]

​Hello I wish there was a function.If have a particular document id and want to find documents that are "similar" to this document, i can do a vector query that references this id directly.

Also I'm looking at how well mixed search can be implemented. I wonder if this is possible. If there are fewer than a few normal search results, then you can switch to vector search? What do you think

[macraig]

Hey folks 👋

v1.3 has been released! 🦁 You can now use Meilisearch as a vector store ✨

Note: This feature is experimental and we need your help to improve it! Share your thoughts and feedback on #677.

📚 https://www.meilisearch.com/docs/learn/experimental/vector_search

[doutatsu]

I've just now upgraded and has been testing this functionality out. But I have two questions, perhaps I am not using it correctly.

1. When I am not able to provide a vector, I still want to index the document - for example with title and description, I can always search by title, but sometimes I have descriptions and those I generate vectors for. I am unable to index those without descriptions
2. Even when a vector is generated, Meilisearch doesn't index the document, but simply returns 202 error. For example, for ~27k documents, only 2k were indexed... even though ~20k of said documents do have vectors, so I am at loss at what is going on

[macraig]

Conversation continued in https://github.com/orgs/meilisearch/discussions/677#discussioncomment-6631350

[gmourier]

Hey everyone! 👋

A quick message just to let you know that the v1.6 is in pre-release stage.

It brings hybrid search & auto-embedding features plus optimizations on the indexing speed, so if you are willing to play with it as an early bird, please do so!

The official release date for the v1.6 is scheduled for January 15th.

[macraig]

Hey folks 👋

v1.6 has been released! 🦊 Hybrid search and auto-embedding features are now available for your use ✨

Check out our documentation to learn how to use them. We're looking forward to your feedback!

[ajohnclark]

I can't get auto-embedding working at all with OpenAI via self-hosted/Ubuntu 22 latest. It is hitting the embedding OpenAI API but none of the results are showing _vectors or anything like that. When I search with hybrid it just starts with the first few records in index. I've even try changing semanticRatio to 2 to break it and it doesn't return any sort of error which I assume it would. I also checked the POST data and it seems to be passing to server just fine. Any ideas where I am doing wrong here?

`

enable vector store

url = '#'
headers = {
'Content-Type': 'application/json',
'Authorization': f'Bearer {key}'}

data = json.dumps({
"vectorStore": True,
})

response = requests.patch(url, headers=headers, data=data)

config

r = client.index('wp_posts').update_settings({ "embedders": {
"default": {
"source": "openAi",
"apiKey": "1234",
"model": "text-embedding-ada-002",
"documentTemplate": "Post title is '{{doc.title}}' and content of post starts as follows: {{doc.content|truncatewords: 50}}",
}
}
})

search vector store

url = '#'
headers = {
'Content-Type': 'application/json',
'Authorization': f'Bearer {key}'
}
data = json.dumps({
"q": "Anything",
"hybrid": {
"semanticRatio": 1,
"embedder": "default"
},
})
r = requests.get(url, headers=headers, data=data)
`

[dureuill]

hey @ajohnclark 👋

Could you enable the scoreDetails experimental feature, and then retry your search with the showRankingScoreDetails to true? the result should tell you which ranking rules were actually executed by Meilisearch.

Enabling score details experimental feature

url = '#'
headers = {
'Content-Type': 'application/json',
'Authorization': f'Bearer {key}'}

data = json.dumps({
"scoreDetails": True,
})

Searching with the score details

url = '#'
headers = {
'Content-Type': 'application/json',
'Authorization': f'Bearer {key}'
}
data = json.dumps({
"q": "Anything",
"showRankingScoreDetails": true,
"hybrid": {
"semanticRatio": 1,
"embedder": "default"
},
})
r = requests.get(url, headers=headers, data=data)

Can you post the content of _rankingScoreDetails in the returned documents?

Thanks!

[dureuill]

(FWIW, I do not reproduce the issue from here: setting semanticRatioto 2.0 yields an error as expected, and the semantic search is returning results)

[ajohnclark]

Thank you kindly for taking the time out of your day to respond @dureuill so quickly. After trying your changes it led me to the culprit which was using .get and not using .post, woopsie! I can see those sweet sweet relevant results now. Here is fixed code in case anyone can use it to get going faster.

enable vector store

url = f'{host}/experimental-features/'
headers = {
    'Content-Type': 'application/json',
    'Authorization': f'Bearer {api_key}'}

data = json.dumps({
    "vectorStore": True,
    "scoreDetails": True,
})

response = requests.patch(url, headers=headers, data=data)

print(response.content)

setup for your index (blog_posts)

r = client.index('blog_posts').update_settings({ "embedders": {
      "default": {
        "source":  "openAi",
        "apiKey": "api_key_here",
        "model": "text-embedding-ada-002",
        "documentTemplate": "Post title is '{{doc.title}}' and content of post starts as follows: {{doc.content|truncatewords: 50}}",
      }
    }
})
print(r)

search vector store

url = f'{host}/indexes/blog_posts/search'
headers = {
    'Content-Type': 'application/json',
    'Authorization': f'Bearer {api_key}'
}
data = json.dumps({
    "q": "Search Query Here",
    "showRankingScoreDetails": True,
    "hybrid": {
        "semanticRatio": 0.9,
        "embedder": "default"
    },
    "limit": 20
})
r = requests.post(url, headers=headers, data=data)

print(r.json())

[oaustegard]

For RAG purposes documents are typically chunked so as to be able to only bring back to the LLM those chunks relevant to the search query. Does Meilisearch provide any "native" functionality for this, or are there any best practices guidelines for implementation with RAG as the use case? I would assume treating the Meilisearch index as any other vector db and storing each chunk as a document ("chunk-document"), with the "parent document" either stored as a separate document or in a different datastore and maintaining referential integrity and sequencing (along with the plain text of the chunk) in chunk-document attributes?

[dureuill]

Hello @oaustegard 👋

Yes, you want to chunk larger documents when using the semantic search and even for regular keyword search.

Note that Meilisearch makes use of a document template (using the liquid template language) to turn your JSON documents into the text that is sent to the embedder, so that gives you the possibility to make sure each document includes only the relevant field, possibly truncated.

The chunking in itself though needs to be done manually. The way you described sounds good.