Semantic search

[kropsi]

Summary

Support pgvecotr extension for PostgreSQL or support for the Pinecone database for semantic indexing and search using embeddings.

We need to mark which fields we wish to index for different collections and these will then become searchable using semantic search.

Basic Example

Operator: Semantic search

Finds matches based on semantic similarity rather than the exact matching of keywords or phrases. It uses word embeddings and cosine similarity techniques to identify semantically similar content to the search query.

{
	"title": {
		"_alike": "Greate idea"
	}
}

Other alternatives _similar_to, _corresponding, _equivalent.

Motivation

There are several potential use cases for semantic search in Directus, including:

1. Enhanced search functionality: With the ability to perform a semantic search, users can more easily and effectively locate the information they need within the Directus platform.
2. Improved content discovery: Semantic search can help users discover content that they may not have otherwise found through traditional keyword-based search.
3. Better recommendations: By analyzing the semantic similarity of content, Directus can provide more relevant recommendations for related content or resources.

I think it can be an elegant addition to what Directus has to offer, which might impact Directus adoption positively as semantic search, in combination with some intelligent OpenAI operations in Flows, will be very popular.

It will be possible to create stores for OpenAI chat to make it more aware of a specific subject. Here is support database is a good example. It will also be a great addition to conventional search capabilities. Here are a few real-life examples where it can be helpful concerning semantic search functionality:

1. E-commerce: A customer searches for a "red dress" on an e-commerce website. A traditional keyword-based search may return results for dresses with the keywords "red" or "dress". Still, it may not capture other semantically similar dresses, such as "crimson gown" or "scarlet maxi."
2. Job search: A job seeker is seeking a "software engineer" position. A keyword-based search may return results for job postings that include the exact phrase "software engineer". Still, it may not capture postings that use related terms like "developer", "programmer", "software architect", or even "CTO".
3. News aggregation: A reader is interested in news about a recent political scandal. A keyword-based search may return articles that use specific keywords related to the scandal but may not capture pieces that use semantically similar phrases or concepts.

Detailed Design

The main idea behind this feature is to enable Directus to perform semantic searches by leveraging the power of vector embeddings using OpenAI text-embedding-ada-002 data model that can help with the following:

1. Indexing content selected for semantic indexing by converting text into embeddings or vector representations of text or meaning and storing them in the database of choice: Pinecone or PostgreSQL.
2. Searching by converting the searched phrase passed into _sem into embedding or vector representation of their own and then comparing the distances between the vectors using the functionality of pgvector or Pinecone.

It will fit right into the overall experience of Directus filters, besides the indexing part, where it might require some explanation and additional field configuration.

Requirements List

Must Have:

• A way to specify which fields in Directus must be indexed for semantic search.
• The ability to transform the information from the collections in these fields into embeddings and store them in the vector database for search.
• An extended filtering functionality that supports the new _sem operator, which works across Directus.

Should Have:

• The ability to use the OpenAI text-embedding-ada-002 model for generating embeddings.
• An environment variable (ENV VAR) to store the OpenAI credentials required for this feature.

Could Have:

• Additional models for generating embeddings.
• An option to customize the weights assigned to different fields during the embedding process.

Won't Have:

• Support for other vector databases besides Pinecone and PostgreSQL.
• An automatic way to detect the fields that should be indexed for semantic search.

Drawbacks

One potential drawback is the code complexity required to implement these features. Even if the amount of code required is relatively small, it could still add to the overall complexity of the Directus codebase and make it harder to maintain and debug in the future.

Additionally, while OpenAI's APIs may be easy to implement, there could still be issues with integration and maintenance over time. OpenAI is a third-party platform, and changes to its APIs or services could unexpectedly impact Directus. This could lead to additional development and maintenance costs over time.

Alternatives

Not sure right now what good alternatives there are, but it might require additional investigation.

Adoption Strategy

Making semantic search available will make a lot of sense for anyone and will be easy to adopt.

Unresolved Questions

Maybe support for pgvector is sufficient. How much better Pinecone is?

[kropsi]

Hi @rijkvanzanten, are you planning to introduce support for pgvector extension for PostgrSQL in Directus, in the near future?

[rijkvanzanten]

I don't know! This is a fairly new feature request, so hasn't really had the chance yet to gauge community interest. I am a little worried however that it's a postgres-specific extension, rather than a system that might work regardless of data store backend. I've also been considering adding a similar set of filter parameters, but have them powered by a "search-plugin" (which could then use like algolia, elasticsearch, or maybe pgvector!)

[M-GRV]

algolia is a great. Nice free tier as well

[seegeoff]

This seems like something you could accomplish through an extension

[kropsi]

Yes, indeed, but I think it is very useful as a core feature. It can be used for semantic search and working with LLM flows and applications.

[rijkvanzanten]

Heya! Thanks for opening this feature request!

This feature request has received over 15 votes from the community. This means we'll move this feature request to the Under Review state!

The Core team will schedule a meeting to review this request as soon as possible. The discussion will then be approved or denied. You may or may not be invited to join this meeting with the core team.

For more information, see our Feature Request Process.

[MasonGeloso]

@rijkvanzanten

My gosh this would be an excellent feature. Right now we are using Supabase just for the embeddings support, which is just pgvector. We use Directus for literally everything else. If this was natively supported, that would be amazing. I can imagine it being a pretty big piece of marketing as well.

You could even easily add a LangChain integration. Which I would imagine, could attract more users or companies currently building AI stacks. Something to think about. I will probably build an extension in the meantime.

[M-GRV]

@rijkvanzanten I'm sure you seen this but:

https://typesense.org/ is open source and could fulfill this need without needing to solely rely on Postgres or having to pay for Algolia/Elastic

[rijkvanzanten]

Yeah! I do think there's a difference between "Vector" support and "semantic search" though which are kinda getting mixed in this feature requests. I'd love to somehow allow pgvector or PlanetScale's vector for MySQL through the API as well as allowing you to use something like typesense or algolia for search 🤔

[kropsi]

@rijkvanzanten what I meant was: Semantic search as an application of vector support in the context of search, where it leverages vectors to understand and match the meaning behind text rather than just the explicit words used, but I understand that it might be mixed up if you have plans to use some other techniques, then we can just generalise it. I guess the most important is that you are able to understand the intent of the search and provide the closest results.

My point is that semantic search could be a great addition to the core of Directus and support for pgvector and embeddings could be an easy start. I agree with @MasonGeloso that there are a lot of benefits for Directus to have some tools in place to leverage the momentum that AI wave gives. Combination of Directus CMS based knowledge bases which can be searched semantically and then fed into LLMs through Directus Flows creates a whole ocean of interesting use-cases.

[willemmulder]

Any update on this one?

It would be amazing to be able to identify a field as a 'vector field' that we could fill with an arbitrary vector value, and then it would magically become vector-searchable through a set of new API endpoints/filters!

We could start by supporting SQLite VSS (e.g. through LangChain https://python.langchain.com/docs/integrations/vectorstores/sqlitevss/) to give an awesome out-of-the-box experience for self-hosted.

Would be great!