Kernel Memory with Elasticsearch

Use Elasticsearch as vector storage for Microsoft Kernel Memory.

---

Kernel Memory (KM) is a multi-modal AI Service specialized in the efficient indexing of datasets through custom continuous data hybrid pipelines, with support for Retrieval Augmented Generation (RAG), synthetic memory, prompt engineering, and custom semantic memory processing.

Utilizing advanced embeddings and LLMs, the system enables Natural Language querying for obtaining answers from the indexed data, complete with citations and links to the original sources.

---

This repository contains the Elasticsearch adapter that allows KM to use Elasticsearch as vector database, thus allowing developers to perform lexical and semantic search, in addition to hybrid, keyword and full-text search on your semantic content.

Pre-requisites

1. A running instance of Elasticsearch

   1. You can install a local instance of Elasticsearch using Docker. To simplify the setup of a running instance of Elasticsearch we prepared the article Installing the Elastic Stack using Docker Compose that guides you through the process. The following diagram shows what will be running once the installation is complete.

2. Alternatively you can use a cloud service like Elastic Cloud. The free tier is enough.

Configuration

The xUnit project UnitTests contains an appSettings.json file that lists all available options. The file reads as follows:

{
  "OpenAI": {
    "ApiKey": "...SECRETS...",
    "EmbeddingModelId": "text-embedding-ada-002",
    "ModelId": "text-davinci-003",
    "ChatModelId": "gpt-3.5-turbo"
  },  
  "Elasticsearch": {
    "CertificateFingerPrint": "...SECRETS...",
    "Endpoint": "https://localhost:9200",
    "UserName": "...SECRETS...",
    "Password": "...SECRETS..."
  }
}

The class used to store configuration is ElasticsearchConfig.

This file is supposed to show the available options but it is not meant to store sensitive information such as ApiKey, Password or CertificateFingerPrint. Modify this file as necessary (e.g. by changing the Endpoint), but add the values for the certificate fingerprint and the password in user secrets.

How to add user secrets

To add secrets either:

• Open the secrets file in your IDE by right clicking on the project name and selecting Manage User Secrets.

  • To read more about user secrets click here

• Add the secrets from the command line by running the following commands:

> dotnet user-secrets set "OpenAI:ApiKey" "...your Open AI API key..."
> dotnet user-secrets set "Elasticsearch:CertificateFingerPrint" "...your value..."
> dotnet user-secrets set "Elasticsearch:Password" "...your value..."

This ultimately results in the following secrets.json additions:

{  
  [..]
  "OpenAI:ApiKey": "...your Open AI API key...",
  "Elasticsearch:CertificateFingerPrint": "...your value...",
  "Elasticsearch:Password": "...your value...",  
}

The .NET Solution

This is a screenshot of the solution. We highlighted some of the most important files for you to explore and look at.

---

Here are some screenshots of the tests included in the project. This project tries to follow TDD an uses a test-first approach. The tests are meant to show how to use the library and to teach of the available features.

Click here to see the source code of the test.

Always make sure to look at the output window to see details about the execution. 👀

Click here to see the source code of the test.

How to add the Elasticsearch adapter to your Kernel Memory project

In order to add the Elasticsearch adapter to your project you first need to add a reference to the Freemindlabs.KernelMemory.Elasticsearch NuGet package.

> dotnet add package Freemindlabs.KernelMemory.Elasticsearch

Then you can chose to use one of the WithElasticsearch extensions methods of the interface IKernelMemoryBuilder.

// From Program.cs of the Service project of the Kernel Memory repository. Line 86.

[..]
// Loads the Elasticsearch configuration
var esConfig = config.GetServiceConfig<ElasticsearchConfig>(appBuilder.Configuration, "ElasticsearchVectorDb");

// Inject memory client and its dependencies
// Note: pass the current service collection to the builder, in order to start the pipeline handlers
IKernelMemory memory = new KernelMemoryBuilder(appBuilder.Services)
    .FromAppSettings()
    // .With...() // in case you need to set something not already defined by `.FromAppSettings()`
    .WithElasticsearch(esConfig) // <--- this
    .Build();

appBuilder.Services.AddSingleton(memory);

// Build .NET web app as usual
var app = appBuilder.Build();
[..]

Resources

1. 🔥 How to build a Kernel Memory connector and use Elasticsearch as vector database - Part 1

   1. To be relocated and published officially on Microsoft's devblogs for Semantic kernel.

2. A Quick Introduction to Vector Search

3. Elasticsearch Hybrid Search

4. Elastic's official docs on the client.

   1. NEST 7.17: https://www.elastic.co/guide/en/elasticsearch/client/net-api/7.17/nest-getting-started.html
   2. New client 8.9: https://www.elastic.co/guide/en/elasticsearch/client/net-api/8.9/introduction.html
      1. This client is not yet feature complete.
         1. Look here for details: https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/release-notes-8.0.0.html
      2. In addition, the docs are not up to date. For some stuff we need to lok at NEST's docs.

5. Elasticsearch.net GitHub repository

6. Semantic Kernel/Memory-Kernel

   1. Introduction to Semantic Memory (feat. Devis Lucato) | Semantic Kernel
   2. 11.29.2023 - Semantic Kernel Office Hours (US/Europe Region)