Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions dotnet/src/Connectors/Connectors.Memory.Chroma/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,6 @@ docker-compose up -d --build

3. Use Semantic Kernel with Chroma, using server local endpoint `http://localhost:8000`:

> See [Example 14](../../../samples/Concepts/Memory/SemanticTextMemory_Building.cs) and [Example 15](../../../samples/Concepts/Memory/TextMemoryPlugin_MultipleMemoryStore.cs) for more memory usage examples with the kernel.

```csharp
const string endpoint = "http://localhost:8000";

Expand Down
9 changes: 4 additions & 5 deletions dotnet/src/Connectors/Connectors.Memory.Kusto/README.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,12 @@
# Microsoft.SemanticKernel.Connectors.Kusto

This connector uses [Azure Data Explorer (Kusto)](https://learn.microsoft.com/en-us/azure/data-explorer/) to implement Semantic Memory.
This connector uses [Azure Data Explorer (Kusto)](https://learn.microsoft.com/azure/data-explorer/) to implement Semantic Memory.

## Quick Start

1. Create a cluster and database in Azure Data Explorer (Kusto) - see https://learn.microsoft.com/en-us/azure/data-explorer/create-cluster-and-database?tabs=free
1. Create a cluster and database in Azure Data Explorer (Kusto) - see https://learn.microsoft.com/azure/data-explorer/create-cluster-and-database?tabs=free

2. To use Kusto as a semantic memory store, use the following code:
> See [Example 14](../../../samples/Concepts/Memory/SemanticTextMemory_Building.cs) and [Example 15](../../../samples/Concepts/Memory/TextMemoryPlugin_MultipleMemoryStore.cs) for more memory usage examples with the kernel.

```csharp
using Kusto.Data;
Expand Down Expand Up @@ -37,9 +36,9 @@ The function is called `series_cosine_similarity_fl` and is located in the `Func

Kusto is an append-only store. This means that when a fact is updated, the old fact is not deleted.
This isn't a problem for the semantic memory connector, as it always utilizes the most recent fact.
This is made possible by using the [arg_max](https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/arg-max-aggfunction) aggregation function in conjunction with the [ingestion_time](https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/ingestiontimefunction) function.
This is made possible by using the [arg_max](https://learn.microsoft.com/azure/data-explorer/kusto/query/arg-max-aggfunction) aggregation function in conjunction with the [ingestion_time](https://learn.microsoft.com/azure/data-explorer/kusto/query/ingestiontimefunction) function.
However, users manually querying the underlying table should be aware of this behavior.

### Authentication

Please note that the authentication used in the example above is not recommended for production use. You can find more details here: https://learn.microsoft.com/en-us/azure/data-explorer/kusto/api/connection-strings/kusto
Please note that the authentication used in the example above is not recommended for production use. You can find more details here: https://learn.microsoft.com/azure/data-explorer/kusto/api/connection-strings/kusto
1 change: 0 additions & 1 deletion dotnet/src/Connectors/Connectors.Memory.Milvus/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,6 @@ docker-compose up -d
```

3. Use Semantic Kernel with Milvus, connecting to `localhost` with the default (gRPC) port of 1536:
> See [Example 14](../../../samples/Concepts/Memory/SemanticTextMemory_Building.cs) and [Example 15](../../../samples/Concepts/Memory/TextMemoryPlugin_MultipleMemoryStore.cs) for more memory usage examples with the kernel.

```csharp
using MilvusMemoryStore memoryStore = new("localhost");
Expand Down
40 changes: 2 additions & 38 deletions dotnet/src/Connectors/Connectors.Memory.MongoDB/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,44 +6,8 @@ This connector uses [MongoDB Atlas Vector Search](https://www.mongodb.com/produc

1. Create [Atlas cluster](https://www.mongodb.com/docs/atlas/getting-started/)

2. Create a [collection](https://www.mongodb.com/docs/atlas/atlas-ui/collections/)
2. Create a Mongo DB Vector Store using instructions on the [Microsoft Learn site](https://learn.microsoft.com/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/mongodb-connector).

3. Create [Vector Search Index](https://www.mongodb.com/docs/atlas/atlas-vector-search/vector-search-overview/) for the collection. The index has to be defined on a field called `embedding`. For example:

```
{
"type": "vectorSearch",
"fields": [
{
"numDimensions": <number-of-dimensions>,
"path": "embedding",
"similarity": "euclidean | cosine | dotProduct",
"type": "vector"
}
]
}
```

4. Create the MongoDB memory store
> See [Example 14](../../../samples/Concepts/Memory/SemanticTextMemory_Building.cs) and [Example 15](../../../samples/Concepts/Memory/TextMemoryPlugin_MultipleMemoryStore.cs) for more memory usage examples with the kernel.

```csharp
var connectionString = "MONGODB ATLAS CONNECTION STRING"
MongoDBMemoryStore memoryStore = new(connectionString, "MyDatabase");

var embeddingGenerator = new OpenAITextEmbeddingGenerationService("text-embedding-ada-002", apiKey);

SemanticTextMemory textMemory = new(memoryStore, embeddingGenerator);

var memoryPlugin = kernel.ImportPluginFromObject(new TextMemoryPlugin(textMemory));
```
3. Use the [getting started instructions](https://learn.microsoft.com/semantic-kernel/concepts/vector-store-connectors/?pivots=programming-language-csharp#getting-started-with-vector-store-connectors) on the Microsoft Leearn site to learn more about using the vector store.

> Guide to find the connection string: https://www.mongodb.com/docs/manual/reference/connection-string/

## Important Notes

### Vector search indexes

In this version, vector search index management is outside of `MongoDBMemoryStore` scope.
Creation and maintenance of the indexes have to be done by the user. Please note that deleting a collection
(`memoryStore.DeleteCollectionAsync`) will delete the index as well.
56 changes: 2 additions & 54 deletions dotnet/src/Connectors/Connectors.Memory.Postgres/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,58 +37,6 @@ sk_demo=# CREATE EXTENSION vector;

See [this sample](../../../samples/Concepts/Memory/VectorStore_VectorSearch_MultiStore_Postgres.cs) for an example of using the vector store.

### Using PostgresMemoryStore
For more information on using Postgres as a vector store, see the [PostgresVectorStore](https://learn.microsoft.com/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector) documentation.

> See [Example 14](../../../samples/Concepts/Memory/SemanticTextMemory_Building.cs) and [Example 15](../../../samples/Concepts/Memory/TextMemoryPlugin_MultipleMemoryStore.cs) for more memory usage examples with the kernel.

```csharp
NpgsqlDataSourceBuilder dataSourceBuilder = new NpgsqlDataSourceBuilder("Host=localhost;Port=5432;Database=sk_demo;User Id=postgres;Password=mysecretpassword");
dataSourceBuilder.UseVector();
NpgsqlDataSource dataSource = dataSourceBuilder.Build();

var memoryWithPostgres = new MemoryBuilder()
.WithPostgresMemoryStore(dataSource, vectorSize: 1536/*, schema: "public" */)
.WithLoggerFactory(loggerFactory)
.WithOpenAITextEmbeddingGeneration("text-embedding-ada-002", apiKey)
.Build();

var memoryPlugin = kernel.ImportPluginFromObject(new TextMemoryPlugin(memoryWithPostgres));
```

### Create Index

> By default, pgvector performs exact nearest neighbor search, which provides perfect recall.

> You can add an index to use approximate nearest neighbor search, which trades some recall for performance. Unlike typical indexes, you will see different results for queries after adding an approximate index.

> Three keys to achieving good recall are:
>
> - Create the index after the table has some data
> - Choose an appropriate number of lists - a good place to start is rows / 1000 for up to 1M rows and sqrt(rows) for over 1M rows
> - When querying, specify an appropriate number of probes (higher is better for recall, lower is better for speed) - a good place to start is sqrt(lists)

Please read [the documentation](https://github.com/pgvector/pgvector#indexing) for more information.

Based on the data rows of your collection table, consider the following statement to create an index.

```sql
DO $$
DECLARE
collection TEXT;
c_count INTEGER;
BEGIN
SELECT 'REPLACE YOUR COLLECTION TABLE NAME' INTO collection;

-- Get count of records in collection
EXECUTE format('SELECT count(*) FROM public.%I;', collection) INTO c_count;

-- Create Index (https://github.com/pgvector/pgvector#indexing)
IF c_count > 10000000 THEN
EXECUTE format('CREATE INDEX %I ON public.%I USING ivfflat (embedding vector_cosine_ops) WITH (lists = %s);',
collection || '_ix', collection, ROUND(sqrt(c_count)));
ELSIF c_count > 10000 THEN
EXECUTE format('CREATE INDEX %I ON public.%I USING ivfflat (embedding vector_cosine_ops) WITH (lists = %s);',
collection || '_ix', collection, c_count / 1000);
END IF;
END $$;
```
Use the [getting started instructions](https://learn.microsoft.com/semantic-kernel/concepts/vector-store-connectors/?pivots=programming-language-csharp#getting-started-with-vector-store-connectors) on the Microsoft Leearn site to learn more about using the vector store.
17 changes: 2 additions & 15 deletions dotnet/src/Connectors/Connectors.Memory.Redis/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,19 +22,6 @@ Ways to get RediSearch:
docker run -d --name redis-stack-server -p 6379:6379 redis/redis-stack-server:latest
```

2. To use Redis as a semantic memory store:
> See [Example 14](../../../samples/Concepts/Memory/SemanticTextMemory_Building.cs) and [Example 15](../../../samples/Concepts/Memory/TextMemoryPlugin_MultipleMemoryStore.cs) for more memory usage examples with the kernel.
2. Create a Redis Vector Store using instructions on the [Microsoft Learn site](https://learn.microsoft.com/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/redis-connector).

```csharp
// ConnectionMultiplexer should be a singleton instance in your application, please consider to dispose of it when your application shuts down.
// See https://stackexchange.github.io/StackExchange.Redis/Basics#basic-usage
ConnectionMultiplexer connectionMultiplexer = await ConnectionMultiplexer.ConnectAsync("localhost:6379");
IDatabase database = connectionMultiplexer.GetDatabase();
RedisMemoryStore memoryStore = new RedisMemoryStore(database, vectorSize: 1536);

var embeddingGenerator = new OpenAITextEmbeddingGenerationService("text-embedding-ada-002", apiKey);

SemanticTextMemory textMemory = new(memoryStore, embeddingGenerator);

var memoryPlugin = kernel.ImportPluginFromObject(new TextMemoryPlugin(textMemory));
```
3. Use the [getting started instructions](https://learn.microsoft.com/semantic-kernel/concepts/vector-store-connectors/?pivots=programming-language-csharp#getting-started-with-vector-store-connectors) on the Microsoft Leearn site to learn more about using the vector store.
Loading