-
Notifications
You must be signed in to change notification settings - Fork 3k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Semantic Memory service start [extraction] (#1965)
First part of the encoding pipeline, with support for synchronous and asynchronous processing. Pipelines can run locally, storing data on disk and in the cloud using Azure blobs. Asynchronous processing supports RabbitMQ and Azure Queues.
- Loading branch information
Showing
79 changed files
with
4,245 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,77 @@ | ||
| # Semantic Memory Service | ||
|
|
||
| Semantic Memory service allows to index/store and query your data using natural language. | ||
|
|
||
| The solution is divided in two main areas: **Encoding** and **Retrieval**. | ||
|
|
||
| # Encoding | ||
|
|
||
| The encoding phase allows to ingest data and index it, using Embeddings and LLMs. | ||
|
|
||
| Documents are encoded using one or more "data pipelines" where consecutive | ||
| "handlers" take the input and process it, turning raw data into memories. | ||
|
|
||
| Pipelines can be customized, and they typically consist of: | ||
|
|
||
| * **storage**: store a copy of the document (if necessary, copies can be deleted | ||
| after processing). | ||
| * text **extraction**: extract text from documents, presentations, etc. | ||
| * text **partitioning**: chunk the text in small blocks. | ||
| * text **indexing**: calculate embedding for each text block, store the embedding | ||
| with a reference to the original document. | ||
|
|
||
| ## Runtime mode | ||
|
|
||
| Encoding can run **in process**, e.g. running all the handlers synchronously, | ||
| in real time, as soon as some content is loaded/uploaded. | ||
| In this case the upload process and handlers must be written in the same | ||
| language, e.g. C#. | ||
|
|
||
| Encoding can also run **as a distributed service**, deployed locally or in | ||
| the cloud. This mode provides some important benefits: | ||
|
|
||
| * **Handlers can be written in different languages**, e.g. extract | ||
| data using Python libraries, index using C#, etc. This can be useful when | ||
| working with file types supported better by specific libraries available | ||
| only in some programming language like Python. | ||
| * Content ingestion can be started using a **web service**. The repository | ||
| contains a web service ready to use in C# and Python (work in progress). | ||
| The web service can also be used by Copilot Chat, to store data and | ||
| search for answers. | ||
| * Content processing runs **asynchronously** in the background, allowing | ||
| to process several files in parallel, with support for retry logic. | ||
|
|
||
| # Retrieval | ||
|
|
||
| Memories can be retrieved using natural language queries. The service | ||
| supports also RAG, generating answers using prompts, relevant memories, | ||
| and plugins. | ||
|
|
||
| Similar to the encoding process, retrieval is available as a library and | ||
| as a web service. | ||
|
|
||
| # Folder structure | ||
|
|
||
| * `handlers-dotnet`: set of reusable .NET handlers for typical data pipelines. | ||
| You can use these in production, or use them as a starting point for your | ||
| custom business logic. | ||
| * `handlers-python`: set of reusable Python handlers for typical data pipelines. | ||
| You can use these in production, or use them as a starting point for your | ||
| custom business logic. | ||
| * `webservice-dotnet`: C# web service to upload documents and search memories. | ||
| * `webservice-python`: Python web service to upload documents and search memories. | ||
| * `lib-dotnet`: reusable libraries for C# webservices and handlers. | ||
| * `lib-python`: reusable libraries for python webservices and handlers. | ||
| * `tools`: command line tools, e.g. scripts to start RabbitMQ locally. | ||
| * `samples`: samples showing how to upload files, how to use encoder/retrieval, etc. | ||
| * `upload-one-file.sh`: Bash command line tool to upload one document to the | ||
| memory web service. | ||
| * `FileImportExamples`: C# examples showing how to upload multiple files to the | ||
| memory web service, using different approaches: | ||
| * **in process**, synchronously, without asynchronous distributed queues | ||
| * **multipart form** upload to web service | ||
| * **Semantic Memory client** (work in progress) | ||
| * `CustomHandlerSample`: C# example of a console app running a custom pipeline handler. | ||
| * `TextExtractionFromDocsExamples`: simple C# console app to test text | ||
| extraction from PDF and MSWord files. | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,102 @@ | ||
|
|
||
| Microsoft Visual Studio Solution File, Format Version 12.00 | ||
|
|
||
| Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Handlers", "handlers-dotnet\Handlers.csproj", "{9A23325F-FCF5-4BF7-9413-D325A779AE42}" | ||
| EndProject | ||
|
|
||
| Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "lib", "lib", "{8E36E21B-00CC-4B85-98FB-6CB2CA5F978C}" | ||
| EndProject | ||
| Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Configuration", "lib-dotnet\Configuration\Configuration.csproj", "{DB5CA047-1184-4878-B1FC-0C17C9FAE7AA}" | ||
| EndProject | ||
| Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "DataFormats.Office", "lib-dotnet\DataFormats.Office\DataFormats.Office.csproj", "{174E9AA0-E379-4B5C-8125-54CD4EE57ADF}" | ||
| EndProject | ||
| Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "DataFormats.Pdf", "lib-dotnet\DataFormats.Pdf\DataFormats.Pdf.csproj", "{E383AB2F-CFCD-4EAF-A612-85E7FC85667F}" | ||
| EndProject | ||
| Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Diagnostics", "lib-dotnet\Diagnostics\Diagnostics.csproj", "{60130AA3-B980-4DF6-8BFF-6F0CF09AD629}" | ||
| EndProject | ||
| Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Storage", "lib-dotnet\Storage\Storage.csproj", "{E7906A61-09F5-44D4-ACDC-1486E72B4775}" | ||
| EndProject | ||
|
|
||
| Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "samples", "samples", "{6335B02C-9964-4B39-9795-C5F5F0392515}" | ||
| ProjectSection(SolutionItems) = preProject | ||
| samples\upload-one-file.sh = samples\upload-one-file.sh | ||
| EndProjectSection | ||
| EndProject | ||
| Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "FileImportExamples", "samples\FileImportExamples\FileImportExamples.csproj", "{ACD9F6E9-0B7E-4B69-9468-28C42657AB10}" | ||
| EndProject | ||
|
|
||
| Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "WebService", "webservice-dotnet\WebService.csproj", "{57663B9D-08FD-49F8-B1D3-20F2CA28A39C}" | ||
| EndProject | ||
|
|
||
| Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "tools", "tools", "{E097BACD-C329-4FF6-A4AC-260F7BA143FF}" | ||
| ProjectSection(SolutionItems) = preProject | ||
| tools\run-rabbitmq.sh = tools\run-rabbitmq.sh | ||
| EndProjectSection | ||
| EndProject | ||
| Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "root", "root", "{6EF76FD8-4C35-4370-8539-5DDF45357A50}" | ||
| ProjectSection(SolutionItems) = preProject | ||
| README.md = README.md | ||
| EndProjectSection | ||
| EndProject | ||
| Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "TextExtractionFromDocsExamples", "samples\TextExtractionFromDocsExamples\TextExtractionFromDocsExamples.csproj", "{BF98AFF3-8997-400B-A1DB-E7146A118DB6}" | ||
| EndProject | ||
| Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "CustomHandlerExample", "samples\CustomHandlerExample\CustomHandlerExample.csproj", "{15CEB46F-AE80-4D0F-85E7-1F666078874D}" | ||
| EndProject | ||
| Global | ||
| GlobalSection(SolutionConfigurationPlatforms) = preSolution | ||
| Debug|Any CPU = Debug|Any CPU | ||
| Release|Any CPU = Release|Any CPU | ||
| EndGlobalSection | ||
| GlobalSection(NestedProjects) = preSolution | ||
| {E7906A61-09F5-44D4-ACDC-1486E72B4775} = {8E36E21B-00CC-4B85-98FB-6CB2CA5F978C} | ||
| {60130AA3-B980-4DF6-8BFF-6F0CF09AD629} = {8E36E21B-00CC-4B85-98FB-6CB2CA5F978C} | ||
| {DB5CA047-1184-4878-B1FC-0C17C9FAE7AA} = {8E36E21B-00CC-4B85-98FB-6CB2CA5F978C} | ||
| {ACD9F6E9-0B7E-4B69-9468-28C42657AB10} = {6335B02C-9964-4B39-9795-C5F5F0392515} | ||
| {174E9AA0-E379-4B5C-8125-54CD4EE57ADF} = {8E36E21B-00CC-4B85-98FB-6CB2CA5F978C} | ||
| {E383AB2F-CFCD-4EAF-A612-85E7FC85667F} = {8E36E21B-00CC-4B85-98FB-6CB2CA5F978C} | ||
| {BF98AFF3-8997-400B-A1DB-E7146A118DB6} = {6335B02C-9964-4B39-9795-C5F5F0392515} | ||
| {15CEB46F-AE80-4D0F-85E7-1F666078874D} = {6335B02C-9964-4B39-9795-C5F5F0392515} | ||
| EndGlobalSection | ||
| GlobalSection(ProjectConfigurationPlatforms) = postSolution | ||
| {E7906A61-09F5-44D4-ACDC-1486E72B4775}.Debug|Any CPU.ActiveCfg = Debug|Any CPU | ||
| {E7906A61-09F5-44D4-ACDC-1486E72B4775}.Debug|Any CPU.Build.0 = Debug|Any CPU | ||
| {E7906A61-09F5-44D4-ACDC-1486E72B4775}.Release|Any CPU.ActiveCfg = Release|Any CPU | ||
| {E7906A61-09F5-44D4-ACDC-1486E72B4775}.Release|Any CPU.Build.0 = Release|Any CPU | ||
| {60130AA3-B980-4DF6-8BFF-6F0CF09AD629}.Debug|Any CPU.ActiveCfg = Debug|Any CPU | ||
| {60130AA3-B980-4DF6-8BFF-6F0CF09AD629}.Debug|Any CPU.Build.0 = Debug|Any CPU | ||
| {60130AA3-B980-4DF6-8BFF-6F0CF09AD629}.Release|Any CPU.ActiveCfg = Release|Any CPU | ||
| {60130AA3-B980-4DF6-8BFF-6F0CF09AD629}.Release|Any CPU.Build.0 = Release|Any CPU | ||
| {DB5CA047-1184-4878-B1FC-0C17C9FAE7AA}.Debug|Any CPU.ActiveCfg = Debug|Any CPU | ||
| {DB5CA047-1184-4878-B1FC-0C17C9FAE7AA}.Debug|Any CPU.Build.0 = Debug|Any CPU | ||
| {DB5CA047-1184-4878-B1FC-0C17C9FAE7AA}.Release|Any CPU.ActiveCfg = Release|Any CPU | ||
| {DB5CA047-1184-4878-B1FC-0C17C9FAE7AA}.Release|Any CPU.Build.0 = Release|Any CPU | ||
| {ACD9F6E9-0B7E-4B69-9468-28C42657AB10}.Debug|Any CPU.ActiveCfg = Debug|Any CPU | ||
| {ACD9F6E9-0B7E-4B69-9468-28C42657AB10}.Debug|Any CPU.Build.0 = Debug|Any CPU | ||
| {ACD9F6E9-0B7E-4B69-9468-28C42657AB10}.Release|Any CPU.ActiveCfg = Release|Any CPU | ||
| {ACD9F6E9-0B7E-4B69-9468-28C42657AB10}.Release|Any CPU.Build.0 = Release|Any CPU | ||
| {9A23325F-FCF5-4BF7-9413-D325A779AE42}.Debug|Any CPU.ActiveCfg = Debug|Any CPU | ||
| {9A23325F-FCF5-4BF7-9413-D325A779AE42}.Debug|Any CPU.Build.0 = Debug|Any CPU | ||
| {9A23325F-FCF5-4BF7-9413-D325A779AE42}.Release|Any CPU.ActiveCfg = Release|Any CPU | ||
| {9A23325F-FCF5-4BF7-9413-D325A779AE42}.Release|Any CPU.Build.0 = Release|Any CPU | ||
| {174E9AA0-E379-4B5C-8125-54CD4EE57ADF}.Debug|Any CPU.ActiveCfg = Debug|Any CPU | ||
| {174E9AA0-E379-4B5C-8125-54CD4EE57ADF}.Debug|Any CPU.Build.0 = Debug|Any CPU | ||
| {174E9AA0-E379-4B5C-8125-54CD4EE57ADF}.Release|Any CPU.ActiveCfg = Release|Any CPU | ||
| {174E9AA0-E379-4B5C-8125-54CD4EE57ADF}.Release|Any CPU.Build.0 = Release|Any CPU | ||
| {E383AB2F-CFCD-4EAF-A612-85E7FC85667F}.Debug|Any CPU.ActiveCfg = Debug|Any CPU | ||
| {E383AB2F-CFCD-4EAF-A612-85E7FC85667F}.Debug|Any CPU.Build.0 = Debug|Any CPU | ||
| {E383AB2F-CFCD-4EAF-A612-85E7FC85667F}.Release|Any CPU.ActiveCfg = Release|Any CPU | ||
| {E383AB2F-CFCD-4EAF-A612-85E7FC85667F}.Release|Any CPU.Build.0 = Release|Any CPU | ||
| {57663B9D-08FD-49F8-B1D3-20F2CA28A39C}.Debug|Any CPU.ActiveCfg = Debug|Any CPU | ||
| {57663B9D-08FD-49F8-B1D3-20F2CA28A39C}.Debug|Any CPU.Build.0 = Debug|Any CPU | ||
| {57663B9D-08FD-49F8-B1D3-20F2CA28A39C}.Release|Any CPU.ActiveCfg = Release|Any CPU | ||
| {57663B9D-08FD-49F8-B1D3-20F2CA28A39C}.Release|Any CPU.Build.0 = Release|Any CPU | ||
| {BF98AFF3-8997-400B-A1DB-E7146A118DB6}.Debug|Any CPU.ActiveCfg = Debug|Any CPU | ||
| {BF98AFF3-8997-400B-A1DB-E7146A118DB6}.Debug|Any CPU.Build.0 = Debug|Any CPU | ||
| {BF98AFF3-8997-400B-A1DB-E7146A118DB6}.Release|Any CPU.ActiveCfg = Release|Any CPU | ||
| {BF98AFF3-8997-400B-A1DB-E7146A118DB6}.Release|Any CPU.Build.0 = Release|Any CPU | ||
| {15CEB46F-AE80-4D0F-85E7-1F666078874D}.Debug|Any CPU.ActiveCfg = Debug|Any CPU | ||
| {15CEB46F-AE80-4D0F-85E7-1F666078874D}.Debug|Any CPU.Build.0 = Debug|Any CPU | ||
| {15CEB46F-AE80-4D0F-85E7-1F666078874D}.Release|Any CPU.ActiveCfg = Release|Any CPU | ||
| {15CEB46F-AE80-4D0F-85E7-1F666078874D}.Release|Any CPU.Build.0 = Release|Any CPU | ||
| EndGlobalSection | ||
| EndGlobal |
Oops, something went wrong.