-
Notifications
You must be signed in to change notification settings - Fork 3.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Semantic Memory service start [extraction] #1965
Merged
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,77 @@ | ||
| # Semantic Memory Service | ||
|
|
||
| Semantic Memory service allows to index/store and query your data using natural language. | ||
|
|
||
| The solution is divided in two main areas: **Encoding** and **Retrieval**. | ||
|
|
||
| # Encoding | ||
|
|
||
| The encoding phase allows to ingest data and index it, using Embeddings and LLMs. | ||
|
|
||
| Documents are encoded using one or more "data pipelines" where consecutive | ||
| "handlers" take the input and process it, turning raw data into memories. | ||
|
|
||
| Pipelines can be customized, and they typically consist of: | ||
|
|
||
| * **storage**: store a copy of the document (if necessary, copies can be deleted | ||
| after processing). | ||
| * text **extraction**: extract text from documents, presentations, etc. | ||
| * text **partitioning**: chunk the text in small blocks. | ||
| * text **indexing**: calculate embedding for each text block, store the embedding | ||
| with a reference to the original document. | ||
|
|
||
| ## Runtime mode | ||
|
|
||
| Encoding can run **in process**, e.g. running all the handlers synchronously, | ||
| in real time, as soon as some content is loaded/uploaded. | ||
| In this case the upload process and handlers must be written in the same | ||
| language, e.g. C#. | ||
|
|
||
| Encoding can also run **as a distributed service**, deployed locally or in | ||
| the cloud. This mode provides some important benefits: | ||
|
|
||
| * **Handlers can be written in different languages**, e.g. extract | ||
| data using Python libraries, index using C#, etc. This can be useful when | ||
| working with file types supported better by specific libraries available | ||
| only in some programming language like Python. | ||
| * Content ingestion can be started using a **web service**. The repository | ||
| contains a web service ready to use in C# and Python (work in progress). | ||
| The web service can also be used by Copilot Chat, to store data and | ||
| search for answers. | ||
| * Content processing runs **asynchronously** in the background, allowing | ||
| to process several files in parallel, with support for retry logic. | ||
|
|
||
| # Retrieval | ||
|
|
||
| Memories can be retrieved using natural language queries. The service | ||
| supports also RAG, generating answers using prompts, relevant memories, | ||
dluc marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| and plugins. | ||
|
|
||
| Similar to the encoding process, retrieval is available as a library and | ||
| as a web service. | ||
|
|
||
| # Folder structure | ||
|
|
||
| * `handlers-dotnet`: set of reusable .NET handlers for typical data pipelines. | ||
| You can use these in production, or use them as a starting point for your | ||
| custom business logic. | ||
| * `handlers-python`: set of reusable Python handlers for typical data pipelines. | ||
| You can use these in production, or use them as a starting point for your | ||
| custom business logic. | ||
| * `webservice-dotnet`: C# web service to upload documents and search memories. | ||
| * `webservice-python`: Python web service to upload documents and search memories. | ||
| * `lib-dotnet`: reusable libraries for C# webservices and handlers. | ||
| * `lib-python`: reusable libraries for python webservices and handlers. | ||
| * `tools`: command line tools, e.g. scripts to start RabbitMQ locally. | ||
| * `samples`: samples showing how to upload files, how to use encoder/retrieval, etc. | ||
| * `upload-one-file.sh`: Bash command line tool to upload one document to the | ||
| memory web service. | ||
| * `FileImportExamples`: C# examples showing how to upload multiple files to the | ||
| memory web service, using different approaches: | ||
| * **in process**, synchronously, without asynchronous distributed queues | ||
| * **multipart form** upload to web service | ||
| * **Semantic Memory client** (work in progress) | ||
| * `CustomHandlerSample`: C# example of a console app running a custom pipeline handler. | ||
| * `TextExtractionFromDocsExamples`: simple C# console app to test text | ||
| extraction from PDF and MSWord files. | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,102 @@ | ||
| | ||
| Microsoft Visual Studio Solution File, Format Version 12.00 | ||
|
|
||
| Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Handlers", "handlers-dotnet\Handlers.csproj", "{9A23325F-FCF5-4BF7-9413-D325A779AE42}" | ||
| EndProject | ||
|
|
||
| Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "lib", "lib", "{8E36E21B-00CC-4B85-98FB-6CB2CA5F978C}" | ||
| EndProject | ||
| Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Configuration", "lib-dotnet\Configuration\Configuration.csproj", "{DB5CA047-1184-4878-B1FC-0C17C9FAE7AA}" | ||
| EndProject | ||
| Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "DataFormats.Office", "lib-dotnet\DataFormats.Office\DataFormats.Office.csproj", "{174E9AA0-E379-4B5C-8125-54CD4EE57ADF}" | ||
| EndProject | ||
| Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "DataFormats.Pdf", "lib-dotnet\DataFormats.Pdf\DataFormats.Pdf.csproj", "{E383AB2F-CFCD-4EAF-A612-85E7FC85667F}" | ||
| EndProject | ||
| Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Diagnostics", "lib-dotnet\Diagnostics\Diagnostics.csproj", "{60130AA3-B980-4DF6-8BFF-6F0CF09AD629}" | ||
| EndProject | ||
| Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Storage", "lib-dotnet\Storage\Storage.csproj", "{E7906A61-09F5-44D4-ACDC-1486E72B4775}" | ||
| EndProject | ||
|
|
||
| Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "samples", "samples", "{6335B02C-9964-4B39-9795-C5F5F0392515}" | ||
| ProjectSection(SolutionItems) = preProject | ||
| samples\upload-one-file.sh = samples\upload-one-file.sh | ||
| EndProjectSection | ||
| EndProject | ||
| Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "FileImportExamples", "samples\FileImportExamples\FileImportExamples.csproj", "{ACD9F6E9-0B7E-4B69-9468-28C42657AB10}" | ||
| EndProject | ||
|
|
||
| Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "WebService", "webservice-dotnet\WebService.csproj", "{57663B9D-08FD-49F8-B1D3-20F2CA28A39C}" | ||
| EndProject | ||
|
|
||
| Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "tools", "tools", "{E097BACD-C329-4FF6-A4AC-260F7BA143FF}" | ||
| ProjectSection(SolutionItems) = preProject | ||
| tools\run-rabbitmq.sh = tools\run-rabbitmq.sh | ||
| EndProjectSection | ||
| EndProject | ||
| Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "root", "root", "{6EF76FD8-4C35-4370-8539-5DDF45357A50}" | ||
| ProjectSection(SolutionItems) = preProject | ||
| README.md = README.md | ||
| EndProjectSection | ||
| EndProject | ||
| Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "TextExtractionFromDocsExamples", "samples\TextExtractionFromDocsExamples\TextExtractionFromDocsExamples.csproj", "{BF98AFF3-8997-400B-A1DB-E7146A118DB6}" | ||
| EndProject | ||
| Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "CustomHandlerExample", "samples\CustomHandlerExample\CustomHandlerExample.csproj", "{15CEB46F-AE80-4D0F-85E7-1F666078874D}" | ||
| EndProject | ||
| Global | ||
| GlobalSection(SolutionConfigurationPlatforms) = preSolution | ||
| Debug|Any CPU = Debug|Any CPU | ||
| Release|Any CPU = Release|Any CPU | ||
| EndGlobalSection | ||
| GlobalSection(NestedProjects) = preSolution | ||
| {E7906A61-09F5-44D4-ACDC-1486E72B4775} = {8E36E21B-00CC-4B85-98FB-6CB2CA5F978C} | ||
| {60130AA3-B980-4DF6-8BFF-6F0CF09AD629} = {8E36E21B-00CC-4B85-98FB-6CB2CA5F978C} | ||
| {DB5CA047-1184-4878-B1FC-0C17C9FAE7AA} = {8E36E21B-00CC-4B85-98FB-6CB2CA5F978C} | ||
| {ACD9F6E9-0B7E-4B69-9468-28C42657AB10} = {6335B02C-9964-4B39-9795-C5F5F0392515} | ||
| {174E9AA0-E379-4B5C-8125-54CD4EE57ADF} = {8E36E21B-00CC-4B85-98FB-6CB2CA5F978C} | ||
| {E383AB2F-CFCD-4EAF-A612-85E7FC85667F} = {8E36E21B-00CC-4B85-98FB-6CB2CA5F978C} | ||
| {BF98AFF3-8997-400B-A1DB-E7146A118DB6} = {6335B02C-9964-4B39-9795-C5F5F0392515} | ||
| {15CEB46F-AE80-4D0F-85E7-1F666078874D} = {6335B02C-9964-4B39-9795-C5F5F0392515} | ||
| EndGlobalSection | ||
| GlobalSection(ProjectConfigurationPlatforms) = postSolution | ||
| {E7906A61-09F5-44D4-ACDC-1486E72B4775}.Debug|Any CPU.ActiveCfg = Debug|Any CPU | ||
| {E7906A61-09F5-44D4-ACDC-1486E72B4775}.Debug|Any CPU.Build.0 = Debug|Any CPU | ||
| {E7906A61-09F5-44D4-ACDC-1486E72B4775}.Release|Any CPU.ActiveCfg = Release|Any CPU | ||
| {E7906A61-09F5-44D4-ACDC-1486E72B4775}.Release|Any CPU.Build.0 = Release|Any CPU | ||
| {60130AA3-B980-4DF6-8BFF-6F0CF09AD629}.Debug|Any CPU.ActiveCfg = Debug|Any CPU | ||
| {60130AA3-B980-4DF6-8BFF-6F0CF09AD629}.Debug|Any CPU.Build.0 = Debug|Any CPU | ||
| {60130AA3-B980-4DF6-8BFF-6F0CF09AD629}.Release|Any CPU.ActiveCfg = Release|Any CPU | ||
| {60130AA3-B980-4DF6-8BFF-6F0CF09AD629}.Release|Any CPU.Build.0 = Release|Any CPU | ||
| {DB5CA047-1184-4878-B1FC-0C17C9FAE7AA}.Debug|Any CPU.ActiveCfg = Debug|Any CPU | ||
| {DB5CA047-1184-4878-B1FC-0C17C9FAE7AA}.Debug|Any CPU.Build.0 = Debug|Any CPU | ||
| {DB5CA047-1184-4878-B1FC-0C17C9FAE7AA}.Release|Any CPU.ActiveCfg = Release|Any CPU | ||
| {DB5CA047-1184-4878-B1FC-0C17C9FAE7AA}.Release|Any CPU.Build.0 = Release|Any CPU | ||
| {ACD9F6E9-0B7E-4B69-9468-28C42657AB10}.Debug|Any CPU.ActiveCfg = Debug|Any CPU | ||
| {ACD9F6E9-0B7E-4B69-9468-28C42657AB10}.Debug|Any CPU.Build.0 = Debug|Any CPU | ||
| {ACD9F6E9-0B7E-4B69-9468-28C42657AB10}.Release|Any CPU.ActiveCfg = Release|Any CPU | ||
| {ACD9F6E9-0B7E-4B69-9468-28C42657AB10}.Release|Any CPU.Build.0 = Release|Any CPU | ||
| {9A23325F-FCF5-4BF7-9413-D325A779AE42}.Debug|Any CPU.ActiveCfg = Debug|Any CPU | ||
| {9A23325F-FCF5-4BF7-9413-D325A779AE42}.Debug|Any CPU.Build.0 = Debug|Any CPU | ||
| {9A23325F-FCF5-4BF7-9413-D325A779AE42}.Release|Any CPU.ActiveCfg = Release|Any CPU | ||
| {9A23325F-FCF5-4BF7-9413-D325A779AE42}.Release|Any CPU.Build.0 = Release|Any CPU | ||
| {174E9AA0-E379-4B5C-8125-54CD4EE57ADF}.Debug|Any CPU.ActiveCfg = Debug|Any CPU | ||
| {174E9AA0-E379-4B5C-8125-54CD4EE57ADF}.Debug|Any CPU.Build.0 = Debug|Any CPU | ||
| {174E9AA0-E379-4B5C-8125-54CD4EE57ADF}.Release|Any CPU.ActiveCfg = Release|Any CPU | ||
| {174E9AA0-E379-4B5C-8125-54CD4EE57ADF}.Release|Any CPU.Build.0 = Release|Any CPU | ||
| {E383AB2F-CFCD-4EAF-A612-85E7FC85667F}.Debug|Any CPU.ActiveCfg = Debug|Any CPU | ||
| {E383AB2F-CFCD-4EAF-A612-85E7FC85667F}.Debug|Any CPU.Build.0 = Debug|Any CPU | ||
| {E383AB2F-CFCD-4EAF-A612-85E7FC85667F}.Release|Any CPU.ActiveCfg = Release|Any CPU | ||
| {E383AB2F-CFCD-4EAF-A612-85E7FC85667F}.Release|Any CPU.Build.0 = Release|Any CPU | ||
| {57663B9D-08FD-49F8-B1D3-20F2CA28A39C}.Debug|Any CPU.ActiveCfg = Debug|Any CPU | ||
| {57663B9D-08FD-49F8-B1D3-20F2CA28A39C}.Debug|Any CPU.Build.0 = Debug|Any CPU | ||
| {57663B9D-08FD-49F8-B1D3-20F2CA28A39C}.Release|Any CPU.ActiveCfg = Release|Any CPU | ||
| {57663B9D-08FD-49F8-B1D3-20F2CA28A39C}.Release|Any CPU.Build.0 = Release|Any CPU | ||
| {BF98AFF3-8997-400B-A1DB-E7146A118DB6}.Debug|Any CPU.ActiveCfg = Debug|Any CPU | ||
| {BF98AFF3-8997-400B-A1DB-E7146A118DB6}.Debug|Any CPU.Build.0 = Debug|Any CPU | ||
| {BF98AFF3-8997-400B-A1DB-E7146A118DB6}.Release|Any CPU.ActiveCfg = Release|Any CPU | ||
| {BF98AFF3-8997-400B-A1DB-E7146A118DB6}.Release|Any CPU.Build.0 = Release|Any CPU | ||
| {15CEB46F-AE80-4D0F-85E7-1F666078874D}.Debug|Any CPU.ActiveCfg = Debug|Any CPU | ||
| {15CEB46F-AE80-4D0F-85E7-1F666078874D}.Debug|Any CPU.Build.0 = Debug|Any CPU | ||
| {15CEB46F-AE80-4D0F-85E7-1F666078874D}.Release|Any CPU.ActiveCfg = Release|Any CPU | ||
| {15CEB46F-AE80-4D0F-85E7-1F666078874D}.Release|Any CPU.Build.0 = Release|Any CPU | ||
| EndGlobalSection | ||
| EndGlobal |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Comment for overall PR: Where are all the tests?