Skip to content

Python: .Net: MEVD Preview 2 APR 2025 #11028

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 87 commits into from
Apr 29, 2025
Merged

Python: .Net: MEVD Preview 2 APR 2025 #11028

merged 87 commits into from
Apr 29, 2025

Conversation

westey-m
Copy link
Contributor

Motivation and Context

Description

Contribution Checklist

roji and others added 19 commits January 24, 2025 20:29
@roji
.Net: Remove DeleteRecordOptions and UpsertRecordOptions from MEVD (#…
bd9b83a 
…10192)

Closes #10172

/cc @westey-m
@roji
.Net MEVD: LINQ-based filtering (#10273)
2434faf 
Implement LINQ-based vector search filtering
    
Closes #10156
Does most of #10194
@roji
.Net: Refactor vector store integration tests (#10556)
a9abe9a 
Follup-up to #10273
@roji
.Net: Obsolete MEVD connectors collection factories (#10590)
3a36d40 
Closes #10534
@roji
.Net: Rename VectorSearchOptions.NewFilter to Filter (#10790)
10cde20 
And the obsoleted Filter to OldFilter. Continues #10273.
@westey-m @moonbox3
@dmytrostruk @crickman @dependabot @TaoChenOSU @RogerBarreto @markwallace-microsoft @esttenorio @davidpene @ThDuquennoy @TDuquennoy @eavanvalkenburg @lossyrob @drdrew42 @SergeyMenshykh @jenfoxbot @alliscode @shethaadit @ymuichiro @baywet @davidatorres @gewarren @atiq-bs23 @atiqur-bs-23 @fabianwilliams @RamType0 @lastcoolnameleft @gtang31-te @eiriktsarpalis @tommasodotNET @dluc @joslat @cplemm @stephentoub @musale @ross-p-smith
Python: Preb1 merge from main 1 (#10805)
865d67e 
### Description

Merging changes from main and resolving merge conflicts

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [ ] The code builds clean without any errors or warnings
- [ ] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [ ] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄

---------

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Vincent Biret <vibiret@microsoft.com>
Co-authored-by: Evan Mattson <35585003+moonbox3@users.noreply.github.com>
Co-authored-by: Dmytro Struk <13853051+dmytrostruk@users.noreply.github.com>
Co-authored-by: Chris <66376200+crickman@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Tao Chen <taochen@microsoft.com>
Co-authored-by: Roger Barreto <19890735+RogerBarreto@users.noreply.github.com>
Co-authored-by: Mark Wallace <127216156+markwallace-microsoft@users.noreply.github.com>
Co-authored-by: Estefanía Tenorio <8483207+esttenorio@users.noreply.github.com>
Co-authored-by: davidpene <d.c.pene@gmail.com>
Co-authored-by: ThDuquennoy <tduquennoy@chapsvision.com>
Co-authored-by: Thomas DUQUENNOY <Thomas.DUQUENNOY@sinequa.com>
Co-authored-by: Eduard van Valkenburg <eavanvalkenburg@users.noreply.github.com>
Co-authored-by: Evan Mattson <evan.mattson@microsoft.com>
Co-authored-by: Rob Emanuele <rdemanuele@gmail.com>
Co-authored-by: K. Andrew Parker <andrew+github@rationarium.org>
Co-authored-by: SergeyMenshykh <68852919+SergeyMenshykh@users.noreply.github.com>
Co-authored-by: jenfoxbot <jen.fox@microsoft.com>
Co-authored-by: Ben Thomas <ben.thomas@microsoft.com>
Co-authored-by: Ben Thomas <bentho@microsoft.com>
Co-authored-by: Adit Sheth <shethaadit@gmail.com>
Co-authored-by: Adit Sheth <adsheth@microsoft.com>
Co-authored-by: ふぁー <47295014+ymuichiro@users.noreply.github.com>
Co-authored-by: Vincent Biret <vibiret@microsoft.com>
Co-authored-by: David A. Torres <10944960+davidatorres@users.noreply.github.com>
Co-authored-by: Genevieve Warren <24882762+gewarren@users.noreply.github.com>
Co-authored-by: Atiqur Rahman Foyshal <113086917+atiq-bs23@users.noreply.github.com>
Co-authored-by: Md. Atiqur Rahman Foyshal <atiqur.foyshal@brainstation-23.com>
Co-authored-by: Fabian Williams <92543063+fabianwilliams@users.noreply.github.com>
Co-authored-by: fabian.williams@microsoft.com <fabianwilliams@users.noreply.github.com>
Co-authored-by: Ram.Type-0 <39725073+RamType0@users.noreply.github.com>
Co-authored-by: Tommy Falgout <tommy@lastcoolnameleft.com>
Co-authored-by: Gary Tang <113477176+gtang31-te@users.noreply.github.com>
Co-authored-by: Eirik Tsarpalis <eirik.tsarpalis@gmail.com>
Co-authored-by: Tommaso Stocchi <tstocchi@microsoft.com>
Co-authored-by: Chris Rickman <crickman@microsoft.com>
Co-authored-by: Devis Lucato <dluc@users.noreply.github.com>
Co-authored-by: K. Andrew Parker <kparker@citytech.cuny.edu>
Co-authored-by: Jose Luis Latorre Millas <joslat@gmail.com>
Co-authored-by: Carsten Lemm <clemm@microsoft.com>
Co-authored-by: Stephen Toub <stoub@microsoft.com>
Co-authored-by: Musale Martin <martinmshale@gmail.com>
Co-authored-by: Ross Smith <ross-p-smith@users.noreply.github.com>
@westey-m
Revert "Python: Preb1 merge from main 1 (#10805)"
fd0c55c 
This reverts commit 865d67e.
@westey-m
Merge branch 'main' into feature-vector-data-preb1
be79369
@westey-m
.Net: Hybrid Search (#10814)
0e8adb7 
### Motivation and Context

Adding hybrid search to March release branch

### Description

Adding hybrid search to March release branch

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [ ] The code builds clean without any errors or warnings
- [ ] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [ ] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
@adamsitnik @westey-m @roji
.Net: IVectorStore implementation for Azure SQL (#10623)
7aa7d29 
fixes #10416

---------

Co-authored-by: westey <164392973+westey-m@users.noreply.github.com>
Co-authored-by: Shay Rojansky <roji@roji.org>
@dmytrostruk
Enable integration tests on Windows
864b707
@dmytrostruk
Merge branch 'feature-vector-data-preb1' of https://github.com/micros…
9e068b2 
…oft/semantic-kernel into feature-vector-data-preb1
@dmytrostruk
Revert "Enable integration tests on Windows"
fd53ae8 
This reverts commit 864b707.
@adamsitnik @westey-m
.Net MEVD: Strongly-typed property selectors (#10850)
a14ad8c 
fixes #10420

---------

Co-authored-by: westey <164392973+westey-m@users.noreply.github.com>
@dmytrostruk
Updated integration test pipeline to use net8.0 target framework
261fec4
@dmytrostruk
.Net: [Feature Branch] Vector Store Logging (#10865)
3fca31f 
### Motivation and Context

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->

Related: #10596

This PR enhances the vector store functionality by adding builder
pattern implementations and logging decorators for key vector data
interfaces, along with supporting utilities and DI integration.

Interfaces handled:
- `IKeywordHybridSearch<TRecord>`
- `IVectorizableTextSearch<TRecord>`
- `IVectorizedSearch<TRecord>`
- `IVectorStore`
- `IVectorStoreRecordCollection<TKey, TRecord>`

Class types added:
- Builders: implement a pipeline pattern with `Use` and `Build` methods.
- Logging Extensions: provide reusable logging methods for tasks and
enumerables, handling success, failure and cancellation.
- Logging Decorators: wrap the interfaces to add logging by using
`LoggingExtensions` reusable logging methods.
- Builder Extensions: Add `AsBuilder` methods to convert service
instances into builders.
- Logging Builder Extensions: Add `UseLogging` methods to integrate
logging into builder pipelines.
- Service Collection Extensions: Add `Add{T}` and `AddKeyed{T}` methods
for DI registration with configurable lifetimes.

Other changes:
- Split `Verify` class into `Verify` and `KernelVerify` in order to be
able to use `Verify` methods in `Microsoft.Extensions.VectorData`
package without a reference to Semantic Kernel specific logic, like
`ValidPluginName`, `ValidFunctionName` etc.
- Added unit tests for new classes and methods.
- Added usage example in `Concepts/Memory` folder.

### Usage example
Logging with manual registration:
```csharp
var vectorStore = new InMemoryVectorStore()
    .AsBuilder()
    .UseLogging(this.LoggerFactory)
    .Build();
```

Logging with DI:
```csharp
serviceCollection.AddInMemoryVectorStore();

serviceCollection
    .AddVectorStore(s => s.GetRequiredService<InMemoryVectorStore>())
    .UseLogging(this.LoggerFactory);
```

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
@dmytrostruk
Merge branch 'main' into feature-vector-data-telemetry
1511ab9
@roji
.Net: Change Sqlite connector to accept connection string instead of …
ce572c8 
…DbConnection (#10972)

Closes #10454
@roji
.Net: [TINY] Document the default thread-safety expectation for the M…
89aab09 
…EVD abstractions (#10991)

Closes #10454
@markwallace-microsoft markwallace-microsoft added .NET Issue or Pull requests regarding .NET code kernel Issues or pull requests impacting the core kernel memory labels Mar 18, 2025
@github-actions github-actions bot changed the title MEVD Preview VNext (DRAFT) .Net: MEVD Preview VNext (DRAFT) Mar 18, 2025
@westey-m
.Net: Fix Integration tests and a few small bugs. (#11778)
a881464 
### Motivation and Context

Fix AzureAISearch tests and two small bugs.
Improve MongoDB tests with a small fix too.
Fix CosmosNoSQL integration tests
Fixes for postgres and mongodb
Fixes for redis

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [ ] The code builds clean without any errors or warnings
- [ ] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [ ] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
@westey-m westey-m changed the title Python: .Net: MEVD Preview VNext (DRAFT) Python: .Net: MEVD Preview 2 APR 2025 Apr 28, 2025
@westey-m westey-m marked this pull request as ready for review April 28, 2025 21:41
@westey-m westey-m requested a review from a team as a code owner April 28, 2025 21:41
@westey-m
.Net: Small InMemory bug fix. (#11799)
87dd8b1 
### Motivation and Context

We changed the storage type of the InMemory store to contain a wrapper
with both
the original record and generated vectors, and this code was returning
the wrapper
instead of the original record on get.

### Description

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [ ] The code builds clean without any errors or warnings
- [ ] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [ ] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
@westey-m
.Net: Update vector store readme's (#11801)
fe58faa 
### Motivation and Context

We have some broken links in the readme's and they are still referring
to memory store which is being replaced.

### Description

- Updated readme files to not advertise memory store anymore.
- Updated readme files to point to learn site docs.

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [ ] The code builds clean without any errors or warnings
- [ ] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [ ] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
@westey-m
.Net: Try to reduce mongodb test flakiness (#11802)
454f368 
### Motivation and Context

The docker container for mongo db is often not accessible when running
the tests

### Description

- Switching to the mongo db test container to try and reduce the
flakiness of the tests
- Adding retries

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [ ] The code builds clean without any errors or warnings
- [ ] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [ ] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
@westey-m westey-m added this pull request to the merge queue Apr 29, 2025
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Apr 29, 2025
@westey-m westey-m added this pull request to the merge queue Apr 29, 2025
Merged via the queue into main with commit 68b11d6 Apr 29, 2025
20 checks passed
@westey-m westey-m deleted the feature-vector-data-preb2 branch April 29, 2025 15:43
@westey-m westey-m restored the feature-vector-data-preb2 branch April 29, 2025 19:21
glorious-beard pushed a commit to glorious-beard/semantic-kernel that referenced this pull request May 6, 2025
@westey-m @roji
@moonbox3 @dmytrostruk @crickman @dependabot @TaoChenOSU @RogerBarreto @markwallace-microsoft @esttenorio @davidpene @ThDuquennoy @TDuquennoy @eavanvalkenburg @lossyrob @drdrew42 @SergeyMenshykh @jenfoxbot @alliscode @shethaadit @ymuichiro @baywet @davidatorres @gewarren @atiq-bs23 @atiqur-bs-23 @fabianwilliams @RamType0 @lastcoolnameleft @gtang31-te @eiriktsarpalis @tommasodotNET @dluc @joslat @cplemm @stephentoub @musale @ross-p-smith @adamsitnik @rracanicci @unsafecode @QuocDatHoang @DatHoang83 @damieng @glorious-beard
Python: .Net: MEVD Preview 2 APR 2025 (microsoft#11028)
3fc2904 
### Motivation and Context

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->

### Description

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [ ] The code builds clean without any errors or warnings
- [ ] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [ ] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄

---------

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Vincent Biret <vibiret@microsoft.com>
Co-authored-by: Shay Rojansky <roji@roji.org>
Co-authored-by: Evan Mattson <35585003+moonbox3@users.noreply.github.com>
Co-authored-by: Dmytro Struk <13853051+dmytrostruk@users.noreply.github.com>
Co-authored-by: Chris <66376200+crickman@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Tao Chen <taochen@microsoft.com>
Co-authored-by: Roger Barreto <19890735+RogerBarreto@users.noreply.github.com>
Co-authored-by: Mark Wallace <127216156+markwallace-microsoft@users.noreply.github.com>
Co-authored-by: Estefanía Tenorio <8483207+esttenorio@users.noreply.github.com>
Co-authored-by: davidpene <d.c.pene@gmail.com>
Co-authored-by: ThDuquennoy <tduquennoy@chapsvision.com>
Co-authored-by: Thomas DUQUENNOY <Thomas.DUQUENNOY@sinequa.com>
Co-authored-by: Eduard van Valkenburg <eavanvalkenburg@users.noreply.github.com>
Co-authored-by: Evan Mattson <evan.mattson@microsoft.com>
Co-authored-by: Rob Emanuele <rdemanuele@gmail.com>
Co-authored-by: K. Andrew Parker <andrew+github@rationarium.org>
Co-authored-by: SergeyMenshykh <68852919+SergeyMenshykh@users.noreply.github.com>
Co-authored-by: jenfoxbot <jen.fox@microsoft.com>
Co-authored-by: Ben Thomas <ben.thomas@microsoft.com>
Co-authored-by: Ben Thomas <bentho@microsoft.com>
Co-authored-by: Adit Sheth <shethaadit@gmail.com>
Co-authored-by: Adit Sheth <adsheth@microsoft.com>
Co-authored-by: ふぁー <47295014+ymuichiro@users.noreply.github.com>
Co-authored-by: Vincent Biret <vibiret@microsoft.com>
Co-authored-by: David A. Torres <10944960+davidatorres@users.noreply.github.com>
Co-authored-by: Genevieve Warren <24882762+gewarren@users.noreply.github.com>
Co-authored-by: Atiqur Rahman Foyshal <113086917+atiq-bs23@users.noreply.github.com>
Co-authored-by: Md. Atiqur Rahman Foyshal <atiqur.foyshal@brainstation-23.com>
Co-authored-by: Fabian Williams <92543063+fabianwilliams@users.noreply.github.com>
Co-authored-by: fabian.williams@microsoft.com <fabianwilliams@users.noreply.github.com>
Co-authored-by: Ram.Type-0 <39725073+RamType0@users.noreply.github.com>
Co-authored-by: Tommy Falgout <tommy@lastcoolnameleft.com>
Co-authored-by: Gary Tang <113477176+gtang31-te@users.noreply.github.com>
Co-authored-by: Eirik Tsarpalis <eirik.tsarpalis@gmail.com>
Co-authored-by: Tommaso Stocchi <tstocchi@microsoft.com>
Co-authored-by: Chris Rickman <crickman@microsoft.com>
Co-authored-by: Devis Lucato <dluc@users.noreply.github.com>
Co-authored-by: K. Andrew Parker <kparker@citytech.cuny.edu>
Co-authored-by: Jose Luis Latorre Millas <joslat@gmail.com>
Co-authored-by: Carsten Lemm <clemm@microsoft.com>
Co-authored-by: Stephen Toub <stoub@microsoft.com>
Co-authored-by: Musale Martin <martinmshale@gmail.com>
Co-authored-by: Ross Smith <ross-p-smith@users.noreply.github.com>
Co-authored-by: Adam Sitnik <adam.sitnik@gmail.com>
Co-authored-by: Rodrigo Martins Racanicci <59115712+rracanicci@users.noreply.github.com>
Co-authored-by: unsafecode <r.chiodaroli@live.com>
Co-authored-by: QuocDatHoang <dathq.vn@gmail.com>
Co-authored-by: Dat Hoang Quoc <DAT@jsworldmedia.com>
Co-authored-by: Damien Guard <damien@envytech.co.uk>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@SergeyMenshykh SergeyMenshykh

@markwallace-microsoft markwallace-microsoft

Assignees
No one assigned
Labels
documentation kernel.core kernel Issues or pull requests impacting the core kernel memory .NET Issue or Pull requests regarding .NET code
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
8 participants
@westey-m @SergeyMenshykh @markwallace-microsoft @roji @adamsitnik @dmytrostruk @dluc @damieng