Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

.Net: Update Milvus memory connector to 2.3 #5593

Merged
merged 2 commits into from
Mar 25, 2024

Conversation

roji
Copy link
Member

@roji roji commented Mar 21, 2024

This updates the version of the lower-level Milvus SDK from 2.2 to 2.3 (recently released).

In addition, does the following changes:

  • Changes the Milvus integration tests to use the Milvus testcontainer module for .NET; this means that a Milvus container is brought up when the tests run, removing the need to manually configure an external one.
    • As part of this, I removed the skipping to make the Milvus tests run by default; they're very reliable as far as I can tell, but if you'd rather I roll back this change (so that tests are only executed explicitly as is typical in the repo), let me know and I'll do it.
  • Implement the IMemoryStore Upsert APIs over the new 2.3 Milvus upsert operation.
  • Allow specifying the consistency level of a MilvusMemoryStore on creation.

/cc @shawncal @lemillermicrosoft
/cc @stephentoub @luisquintanilla @SamMonoRT

@roji roji requested a review from a team as a code owner March 21, 2024 08:19
@markwallace-microsoft markwallace-microsoft added .NET Issue or Pull requests regarding .NET code kernel Issues or pull requests impacting the core kernel memory labels Mar 21, 2024
@github-actions github-actions bot changed the title Update Milvus memory connector to 2.3 .Net: Update Milvus memory connector to 2.3 Mar 21, 2024
Copy link
Member

@stephentoub stephentoub left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice. Thanks, Shay.

@SamMonoRT
Copy link

@roji - [Not blocking the PR, more for my understanding] - When I look at https://github.com/microsoft/semantic-kernel/actions/runs/8374907874/job/22931151816 for the test results page, I see it skipped over the Integration tests. How do you know if the test changes you have actually ran successfully?

@roji
Copy link
Member Author

roji commented Mar 21, 2024

@SamMonoRT yeah, that's a good question - I'm obviously not a maintainer on this repo but here's what I suspect is the situation.

The integration tests involve actually communicating with the service or vector database in question, and require the prior setup of that service/database; since that's hard to do for each and every service, I think the integration tests in this repo are executed manually, and it's up to the maintainer of each connector to ensure that everything works correctly (otherwise this build pipeline would have to start up all the supported vector databases, communicate to the Azure services, etc. etc.).

BTW testcontainers are meant to solve this exact problem - the tests themselves are responsible for fetching, configuring and managing the container for the service in question. This PR integrates the new Milvus testcontainer into the integration tests, but most other tests are still manual-only. Maybe with time, more connectors will integrate testcontainers into their test suite, and it would make sense to run the integration test suite in CI.

(of course, before submititng this PR I verified that all integration tests pass)

@markwallace-microsoft markwallace-microsoft added this pull request to the merge queue Mar 22, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Mar 22, 2024
@markwallace-microsoft markwallace-microsoft dismissed their stale review March 22, 2024 10:22

Changes cannot be merged until the integration tests are skipped by default

@markwallace-microsoft markwallace-microsoft added this pull request to the merge queue Mar 25, 2024
Merged via the queue into microsoft:main with commit 9ab9513 Mar 25, 2024
18 checks passed
@roji roji deleted the Milvus2.3 branch March 25, 2024 13:50
@roji
Copy link
Member Author

roji commented Mar 25, 2024

Thanks @markwallace-microsoft!

LudoCorporateShark pushed a commit to LudoCorporateShark/semantic-kernel that referenced this pull request Aug 25, 2024
This updates the version of the lower-level Milvus SDK from 2.2 to 2.3
([recently released](https://www.nuget.org/packages/Milvus.Client)).

In addition, does the following changes:
* Changes the Milvus integration tests to use the [Milvus testcontainer
module for .NET](https://testcontainers.com/modules/milvus/); this means
that a Milvus container is brought up when the tests run, removing the
need to manually configure an external one.
* As part of this, I removed the skipping to make the Milvus tests run
by default; they're very reliable as far as I can tell, but if you'd
rather I roll back this change (so that tests are only executed
explicitly as is typical in the repo), let me know and I'll do it.
* Implement the IMemoryStore Upsert APIs over the new 2.3 Milvus upsert
operation.
* Allow specifying the consistency level of a MilvusMemoryStore on
creation.

/cc @shawncal @lemillermicrosoft 
/cc @stephentoub @luisquintanilla @SamMonoRT

Co-authored-by: Mark Wallace <127216156+markwallace-microsoft@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kernel Issues or pull requests impacting the core kernel memory .NET Issue or Pull requests regarding .NET code
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

5 participants