Python: Feature python vector stores preb

[eavanvalkenburg]

Motivation and Context

Overhaul of the VectorStores to get it matching with dotnet and the latest development there.

Closes:

• Port dotnet feature: .Net: Delete CreateCollection, rename CreateIfNotExists and make EnsureExists thread safe. #11938
• Port dotnet feature: .Net: Rename IsFilterable to IsIndexed and IsFullTextSearchable to IsFullTextIndexed #11598
• Port dotnet feature: .Net: Add delete and exists methods for IVectorStore #11597
• Port dotnet feature: .Net: Refactor MEVD Vector Attribute and make Dimensions mandatory #11517
• Python: Implement Hybrid Search for all memory connectors #10561
• Python: Support BetweenFilterClause or RangeFilterClause for Redis #10391
• .Net: Python: Support listing all keys on vector store #9911
• .Net: Python: API for listing/filtering records without similarity search #9892
• Python: Implement new filters #10867

Description

Many changes, some highlights:

• Moved from different vector fields types, to a single with a enum first VectorStoreField()
• Renamed several items
• Made embeddinggenerrator a part of the vector field spec and do automatic vectorization from there or from the same param on a collection
• Adds hybrid search using the same setup
• Removed the intermediate TextSeachVectorSearch object need, by allowing create_search_function directly on a collection

Will write out full changes in a migration guide in the docs.

Contribution Checklist

• The code builds clean without any errors or warnings
• The PR follows the SK Contribution Guidelines and the pre-submission formatting script raises no violations
• All unit tests pass, and I have added new tests where possible
• I didn't break anyone 😄

[markwallace-microsoft]

Python Test Coverage Report •

File	Stmts	Miss	Cover	Missing
__init__.py	4	0	100%
const.py	7	0	100%
kernel_types.py	10	0	100%
connectors
_search_shared.py	28	0	100%
azure_ai_search.py	303	27	91%	159–160, 208, 210–213, 215, 244, 246, 248, 339, 361–362, 423, 425, 439–440, 451, 522, 544, 554, 566, 586–587, 781–782
azure_cosmos_db.py	401	166	58%	184, 201, 207–209, 319, 321, 396, 400, 416–419, 421–424, 426–427, 440–443, 446–448, 454–455, 457, 464–466, 470–472, 495–497, 503–505, 511–512, 514, 519, 536, 624, 627–628, 636, 646–650, 705, 708–709, 745, 747, 760–761, 765–767, 778–781, 784–785, 787–788, 793–797, 800–803, 805–810, 812–813, 820–825, 844–846, 848–851, 859, 866–871, 873–887, 889–900, 903, 905–906, 909–910, 913, 915–920, 926, 930, 954, 982–983, 987–992, 1081–1086
brave.py	142	23	83%	28, 171, 177, 187, 206–209, 245, 250, 252, 270–281
chroma.py	255	89	65%	104, 107–111, 126–127, 135–136, 165, 167, 175, 187–190, 197, 216, 220, 241, 249, 253, 266, 270, 275–276, 285–288, 304–305, 313, 332, 343, 345, 349, 367–369, 371–386, 388–398, 400–406, 408, 410, 412–413, 416–417, 419–420, 423–425, 447–448, 450
faiss.py	119	12	89%	48, 58–59, 63–64, 121, 129, 133, 170, 217, 220, 240
google_search.py	145	21	85%	29, 217, 227, 249–254, 269–270, 303–312
in_memory.py	169	23	86%	69–70, 74, 78–81, 116, 128, 130, 171, 175, 179, 194, 211, 226, 266–267, 269, 276–277, 282, 317
memory.py	10	10	0%	3, 5, 48–53, 56–57
mongodb.py	258	128	50%	100–101, 105, 117–120, 125–126, 133–134, 139–140, 149, 200, 254, 282, 284, 294–296, 302–304, 311, 315, 327–328, 336, 347–351, 360–363, 366–368, 374–375, 377, 384–386, 390–392, 404–407, 410–412, 418–419, 421, 428–430, 434–436, 444–446, 448–463, 465–475, 477–483, 485–490, 492, 494–495, 498–499, 501–502, 505–507, 512, 516, 577–578, 580, 622–623, 627–628
pinecone.py	333	79	76%	130, 144–147, 152, 157, 183, 216, 220–221, 228, 250, 254, 282, 310–313, 329, 375, 377, 382, 389, 400, 402, 404, 406, 409, 418, 420, 424, 439, 441, 443, 448, 455, 467, 474, 476, 501–507, 513, 515, 519–528, 531–536, 539–541, 543, 549–551, 556, 561, 567–568, 571, 595, 664
postgres.py	371	101	72%	139, 157, 184, 297–298, 383, 389–391, 396–397, 399–400, 407, 429, 485, 487, 489, 506, 518, 568, 577, 627, 647, 667, 672, 678, 690, 720, 724, 726, 748, 775, 780, 797, 854–856, 858–884, 886–892, 894–899, 901, 903–904, 907–908, 910–911, 914–923, 925, 930, 934, 961–963
qdrant.py	251	57	77%	249, 251, 287, 306–310, 316–317, 320–321, 327–330, 332, 334, 351, 375–381, 388, 391, 394–403, 406–411, 414–418, 423, 429–430, 433, 436–437, 495, 509
redis.py	414	170	58%	102, 107, 120, 123, 127–129, 133–134, 137, 147–151, 164–165, 299, 304–305, 316–321, 324–325, 333–335, 337, 344–348, 350, 352–354, 363, 369, 371, 375–385, 387–389, 391–401, 403–423, 425, 427–446, 449–461, 463–468, 470, 472–473, 476–477, 479–480, 483–485, 501–506, 510, 566, 568, 631–638, 694, 696, 747–754, 837, 844–845
search.py	10	10	0%	3, 5, 19–24, 27–28
sql_server.py	501	88	82%	152, 349–351, 371, 390, 401, 403, 426, 472–475, 491, 503–506, 528, 530, 574–580, 586–587, 589–590, 594–608, 611–616, 619–623, 628, 634–635, 638, 643, 645–648, 787, 806–808, 814, 820, 825, 830–833, 839, 873, 915, 922–924, 1090–1091, 1093, 1095, 1116
weaviate.py	335	142	57%	110, 315–318, 327–328, 330–332, 337, 341–342, 353–355, 364–372, 375–376, 380–381, 384–386, 389–390, 395–396, 399–400, 402, 411–413, 415–424, 426, 428–443, 447–453, 455, 457, 459–460, 463–464, 466–467, 470–472, 482–483, 486–487, 493–494, 497–498, 502, 505–509, 524, 539, 548–549, 563, 567–570, 573–574, 580–581, 585, 588–589, 592, 595, 615, 634, 650, 659–660, 665–666, 671, 765, 776–781, 786–791, 796–797
connectors/ai/embeddings
embedding_generator_base.py	10	1	90%	9
connectors/ai/hugging_face
hf_prompt_execution_settings.py	24	1	95%	33
connectors/ai/hugging_face/services
hf_text_embedding.py	32	5	84%	84–88
connectors/ai/open_ai/prompt_execution_settings
azure_chat_prompt_execution_settings.py	73	0	100%
core_plugins
text_memory_plugin.py	33	8	75%	14, 43, 72, 78–80, 82, 103
data
__init__.py	0	0	100%
_shared.py	62	9	85%	136, 142, 168, 172–173, 175, 178–179, 181
text_search.py	69	4	94%	229, 240, 277, 279
vector.py	656	114	82%	83, 413, 579, 698, 896–899, 905, 907–908, 1026, 1064–1065, 1068, 1075, 1095–1102, 1104, 1106–1107, 1109, 1111, 1114–1119, 1121–1126, 1148, 1171, 1308, 1459, 1474, 1480–1481, 1483, 1510, 1584–1585, 1597–1598, 1627, 1712, 1717–1720, 1834, 1861–1862, 1901, 1921, 1929–1930, 1942, 1949, 1994, 1997, 2051–2053, 2056, 2065, 2088, 2090–2095, 2097–2102, 2106–2112, 2116–2121, 2124–2126, 2128, 2156, 2170, 2178, 2217, 2241, 2255, 2301, 2341
exceptions
vector_store_exceptions.py	24	0	100%
functions
kernel_plugin.py	174	3	98%	395, 398, 411
TOTAL	26389	4515	82%

Python Unit Test Overview

Tests	Skipped	Failures	Errors	Time
3614	22 💤	0 ❌	0 🔥	1m 51s ⏱️

[TaoChenOSU]

This has been removed.

[TaoChenOSU]

This has been removed.

[TaoChenOSU]

Suggested change

	With the data model and records defined in step_0_data_model.py, this script will create an Azure AI Search collection,
	With the data model and records defined in data_model.py, this script will create an Azure AI Search collection,

[TaoChenOSU]

Suggested change

	It uses the data model defined in step_0_data_model.py, and with that creates a collection
	It uses the data model defined in data_model.py, and with that creates a collection

[TaoChenOSU]

nit: could we define the instructions and plugins somewhere else and use them here? What we are doing now may scare away people when they look at the number of lines to create the agent.

[TaoChenOSU]

Suggested change

	python step_1_interact_with_the_collection.py
	python 1_interact_with_the_collection.py

[TaoChenOSU]

What kind of connectors are these?

[TaoChenOSU]

Could we still group the files in sub folders but allow lazy loading of the modules at the root?