Python: Feature python vector stores preb

[eavanvalkenburg]

Motivation and Context

Overhaul of the VectorStores to get it matching with dotnet and the latest development there.

Closes:

• Port dotnet feature: .Net: Delete CreateCollection, rename CreateIfNotExists and make EnsureExists thread safe. #11938
• Port dotnet feature: .Net: Rename IsFilterable to IsIndexed and IsFullTextSearchable to IsFullTextIndexed #11598
• Port dotnet feature: .Net: Add delete and exists methods for IVectorStore #11597
• Port dotnet feature: .Net: Refactor MEVD Vector Attribute and make Dimensions mandatory #11517
• Python: Implement Hybrid Search for all memory connectors #10561
• Python: Support BetweenFilterClause or RangeFilterClause for Redis #10391
• .Net: Python: Support listing all keys on vector store #9911
• .Net: Python: API for listing/filtering records without similarity search #9892
• Python: Implement new filters #10867

Description

Many changes, some highlights:

• Moved from different vector fields types, to a single with a enum first VectorStoreField()
• Renamed several items
• Made embeddinggenerrator a part of the vector field spec and do automatic vectorization from there or from the same param on a collection
• Adds hybrid search using the same setup
• Removed the intermediate TextSeachVectorSearch object need, by allowing create_search_function directly on a collection

Will write out full changes in a migration guide in the docs.

Contribution Checklist

• The code builds clean without any errors or warnings
• The PR follows the SK Contribution Guidelines and the pre-submission formatting script raises no violations
• All unit tests pass, and I have added new tests where possible
• I didn't break anyone 😄

[markwallace-microsoft]

Python Test Coverage Report •

File	Stmts	Miss	Cover	Missing
__init__.py	4	0	100%
const.py	7	0	100%
kernel_types.py	10	0	100%
connectors
_search_shared.py	28	0	100%
azure_ai_search.py	303	27	91%	159–160, 208, 210–213, 215, 244, 246, 248, 339, 361–362, 423, 425, 439–440, 451, 522, 544, 554, 566, 586–587, 781–782
azure_cosmos_db.py	401	166	58%	184, 201, 207–209, 319, 321, 396, 400, 416–419, 421–424, 426–427, 440–443, 446–448, 454–455, 457, 464–466, 470–472, 495–497, 503–505, 511–512, 514, 519, 536, 624, 627–628, 636, 646–650, 705, 708–709, 745, 747, 760–761, 765–767, 778–781, 784–785, 787–788, 793–797, 800–803, 805–810, 812–813, 820–825, 844–846, 848–851, 859, 866–871, 873–887, 889–900, 903, 905–906, 909–910, 913, 915–920, 926, 930, 954, 982–983, 987–992, 1081–1086
brave.py	142	23	83%	28, 171, 177, 187, 206–209, 245, 250, 252, 270–281
chroma.py	255	89	65%	104, 107–111, 126–127, 135–136, 165, 167, 175, 187–190, 197, 216, 220, 241, 249, 253, 266, 270, 275–276, 285–288, 304–305, 313, 332, 343, 345, 349, 367–369, 371–386, 388–398, 400–406, 408, 410, 412–413, 416–417, 419–420, 423–425, 447–448, 450
faiss.py	119	12	89%	48, 58–59, 63–64, 121, 129, 133, 170, 217, 220, 240
google_search.py	145	21	85%	29, 217, 227, 249–254, 269–270, 303–312
in_memory.py	169	23	86%	69–70, 74, 78–81, 116, 128, 130, 171, 175, 179, 194, 211, 226, 266–267, 269, 276–277, 282, 317
memory.py	10	10	0%	3, 5, 48–53, 56–57
mongodb.py	258	128	50%	100–101, 105, 117–120, 125–126, 133–134, 139–140, 149, 200, 254, 282, 284, 294–296, 302–304, 311, 315, 327–328, 336, 347–351, 360–363, 366–368, 374–375, 377, 384–386, 390–392, 404–407, 410–412, 418–419, 421, 428–430, 434–436, 444–446, 448–463, 465–475, 477–483, 485–490, 492, 494–495, 498–499, 501–502, 505–507, 512, 516, 577–578, 580, 622–623, 627–628
pinecone.py	334	79	76%	130, 144–147, 152, 157, 183, 216, 220–221, 228, 250, 254, 282, 310–313, 329, 376, 378, 383, 390, 401, 403, 405, 407, 410, 419, 421, 425, 440, 442, 444, 449, 456, 468, 475, 477, 502–508, 514, 516, 520–529, 532–537, 540–542, 544, 550–552, 557, 562, 568–569, 572, 596, 665
postgres.py	371	101	72%	139, 157, 184, 297–298, 383, 389–391, 396–397, 399–400, 407, 429, 485, 487, 489, 506, 518, 568, 577, 627, 647, 667, 672, 678, 690, 720, 724, 726, 748, 775, 780, 797, 854–856, 858–884, 886–892, 894–899, 901, 903–904, 907–908, 910–911, 914–923, 925, 930, 934, 961–963
qdrant.py	251	57	77%	249, 251, 287, 306–310, 316–317, 320–321, 327–330, 332, 334, 351, 375–381, 388, 391, 394–403, 406–411, 414–418, 423, 429–430, 433, 436–437, 495, 509
redis.py	414	170	58%	102, 107, 120, 123, 127–129, 133–134, 137, 147–151, 164–165, 299, 304–305, 316–321, 324–325, 333–335, 337, 344–348, 350, 352–354, 363, 369, 371, 375–385, 387–389, 391–401, 403–423, 425, 427–446, 449–461, 463–468, 470, 472–473, 476–477, 479–480, 483–485, 501–506, 510, 566, 568, 631–638, 694, 696, 747–754, 837, 844–845
search.py	10	10	0%	3, 5, 19–24, 27–28
sql_server.py	501	88	82%	152, 349–351, 371, 390, 401, 403, 426, 472–475, 491, 503–506, 528, 530, 574–580, 586–587, 589–590, 594–608, 611–616, 619–623, 628, 634–635, 638, 643, 645–648, 787, 806–808, 814, 820, 825, 830–833, 839, 873, 915, 922–924, 1090–1091, 1093, 1095, 1116
weaviate.py	335	142	57%	110, 315–318, 327–328, 330–332, 337, 341–342, 353–355, 364–372, 375–376, 380–381, 384–386, 389–390, 395–396, 399–400, 402, 411–413, 415–424, 426, 428–443, 447–453, 455, 457, 459–460, 463–464, 466–467, 470–472, 482–483, 486–487, 493–494, 497–498, 502, 505–509, 524, 539, 548–549, 563, 567–570, 573–574, 580–581, 585, 588–589, 592, 595, 615, 634, 650, 659–660, 665–666, 671, 765, 776–781, 786–791, 796–797
connectors/ai/embeddings
embedding_generator_base.py	10	1	90%	9
connectors/ai/hugging_face
hf_prompt_execution_settings.py	24	1	95%	33
connectors/ai/hugging_face/services
hf_text_embedding.py	32	5	84%	84–88
connectors/ai/open_ai/prompt_execution_settings
azure_chat_prompt_execution_settings.py	73	0	100%
core_plugins
text_memory_plugin.py	33	8	75%	14, 43, 72, 78–80, 82, 103
data
__init__.py	0	0	100%
_shared.py	62	9	85%	136, 142, 168, 172–173, 175, 178–179, 181
text_search.py	69	4	94%	229, 240, 277, 279
vector.py	656	114	82%	83, 413, 579, 698, 896–899, 905, 907–908, 1026, 1064–1065, 1068, 1075, 1095–1102, 1104, 1106–1107, 1109, 1111, 1114–1119, 1121–1126, 1148, 1171, 1308, 1459, 1474, 1480–1481, 1483, 1510, 1584–1585, 1597–1598, 1627, 1712, 1717–1720, 1834, 1861–1862, 1901, 1921, 1929–1930, 1942, 1949, 1994, 1997, 2051–2053, 2056, 2065, 2088, 2090–2095, 2097–2102, 2106–2112, 2116–2121, 2124–2126, 2128, 2156, 2170, 2178, 2217, 2241, 2255, 2301, 2341
exceptions
vector_store_exceptions.py	24	0	100%
functions
kernel_plugin.py	174	3	98%	395, 398, 411
TOTAL	26419	4504	82%

Python Unit Test Overview

Tests	Skipped	Failures	Errors	Time
3638	22 💤	0 ❌	0 🔥	1m 52s ⏱️

[TaoChenOSU]

Could we still group the files in sub folders but allow lazy loading of the modules at the root?