Python: Add core vector store abstractions

[Eduard van Valkenburg (eavanvalkenburg)]

Motivation & Context

Python needs native vector store contracts before in-memory storage and external database connectors can share consistent model, CRUD, search, serialization, and tool behavior. This implements Phase 3 of the vector stores and embeddings plan as an experimental core API.

Description & Review Guide

• What are the major changes? Adds experimental vector field and collection definitions; model decoration and registration with msgspec-backed codecs; batch-oriented collection/store/search base classes and capability protocols; embedding generation, filtering, score thresholds, and agent search-tool creation; plus model-format samples and focused tests.
• What is the impact of these changes? Provider packages can implement future vector connectors against one core contract without taking dependencies on Pydantic, NumPy, or pandas. Applications can keep owned or externally defined model types, optimized vector formats, and dictionary-based records. Existing APIs are unchanged.
• What do you want reviewers to focus on? Please focus on the public API shape and future connector implementability, especially model registration, batch CRUD semantics, search result handling, and configurable multimodal search tools.

Related Issue

Fixes #4165

Part of #1188

Contribution Checklist

• The code builds clean without any errors or warnings
• All unit tests pass, and I have added new tests where possible
• The PR follows the Contribution Guidelines
• This PR is linked to an issue and there is no other open PR for this issue (see Related Issue above).
• This is not a breaking change. If it is a breaking change, add the breaking change label (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.

[Copilot]

🟡 Changes recommended

Serialization and lazy-search edge cases can corrupt records, accept invalid embeddings, or leak provider exceptions.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds experimental Python vector-store contracts for shared connector implementations, model serialization, CRUD, search, embeddings, and agent tools.

Changes:

• Introduces vector model definitions, registration, codecs, and base abstractions.
• Adds embedding generation, filtering, search-result mapping, and capability protocols.
• Adds comprehensive tests, documentation, exports, and usage samples.

File summaries

File	Description
python/samples/AGENTS.md	Registers vector-store samples.
python/samples/02-agents/vector_stores/vector_store_models.py	Demonstrates supported model styles.
python/samples/02-agents/vector_stores/README.md	Documents samples and performance considerations.
python/samples/02-agents/vector_stores/optimized_data_formats.py	Demonstrates NumPy and pandas adaptation.
python/packages/core/tests/core/test_vectors.py	Tests the new abstractions.
python/packages/core/AGENTS.md	Documents vector-store architecture.
python/packages/core/agent_framework/_vectors.py	Implements the core vector-store API.
python/packages/core/agent_framework/_feature_stage.py	Adds the experimental feature identifier.
python/packages/core/agent_framework/__init__.pyi	Exposes typed public imports.
python/packages/core/agent_framework/__init__.py	Adds lazy public exports.
docs/features/vector-stores-and-embeddings/README.md	Updates the feature plan and design.

Review details

Suppressed comments (1)

python/packages/core/agent_framework/_vectors.py:1451

• Query embedding generation also forwards a vector without checking it matches the selected field's declared dimensions. Validate the normalized vector before invoking the connector so an embedding provider mismatch is surfaced consistently as an invalid response.

        generated_vector = embeddings[0].vector
        return _normalize_vector(generated_vector)

• Files reviewed: 11/11 changed files
• Comments generated: 8
• Review effort level: Balanced

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

MAF Automated Review — Iteration 1

Result: Findings reported
Scope: full PR (1 commit(s)): 2b0d7958a890
Model: gpt-5.6-sol-fast

Overview

The PR establishes a broad experimental contract with synchronized public exports, strict model/schema checks, typed integration errors, and focused CRUD/search tests. The strongest guardrails cover field definitions, embedding result cardinality, paging minima, connector capability checks, and malformed search results. Residual risk remains around silent record loss in connector serialization, projection of required vector fields, missing dimension enforcement, inconsistent filtered counts, and unconstrained model-supplied search pagination.

Reviewed the supplied pull-request change set across correctness, security/reliability, architecture, and failure behavior.
5 verified findings remained after source verification (1 high, 4 medium) across 1 file. Details are attached to the affected lines below.

Affected areas: python/packages/core/agent_framework/_vectors.py

[Eduard van Valkenburg (eavanvalkenburg)]

/review

[github-actions]

MAF Automated Review — Iteration 2

Result: No findings
Scope: 3 net-new commit(s): 94d8a2c46d79, 8ebe8bb2d11f, ce82ee575f41
Model: gpt-5.6-sol-fast

Overview

This incremental revision adds feature telemetry and tightens vector serialization, projected decoding, lazy search error handling, result-count semantics, and search-tool paging controls. Its strongest guardrails are explicit cardinality and response-shape checks plus regression tests for dropped records, malformed connector output, lazy stream failures, required tool schemas, paging maxima, and result caps. No new Critical, High, or Medium defect remained after reconciling the changed lines against those guards and the supplied prior feedback.

Reviewed the supplied incremental change set across correctness, security/reliability, architecture, and failure behavior.
No publishable findings remained after source verification for this scope.

[Eduard van Valkenburg (eavanvalkenburg)]

/review

[github-actions]

MAF Automated Review — Iteration 3

Result: No findings
Scope: 1 net-new commit(s): 3b5bf06b89e7
Model: gpt-5.6-sol-fast

Overview

This incremental commit makes validated vector schema definitions immutable, moves record conversion behind private collection/search bases, includes vectors in CRUD reads by default, and exposes search tooling as an adapter over the minimal search protocol. Frozen tuple-backed definitions, aligned runtime/stub exports, query and paging validation, bounded result accumulation, and focused protocol and CRUD tests prevent a publishable concern in this range.

Reviewed the supplied incremental change set across correctness, security/reliability, architecture, and failure behavior.
No publishable findings remained after source verification for this scope.