Expose LSN Header Information

[jhamon]

Expose LSN Header Information in API Responses

Overview

This PR implements exposure of LSN (Log Sequence Number) header information from Pinecone API responses through a new _response_info attribute on response objects. This enables faster test suite execution by using LSN-based freshness checks instead of polling describe_index_stats().

Motivation

Integration tests currently rely on polling describe_index_stats() to verify data freshness, which is slow and inefficient. The Pinecone API includes LSN headers in responses that can be used to determine data freshness more efficiently:

• x-pinecone-request-lsn: Committed LSN from write operations (upsert, delete)
• x-pinecone-max-indexed-lsn: Reconciled LSN from read operations (query)

By extracting and exposing these headers, tests can use LSN-based polling to reduce test execution time significantly. Testing so far shows this will cut the time needed to run db data plane integration times down by half or more.

Changes

Core Implementation

Response Info Module

• Created pinecone/utils/response_info.py with:
  • ResponseInfo TypedDict for structured response metadata
  • extract_response_info() function to extract and normalize raw headers
  • Fields: raw_headers (dictionary of all response headers normalized to lowercase)
  • Case-insensitive header matching
  • LSN extraction is handled by test utilities (lsn_utils) rather than in ResponseInfo

REST API Client Integration

• Updated api_client.py and asyncio_api_client.py to automatically attach _response_info to db data plane response objects
• Always attaches _response_info to ensure raw_headers are always available, even when LSN fields are not present

gRPC Integration

• Updated grpc_runner.py to capture initial metadata from gRPC calls
• Modified all parser functions in grpc/utils.py to accept optional initial_metadata parameter
• Updated index_grpc.py to pass initial metadata to parser functions
• Updated future.py to extract initial metadata from gRPC futures

Response Dataclasses

• Created QueryResponse and UpsertResponse dataclasses in pinecone/db_data/dataclasses/
• Added _response_info field to FetchResponse, FetchByMetadataResponse, QueryResponse, and UpsertResponse
• All response dataclasses inherit from DictLike for dictionary-style access
• _response_info is a required field (always present) with default {"raw_headers": {}}

Index Classes

• Updated index.py and index_asyncio.py to:
  • Convert OpenAPI responses to dataclasses with _response_info attached
  • Handle async_req=True with ApplyResult wrapper for proper dataclass conversion
  • Extract _response_info from upsert_records() responses

Test Infrastructure

LSN Utilities

• Created tests/integration/helpers/lsn_utils.py with helper functions for extracting LSN values
• Created compatibility shim pinecone/utils/lsn_utils.py (deprecated)

Polling Helpers

• Updated poll_until_lsn_reconciled() to use query operations for LSN-based freshness checks
• Added poll_until_lsn_reconciled_async() for async tests
• Falls back to old polling methods when LSN not available

Integration Test Updates

• Updated multiple integration tests to use LSN-based polling:
  • test_query.py, test_upsert_dense.py, test_search_and_upsert_records.py
  • test_fetch.py, test_fetch_by_metadata.py, test_upsert_hybrid.py
  • test_query_namespaces.py, seed.py
  • Async versions: test_query.py (async)
• Added assertions to verify _response_info is present when expected

Documentation

• Created docs/maintainers/lsn-headers-discovery.md documenting discovered headers
• Created scripts/inspect_lsn_headers.py for header discovery

Usage Examples

Accessing Response Info

The _response_info attribute is always available on all Index response objects:

from pinecone import Pinecone

pc = Pinecone(api_key="your-api-key")
index = pc.Index("my-index")

# Upsert operation - get committed LSN
upsert_response = index.upsert(
    vectors=[("id1", [0.1, 0.2, 0.3]), ("id2", [0.4, 0.5, 0.6])]
)

# Access raw headers (always present, contains all response headers)
raw_headers = upsert_response._response_info.get("raw_headers")
print(f"Raw headers: {raw_headers}")
# Example output: Raw headers: {
#   'x-pinecone-request-lsn': '12345',
#   'x-pinecone-api-version': '2025-10',
#   'content-type': 'application/json',
#   'server': 'envoy',
#   ...
# }

# Extract LSN from raw headers using test utilities (for testing/polling)
from tests.integration.helpers.lsn_utils import extract_lsn_committed
lsn_committed = extract_lsn_committed(raw_headers)
print(f"Committed LSN: {lsn_committed}")
# Example output: Committed LSN: 12345

# Query operation
query_response = index.query(
    vector=[0.1, 0.2, 0.3],
    top_k=10
)

# Access raw headers
raw_headers = query_response._response_info.get("raw_headers")
print(f"Raw headers: {raw_headers}")
# Example output: Raw headers: {
#   'x-pinecone-max-indexed-lsn': '12345',
#   'x-pinecone-api-version': '2025-10',
#   'content-type': 'application/json',
#   ...
# }

# Extract LSN from raw headers using test utilities
from tests.integration.helpers.lsn_utils import extract_lsn_reconciled
lsn_reconciled = extract_lsn_reconciled(raw_headers)
print(f"Reconciled LSN: {lsn_reconciled}")
# Example output: Reconciled LSN: 12345

# Fetch operation - response info always available
fetch_response = index.fetch(ids=["id1", "id2"])
print(f"Response info: {fetch_response._response_info}")
# Example output:
# Response info: {
#   'raw_headers': {
#     'x-pinecone-max-indexed-lsn': '12345',
#     'x-pinecone-api-version': '2025-10',
#     'content-type': 'application/json',
#     ...
#   }
# }

Dictionary-Style Access

All response dataclasses inherit from DictLike, enabling dictionary-style access:

query_response = index.query(vector=[...], top_k=10)

# Attribute access (existing)
matches = query_response.matches

# Dictionary-style access (new)
matches = query_response["matches"]

# Response info access
response_info = query_response._response_info
# Example: {'raw_headers': {'x-pinecone-max-indexed-lsn': '12345', 'x-pinecone-api-version': '2025-10', 'content-type': 'application/json', ...}}

Technical Details

Response Info Flow

1. REST API:

   • HTTP headers → api_client.py extracts → attaches _response_info to OpenAPI model → Index classes convert to dataclasses

2. gRPC:

   • Initial metadata → grpc_runner.py captures → parser functions extract → attach _response_info to response objects

Backward Compatibility

• All existing method signatures remain unchanged
• _response_info is always present on response objects (required field)
• raw_headers in _response_info always contains response headers (may be empty dict if no headers)
• Test utilities (poll_until_lsn_reconciled, poll_until_lsn_reconciled_async) accept _response_info directly and extract LSN internally
• Response objects maintain all existing attributes and behavior

Type Safety

• Added proper type hints for _response_info fields
• Updated return type annotations to reflect dataclass usage
• Added type: ignore comments where necessary (e.g., ApplyResult wrapping)

Dataclass Enhancements

• All response dataclasses now inherit from DictLike for dictionary-style access
• QueryResponse and UpsertResponse are new dataclasses replacing OpenAPI models
• _response_info field: ResponseInfo = field(default_factory=lambda: cast(ResponseInfo, {"raw_headers": {}}), repr=True, compare=False)
  • Always present (required field)
  • repr=True for all response dataclasses to aid debugging
  • raw_headers always contains response headers (may be empty dict)
  • ResponseInfo only contains raw_headers

Testing

Unit Tests

• ✅ All gRPC upsert tests pass (32/32)
• ✅ All unit tests pass (340+ passed)
• ✅ Created unit tests for extract_response_info() function
• ✅ Created unit tests for LSN utility functions

Integration Tests

• ✅ Updated integration tests to use LSN-based polling
• ✅ 38 integration tests pass
• ✅ LSN-based polling working correctly (faster test execution)
• ✅ _response_info assertions added to verify LSN data is present

Breaking Changes

None - This is a backward-compatible enhancement.

Response Type Changes

• QueryResponse and UpsertResponse are now dataclasses instead of OpenAPI models
• Impact: Minimal - dataclasses are compatible for attribute access and dictionary-style access (via DictLike)
• Mitigation: Public API exports remain the same (from pinecone import QueryResponse, UpsertResponse)
• Note: If users were doing isinstance() checks against OpenAPI models, they should still work when importing from pinecone

New Attribute

• _response_info is added to all Index response objects (QueryResponse, UpsertResponse, FetchResponse, FetchByMetadataResponse)
• Impact: Minimal - it's a required attribute with underscore prefix (indicates internal use)
• Mitigation: Underscore prefix indicates it's not part of the public API contract
• Note: _response_info is always present and contains raw_headers.

Compatibility Notes

• All response dataclasses inherit from DictLike, enabling dictionary-style access (response['matches'])
• Attribute access remains unchanged (response.matches, response.namespace, etc.)
• OpenAPI-specific methods like to_dict() were not part of the public API

Related Issues

• Enables faster test suite execution through LSN-based polling
• Provides foundation for future LSN-based features

[rohanshah18]

both versions are 3.10?