Skip to content

Add VECTOR and BSON data types support#16

Merged
okramarenko merged 19 commits into
add10supportfrom
add_vector
May 26, 2026
Merged

Add VECTOR and BSON data types support#16
okramarenko merged 19 commits into
add10supportfrom
add_vector

Conversation

@okramarenko
Copy link
Copy Markdown
Collaborator

@okramarenko okramarenko commented Apr 28, 2026

This PR introduces SingleStoreDbType.Vector and SingleStoreDbType.Bson, enables SingleStore extended protocol metadata by default on supported servers (adjustable via EnableExtendedDataTypes conn string parameter), and maps native VECTOR/BSON columns to appropriate .NET types.

Note

Medium Risk
Medium risk: introduces new protocol parsing and session initialization behavior (EnableExtendedDataTypes) that affects how column metadata/values and bulk copy parameter serialization work, with potential compatibility impact on older servers and existing binary/blob workflows.

Overview
Adds native VECTOR and BSON support via SingleStore extended protocol metadata, including new SingleStoreDbType.Bson/Vector mappings, typed VECTOR reads as ReadOnlyMemory<T>, and updated schema (SingleStoreDbColumn) metadata for dimensions/element type.

Introduces EnableExtendedDataTypes connection-string option (default true) that sets enable_extended_types_metadata on supported servers (8.5.28+) and falls back to legacy metadata unless explicitly required; ResetConnectionAsync now re-applies this session initialization.

Extends parameter inference/serialization and bulk copy handling for VECTOR/BSON (including correct casting expressions for SingleStoreBulkCopy), adds shared binary conversion utilities, and updates/expands docs and tests to cover round-trips, legacy-disable behavior, and bulk copy scenarios.

Reviewed by Cursor Bugbot for commit eb57e40. Bugbot is set up for automated code reviews on this repo. Configure here.

Comment thread src/SingleStoreConnector/Protocol/Payloads/ColumnDefinitionPayload.cs Outdated
Comment thread src/SingleStoreConnector/Core/TypeMapper.cs Outdated
Comment thread src/SingleStoreConnector/SingleStoreParameter.cs
Comment thread src/SingleStoreConnector/Core/SingleStoreBinaryValueConverter.cs
Comment thread src/SingleStoreConnector/Core/ServerSession.cs Outdated
Comment thread src/SingleStoreConnector/SingleStoreDbColumn.cs
Comment thread src/SingleStoreConnector/Protocol/Payloads/ColumnDefinitionPayload.cs Outdated
Comment thread src/SingleStoreConnector/SingleStoreParameter.cs Outdated
Comment thread src/SingleStoreConnector/SingleStoreConnection.cs
Comment thread src/SingleStoreConnector/SingleStoreConnection.cs
Comment thread src/SingleStoreConnector/Core/ServerSession.cs Outdated
Comment thread tests/SideBySide/BulkLoaderSync.cs
Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default mode and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit eb57e40. Configure here.

Comment thread src/SingleStoreConnector/SingleStoreDbColumn.cs
Comment thread src/SingleStoreConnector/Core/ConnectionSettings.cs Outdated
Comment thread src/SingleStoreConnector/SingleStoreBulkCopy.cs Outdated
Comment thread src/SingleStoreConnector/SingleStoreDbColumn.cs Outdated
Comment thread src/SingleStoreConnector/SingleStoreParameter.cs Outdated
Comment thread src/SingleStoreConnector/Protocol/Payloads/ColumnDefinitionPayload.cs Outdated
Comment thread src/SingleStoreConnector/SingleStoreConnection.cs Outdated
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds native SingleStore VECTOR and BSON support to the connector by enabling extended type metadata (via a new EnableExtendedDataTypes connection-string option) and updating type mapping, column metadata, value readers, parameter handling, and bulk copy casting to understand these types.

Changes:

  • Introduces SingleStoreDbType.Vector / SingleStoreDbType.Bson, parses extended protocol metadata for these types, and returns VECTOR values as typed ReadOnlyMemory<T> while treating BSON as byte[].
  • Adds session initialization (SET SESSION enable_extended_types_metadata = TRUE) controlled by EnableExtendedDataTypes (default true) with version-gated fallback/exception behavior.
  • Extends parameter inference/serialization and SingleStoreBulkCopy mapping to correctly serialize/cast VECTOR/BSON; adds docs and expands tests for round-trips and bulk copy.

Reviewed changes

Copilot reviewed 28 out of 28 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
tests/SingleStoreConnector.Tests/SingleStoreConnectionStringBuilderTests.cs Updates tests for the new EnableExtendedDataTypes option default/parse/round-trip.
tests/SideBySide/StoredProcedureTests.cs Adds a stored procedure test validating BSON result typing and value retrieval.
tests/SideBySide/ServerFeatures.cs Adds ServerFeatures.ExtendedDataTypes feature flag.
tests/SideBySide/QueryTests.cs Adds round-trip tests for VECTOR/BSON parameters and supported memory formats.
tests/SideBySide/ParameterTests.cs Adds parameter inference tests for vector-like numeric arrays and extended db types.
tests/SideBySide/ExtendedDataTypeTestUtilities.cs New helper utilities for asserting vector values across multiple data representations.
tests/SideBySide/DataTypesFixture.cs Creates/populates VECTOR and BSON datatype tables when supported.
tests/SideBySide/DataTypes.cs Adds schema and query tests for VECTOR/BSON, plus a legacy-metadata test when disabled.
tests/SideBySide/ConnectionTests.cs Adds reset-connection test ensuring extended types session state is preserved.
tests/SideBySide/BulkLoaderSync.cs Adds bulk copy tests for VECTOR/BSON using DataTable and DataReader (sync).
tests/SideBySide/BulkLoaderAsync.cs Adds bulk copy tests for VECTOR/BSON using DataTable (async).
src/SingleStoreConnector/SingleStoreParameter.cs Adds inference/serialization paths for VECTOR/BSON; refactors binary literal writing and vector literal creation.
src/SingleStoreConnector/SingleStoreDbType.cs Adds Bson and Vector provider/logical types.
src/SingleStoreConnector/SingleStoreDbColumn.cs Surfaces vector dimension/element type schema metadata and adjusts DataType/DataTypeName for extended types.
src/SingleStoreConnector/SingleStoreConnectionStringBuilder.cs Adds EnableExtendedDataTypes connection-string option.
src/SingleStoreConnector/SingleStoreConnection.cs Adds session initialization to enable extended metadata on supported servers and re-applies it on reset.
src/SingleStoreConnector/SingleStoreBulkCopy.cs Adds VECTOR/BSON column mappings using UNHEX(...):>VECTOR(...) and UNHEX(...):>BSON, plus broader vector input support.
src/SingleStoreConnector/Protocol/Payloads/ColumnDefinitionPayload.cs Parses extended type metadata from column definitions (type code, vector dimensions/element type).
src/SingleStoreConnector/Protocol/ColumnType.cs Clarifies that BSON/VECTOR are exposed via extended metadata, not new base column types.
src/SingleStoreConnector/Core/TypeMapper.cs Adds type metadata for BSON/VECTOR and maps extended type codes to provider types.
src/SingleStoreConnector/Core/SingleStoreBinaryValueConverter.cs New shared converter for BSON raw bytes and vector element-to-byte conversion plus inference helpers.
src/SingleStoreConnector/Core/ServerVersions.cs Adds SupportsExtendedDataTypes server version gate (8.5.28+).
src/SingleStoreConnector/Core/Row.cs Updates binary column checks to allow BSON but disallow VECTOR-to-bytes casts.
src/SingleStoreConnector/Core/ConnectionSettings.cs Plumbs EnableExtendedDataTypes and “explicitly set” detection into settings.
src/SingleStoreConnector/ColumnReaders/VectorColumnReader.cs New typed readers for vector element types with length/endianness validation.
src/SingleStoreConnector/ColumnReaders/ColumnReader.cs Routes extended types to appropriate readers (BytesColumnReader for BSON; vector readers for VECTOR).
docs/content/tutorials/best-practices.md Documents how to read/write VECTOR/BSON and the EnableExtendedDataTypes option.
docs/content/connection-options.md Adds EnableExtendedDataTypes connection option documentation and behavior details.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread tests/SideBySide/ParameterTests.cs Outdated
Comment thread tests/SideBySide/ParameterTests.cs
Comment thread src/SingleStoreConnector/SingleStoreConnectionStringBuilder.cs
@okramarenko okramarenko merged commit ca2464e into add10support May 26, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants