Skip to content

Fully implement JSON binary writing, and add the ability to write directly from POCOs#179

Merged
alex-clickhouse merged 24 commits intomainfrom
json-binary-writing
Jan 22, 2026
Merged

Fully implement JSON binary writing, and add the ability to write directly from POCOs#179
alex-clickhouse merged 24 commits intomainfrom
json-binary-writing

Conversation

@alex-clickhouse
Copy link
Copy Markdown
Collaborator

@alex-clickhouse alex-clickhouse commented Jan 20, 2026

Adds POCO serialization for JSON columns with typed hints, allowing users to skip the JsonNode serialization when inserting from a POCO to a JSON column.

  • POCOs serialize using ClickHouse binary encoding when column has type hints (e.g., JSON(Id Int64))
  • Unhinted properties are auto-inferred and written as dynamic paths
  • [ClickHouseJsonPath("path")] and [ClickHouseJsonIgnore] attributes control property to path mapping

Implementation:

  • Single-pass buffered writing: we write to a buffer first, because we need to count the number of fields (and the number of fields comes before the actual data). Uses a memory pool for the buffer.
  • Reflection results and type inference results are cached for perf.
  • Nullable handling: Only writes nulls for hinted Nullable types (ClickHouse disallows Nullable in dynamic Variant paths)
  • Case-sensitive matching: Matches ClickHouse behavior allowing userName and UserName as separate paths
  • BinaryTypeEncoder: New encoder for writing type headers on dynamic paths

Breaking: String/JsonNode inputs now use string serialization (requires input_format_binary_read_json_as_string=1). First-class support for this will come in a different pr: #175 (There are some temporary hacks for the tests related to this).


Note

Introduces first-class JSON write support and Dynamic write capabilities.

  • POCO → JSON binary write: New RegisterJsonSerializationType<T>() on ClickHouseConnection/IClickHouseConnection; attributes ClickHouseJsonPath and ClickHouseJsonIgnore; validation errors via ClickHouseJsonSerializationException; cached reflection and nested type handling via JsonTypeRegistry.
  • JSON write behavior: string/JsonNode inputs are sent as JSON strings for server parsing (requires input_format_binary_read_json_as_string=1); hinted paths use column types; unhinted properties are inferred.
  • Dynamic writes: Implemented DynamicType.Write with automatic type inference and new BinaryTypeDescriptionWriter for type headers.
  • Type/mapping tweaks: Adjusted TypeSettings to carry registry; expose EnumType.Values; set default scale for Decimal128 mapping.
  • Tests/notes: Extensive new tests for JSON registration/serialization and Dynamic round-trips; temporary skip of JSON samples in utilities; release notes updated.

Written by Cursor Bugbot for commit b5efc2c. This will update automatically on new commits. Configure here.

@codecov
Copy link
Copy Markdown

codecov bot commented Jan 20, 2026

Codecov Report

❌ Patch coverage is 62.83988% with 123 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...kHouse.Driver/Types/BinaryTypeDescriptionWriter.cs 26.38% 118 Missing and 2 partials ⚠️
ClickHouse.Driver/Types/JsonType.cs 95.23% 1 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR implements comprehensive POCO serialization for ClickHouse JSON columns, enabling type-safe binary encoding when column type hints are present. The implementation adds reflection-based property mapping with caching, attribute-based path customization, and automatic type inference for unhinted fields.

Changes:

  • Adds POCO serialization with binary encoding for JSON columns using type hints from column definitions
  • Implements BinaryTypeEncoder for writing ClickHouse type headers on dynamic paths
  • Changes string/JsonNode inputs to use string serialization (requires input_format_binary_read_json_as_string=1)

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 10 comments.

Show a summary per file
File Description
RELEASENOTES.md Documents breaking change for string/JsonNode JSON writing and new POCO serialization feature
ClickHouse.Driver/Types/JsonType.cs Core implementation: adds reflection caching, POCO field writing, type inference, and binary encoding
ClickHouse.Driver/Types/EnumType.cs Exposes Values dictionary as internal for BinaryTypeEncoder access
ClickHouse.Driver/Types/BinaryTypeEncoder.cs New encoder that writes ClickHouse binary type headers for dynamic JSON paths
ClickHouse.Driver/PublicAPI/PublicAPI.Unshipped.txt Adds public API surface for new JSON attributes
ClickHouse.Driver/Json/ClickHouseJsonPathAttribute.cs New attribute for custom JSON path mapping
ClickHouse.Driver/Json/ClickHouseJsonIgnoreAttribute.cs New attribute to exclude properties from serialization
ClickHouse.Driver.Tests/Utilities/TestUtilities.cs Temporarily disables JSON data type samples pending full string format support
ClickHouse.Driver.Tests/Types/JsonTypeTests.cs Comprehensive test coverage for POCO serialization scenarios
ClickHouse.Driver.Tests/BulkCopy/BulkCopyTests.cs Updates existing tests with required server setting for string-based JSON parsing

Comment thread ClickHouse.Driver/Types/JsonType.cs
Comment thread ClickHouse.Driver/Types/JsonType.cs Outdated
Comment thread ClickHouse.Driver/Types/JsonType.cs Outdated
Comment thread ClickHouse.Driver/Types/JsonType.cs
Comment thread ClickHouse.Driver/Types/JsonType.cs Outdated
Comment thread ClickHouse.Driver/Types/JsonType.cs Outdated
Comment thread ClickHouse.Driver/Types/BinaryTypeDescriptionWriter.cs
Comment thread ClickHouse.Driver/Types/BinaryTypeDescriptionWriter.cs
Comment thread ClickHouse.Driver.Tests/Types/JsonTypeTests.cs
Comment thread ClickHouse.Driver.Tests/BulkCopy/BulkCopyTests.cs
@alex-clickhouse alex-clickhouse marked this pull request as ready for review January 20, 2026 14:29
cursor[bot]

This comment was marked as outdated.

cursor[bot]

This comment was marked as outdated.

cursor[bot]

This comment was marked as outdated.

cursor[bot]

This comment was marked as outdated.

cursor[bot]

This comment was marked as outdated.

Copy link
Copy Markdown
Contributor

@mzitnik mzitnik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's also write unhappy path tests

  • wrong paths/types
  • Empty values

Comment thread ClickHouse.Driver.Tests/Types/JsonTypeTests.cs
Comment thread ClickHouse.Driver/Types/BinaryTypeEncoder.cs Outdated
cursor[bot]

This comment was marked as outdated.

cursor[bot]

This comment was marked as outdated.

cursor[bot]

This comment was marked as outdated.

Copy link
Copy Markdown

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Comment thread ClickHouse.Driver/Json/JsonTypeRegistry.cs
@alex-clickhouse alex-clickhouse merged commit 878b826 into main Jan 22, 2026
17 of 18 checks passed
@alex-clickhouse alex-clickhouse deleted the json-binary-writing branch January 22, 2026 09:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants