feat(blob): support reading map blob values by XiaoHongbo-Hope · Pull Request #278 · apache/paimon-cpp

XiaoHongbo-Hope · 2026-09-02T15:49:17Z

Purpose

Linked issue: #283

Support reading Java-compatible MAP<K, BLOB> payloads from blob files through the real Table API and data-evolution paths.

Decode BOOL, integer, DATE, STRING, BINARY, and DECIMAL keys.
Load top-level MAP<K, BLOB> schemas from table JSON while continuing to reject BLOBs in other nested positions.
Treat existing Java MAP<K, BLOB> tables as read-only in C++: reject C++ table creation, FileStoreWrite::Create, and append compaction until writer support is implemented.
Resolve MAP<K, BLOB> placeholders across multiple sequence layers in both inline and descriptor modes.
Preserve null and empty maps/values, and reject malformed metadata, duplicate keys, non-canonical large DECIMAL encodings, invalid UTF-8 STRING keys, and null map keys.
Reject paimon.map.selected-keys for Map Blob reads because filtering can remove the internal data-evolution fallback placeholder.

TIME keys and ARRAY<BLOB> are intentionally outside this PR and should be handled separately with their end-to-end type and format support.

Tests

ninja -C build-release paimon-blob-format-test
build-release/release/paimon-blob-format-test --gtest_filter='BlobFileBatchReaderTest.*:BlobAsDescriptor/BlobFileBatchReaderTest.*' (17 tests passed)
Focused NestedProjectionUtilsTest.GetMapSelectedKeysRejectsMapBlob test (passed)
Changed production and test translation units compiled with -Werror -Wall
cpplint, codespell, and git diff --check

API and Format

No public API under include/ and no storage format changes. This adds read compatibility for the existing Paimon Java MAP<X, BLOB> payload version 1. C++ writer support is unchanged; table creation, regular writes, and append compaction are explicitly rejected.

Arrow Map requires non-null keys, so Java Map Blob payloads containing null keys are rejected during decoding.

Documentation

No documentation changes.

Generative AI tooling

Generated-by: OpenAI Codex (GPT-5)

duanyyyyyyy · 2026-09-03T08:41:10Z

@lxy-9602 pls take time to review

lxy-9602 · 2026-09-04T07:15:00Z


+Result<std::shared_ptr<arrow::Array>> BlobFileBatchReader::BuildMapBlobArray(
+    int32_t rows_to_read) const {
+    const auto& struct_type = static_cast<const arrow::StructType&>(*target_type_);


This function seems too long. Please break it down into smaller, more fine-grained pieces.

Refactored in 00da44a. Payload parsing/validation, key appends, and value appends are now split into ReadMapBlobPayload, AppendMapBlobKeys, and AppendMapBlobValues; BuildMapBlobArray only coordinates rows and builders.

lxy-9602 · 2026-09-04T07:34:41Z

+    ASSERT_OK_AND_ASSIGN(int64_t written,
+                         output_stream->Write(file_bytes.data(), file_bytes.size()));
+    ASSERT_EQ(file_bytes.size(), written);
+    ASSERT_OK(output_stream->Close());


Consider use file_system->WriteFile

Updated all added golden/corruption fixture writes to use FileSystem::WriteFile in 00da44a.

lxy-9602 · 2026-09-04T07:51:36Z

+        for (int32_t entry = 0; entry < entry_count; ++entry) {
+            const auto key_length = static_cast<int32_t>(key_lengths[entry]);
+            std::vector<uint8_t> key_bytes(key_length);
+            PAIMON_RETURN_NOT_OK(ReadBlobContentAt(key_offset, key_length, key_bytes.data()));


I’m not sure how large key_bytes can get. If it can be large, it would be better to allocate it from the pool, for example via Bytes.

Updated in 00da44a. Key buffers now use pool-backed Bytes; inline value buffers use the same allocation path.

lxy-9602 · 2026-09-04T08:15:30Z

Thank you very much for adding support for map<..., blob>. Would it be possible to open an issue to describe this feature? I’d also like to know whether there is a plan to support writes in the future.

zjw1111 · 2026-09-04T09:23:16Z

Thanks for your work on this! Is there any plan to support reading and writing ARRAY<BLOB> in a follow-up?

lszskye · 2026-09-04T09:57:51Z

+        {"TIME(0)", arrow::time32(arrow::TimeUnit::MILLI)},
+        {"TIME(3)", arrow::time32(arrow::TimeUnit::MILLI)},
+        {"TIME(9)", arrow::time32(arrow::TimeUnit::MILLI)},
+        {"TIME(3) WITHOUT TIME ZONE", arrow::time32(arrow::TimeUnit::MILLI)},


Please also add ut for "TIME(6) WITHOUT TIME ZONE"...

Please also add ut for "TIME(6) WITHOUT TIME ZONE"...

Removed TIME support in this PR.

XiaoHongbo-Hope · 2026-09-04T10:03:10Z

@lxy-9602 I opened #283 to describe the feature and current compatibility boundaries. This PR remains read-only; there is no concrete Map Blob writer timeline yet, current case ready only is enough.

@zjw1111 There is no concrete ARRAY<BLOB> plan yet.

lxy-9602

LGTM. Let’s wait for @lszskye to confirm as well.

XiaoHongbo-Hope added 3 commits September 2, 2026 08:48

feat(blob): support reading map blob values

c90feb4

fix(blob): support map blob table and fallback reads

ce4b0b6

fix(blob): harden map blob schema validation

eddb8ab

XiaoHongbo-Hope marked this pull request as ready for review September 3, 2026 05:46

XiaoHongbo-Hope added 4 commits September 2, 2026 23:08

fix(blob): reject writing map blob tables

ba31553

fix(blob): guard append compaction for map blob

128d47c

fix(schema): require millisecond time32 values

29f54be

fix(test): satisfy gcc range loop warnings

aa5b204

Merge branch 'main' into codex/read-map-blob

2f512b5

lxy-9602 reviewed Sep 4, 2026

View reviewed changes

Comment thread src/paimon/format/blob/blob_file_batch_reader.cpp

lxy-9602 reviewed Sep 4, 2026

View reviewed changes

XiaoHongbo-Hope mentioned this pull request Sep 4, 2026

Support reading MAP<K, BLOB> blob files #283

Open

fix(blob): address map blob review feedback

00da44a

lszskye reviewed Sep 4, 2026

View reviewed changes

XiaoHongbo-Hope and others added 2 commits September 4, 2026 06:06

fix(blob): satisfy clang-tidy

635b1d1

Merge branch 'main' into codex/read-map-blob

a1b3061

lxy-9602 previously approved these changes Sep 4, 2026

View reviewed changes

test: expand map blob read coverage

d287f03

XiaoHongbo-Hope dismissed lxy-9602’s stale review via d287f03 September 4, 2026 14:07

Uh oh!

Conversation

XiaoHongbo-Hope commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Tests

API and Format

Documentation

Generative AI tooling

Uh oh!

duanyyyyyyy commented Sep 3, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lxy-9602 Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

XiaoHongbo-Hope Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

lxy-9602 Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

XiaoHongbo-Hope Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

lxy-9602 Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

XiaoHongbo-Hope Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lxy-9602 commented Sep 4, 2026

Uh oh!

zjw1111 commented Sep 4, 2026

Uh oh!

lszskye Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

XiaoHongbo-Hope Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

XiaoHongbo-Hope commented Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lxy-9602 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

XiaoHongbo-Hope commented Sep 2, 2026 •

edited

Loading

XiaoHongbo-Hope Sep 4, 2026 •

edited

Loading

XiaoHongbo-Hope Sep 4, 2026 •

edited

Loading

XiaoHongbo-Hope Sep 4, 2026 •

edited

Loading

XiaoHongbo-Hope commented Sep 4, 2026 •

edited

Loading