Apache Paimon C++ 0.3.0

Apache Paimon C++ 0.3.0

Apache Paimon C++ 0.3.0 is the first Paimon C++ release published by the Apache Software Foundation.

Paimon C++ provides native, high-performance, and extensible access to the Apache Paimon lake format for C++ engines and services without JVM dependencies.

Supported Features

Core Table Access

Paimon C++ provides native write, commit, scan, and read support for append-only and primary-key tables:

• Batch and streaming scans without changelog
• Primary-key reads with Deletion Vectors and Merge-on-Read
• Arrow-based batch read and write interfaces
• Java-compatible commit messages, data splits, manifests, and storage formats

Append tables support fixed-bucket and unaware-bucket modes. Primary-key tables support fixed-bucket and postpone-bucket modes.

Data Types

Paimon C++ supports boolean, integer, floating-point, string, binary, decimal, date, timestamp, array, map, row, variant, and blob data types.

Notable data capabilities include:

• Variant: Java-compatible encoding, JSON conversion, path-based access, projection, and read/write support with Variant shredding
• Blob: Streaming reads and writes, Blob descriptors, multiple Blob fields, cross-table Blob views, and configurable fallback handling
• Schema evolution: Field addition, removal, rename, reorder, and compatible type changes

Extensible Plugin Architecture

Paimon C++ provides extensible abstractions and factory-based plugin mechanisms for integrating file formats and filesystem implementations. Memory pools, thread pools, and index plugins are also customizable.

Built-in file-format plugins include:

• Apache ORC
• Apache Parquet
• Apache Avro

Built-in filesystem plugins include:

• Local filesystem
• Alibaba Jindo filesystem

Compaction and Write Efficiency

Compaction support includes:

• Append-table automatic and full compaction
• Primary-key universal, level-0, lookup, and Deletion-Vector-aware compaction
• Local and remote lookup files with caching

For memory-constrained primary-key writes, Paimon C++ provides spillable external-sort buffers to limit in-memory data usage.

Indexes and AI-Oriented Features

Paimon C++ supports RowTracking and DataEvolution, including global row IDs, partial-column writes, and reconstruction of rows stored across multiple files.

File-level indexes include Bitmap, Bit-Sliced Index, Bloom Filter, and Range Bitmap.

Global-index capabilities include:

• B-tree indexes for point and range queries
• Range Bitmap indexes for scalar range predicates
• Lumina vector indexes with tag filtering
• Lucene-based full-text indexes
• Experimental Tantivy-based full-text indexes

The global-index framework also supports distributed result serialization, score propagation, filtering, caching, and index-file lifecycle management.

Read Performance

Read-path optimizations include:

• Column pruning, page-level predicate pushdown, Bitmap pushdown, and file-statistics filtering
• File prefetching and multi-threaded row-to-batch conversion
• Optimized COUNT(*) query

System Tables and Table Operations

Table-scoped system tables expose options, schemas, snapshots, manifests, files, audit logs, binlogs, and read-optimized $ro views. Global system tables under the sys database expose catalog-level information about tables, partitions, catalog options, and table options.

Supported table operations include snapshot creation and timestamp-based lookup, truncate, abort, rollback-to-latest, conflict detection, idempotent commits, and cleanup of snapshots, partitions, orphan files, and index files.

Source Release

The official Apache source release is available from:

• Source archive
• SHA-512 checksum
• OpenPGP signature
• Apache Paimon KEYS

Please use the official ASF source archive. The source archives automatically generated by GitHub are not official Apache release artifacts.

Documentation

• Paimon C++ documentation
• Paimon C++ repository

Acknowledgements

Thank you to everyone who contributed code, documentation, testing, reviews, release verification, and project infrastructure to Apache Paimon C++.

What's Changed

• chore: Add development tooling configs, contribution guidelines, and third-party license notices by @lxy-9602 in #1
• chore: add apidoc doc-config files by @lucasfang in #3
• feat(cmake): migrate build utility modules by @zjw1111 in #4
• feat: add common utilities including BinPacking, LinkedHashMap, LongCounter, Math, ScopeGuard, ThreadsafeQueue, and UUID by @dalingmeng in #5
• feat: introduce base utilities including Status, Defs, Executor, and MemoryPool by @lxy-9602 in #2
• feat: introduce basic data types including decimal, blob and timestamp by @lszskye in #10
• feat(cmake): add find package modules by @zjw1111 in #7
• fix(cmake): parse add_paimon_lib link arguments. by @slfan1989 in #9
• chore: add thirdparty toolchain and thirdparty diff by @lucasfang in #6
• feat: introduce data type with JSON serialization by @lszskye in #13
• feat(build): migrate build support scripts by @zjw1111 in #12
• feat: introduce common/memory module by @lxy-9602 in #11
• feat: add Range, RangeHelper, RowRangeIndex, and ByteRangeCombiner ut… by @dalingmeng in #19
• feat: add common data structures (BinarySection, BinaryString, InternalRow) by @lxy-9602 in #18
• Configure squash-only merge button by @leaves12138 in #29
• feat: add GenericLruCache, ConcurrentHashMap, MurmurHash, and Precond… by @dalingmeng in #21
• feat: introduce binary row format with reader/writer support by @lszskye in #22
• feat: add StringUtils, DateTimeUtils, PathUtil, OptionsUtils, RapidJsonUtil, and Jsonizable utilities by @dalingmeng in #23
• feat: migrate common/data/columnar module by @lxy-9602 in #24
• chore: add fix_includes.py for include cleanup workflow by @lucasfang in #25
• feat: add GenericRow, RecordBatch, MemorySize and TimeDuration by @lszskye in #28
• feat: add predicate module with Literal, LeafPredicate, and CompoundPredicate by @lxy-9602 in #27
• fix(types): make RowKind short string parsing case-insensitive by @slfan1989 in #15
• feat(fs): introduce file system abstractions by @zjw1111 in #26
• feat: add binary row serializers and blob implementation by @lszskye in #31
• feat: add projected_array, projected_row, field comparator, partition… by @lszskye in #32
• feat(third-party): introduce roaring bitmap and xxhash by @zjw1111 in #34
• feat: add RoaringBitmap32, RoaringBitmap64, and BitSet utilities by @lxy-9602 in #35
• feat: add predicate leaf functions by @lxy-9602 in #33
• feat: add search predicates and PredicateBuilder and tests by @lxy-9602 in #36
• feat: add IO cache (LRUCache, CacheManager, CacheInputStream) and MemorySlice utilities by @lxy-9602 in #40
• feat(fs): introduce file system tests by @zjw1111 in #42
• feat: add BloomFilter, CRC32C, MurmurHash, varint utilities, and DeltaVarintCompressor by @lxy-9602 in #37
• feat: Migrate format interfaces, column_stats, table special_fields, and lookup store by @lxy-9602 in #43
• feat: Migrate SST block infrastructure and file reader/writer by @lxy-9602 in #44
• feat: add data_converter_utils, field_type_utils, file_type and decim… by @lszskye in #38
• feat: add IO stream infrastructure by @lszskye in #39
• feat: add executor future helpers and metrics utilities by @lucasfang in #47
• feat(format): introduce parquet format components by @zjw1111 in #48
• feat: Migrate file index interfaces, format, reader, result, and factory by @lxy-9602 in #45
• feat: add batch reader infrastructure by @lszskye in #46
• feat: Migrate bitmap file index implementation by @lxy-9602 in #49
• feat(fs): introduce local file system by @zjw1111 in #41
• feat: Migrate bloom filter and BSI file index implementations by @lxy-9602 in #50
• feat(format): introduce parquet reader and writer by @zjw1111 in #51
• feat(parquet): introduce stats extraction and predicate pushdown by @zjw1111 in #53
• feat: Migrate block compression infrastructure with LZ4, ZSTD, and none backends by @lxy-9602 in #52
• feat(schema): add table schema and schema manager by @lucasfang in #54
• feat(schema): add schema validation utilities by @lucasfang in #55
• feat: Migrate range bitmap file index and chunked dictionary by @lxy-9602 in #56
• feat: add core options and utility helpers by @lucasfang in #59
• feat: migrate record reader and IO modules by @lxy-9602 in #58
• feat: Migrate projection reader and IO modules by @lxy-9602 in #61
• feat: add core table metadata utilities by @lucasfang in #60
• feat(core): introduce storage metadata components by @zjw1111 in #62
• feat: add data evolution reader infrastructure by @lszskye in #64
• feat: add global index infrastructure by @lszskye in #65
• feat: Migrate file store scan modules by @lxy-9602 in #66
• feat: add simple stats core utilities by @lucasfang in #68
• feat: add deletion vector support with bitmap implementation and index file I/O by @lucasfang in #69
• feat: add prefetch file batch reader implementation by @lucasfang in #70
• feat(core): introduce read context by @zjw1111 in #63
• feat(core): introduce merge tree spill buffers by @zjw1111 in #67
• feat: Migrate bucket module by @lxy-9602 in #71
• feat(core): introduce merge tree write buffers by @zjw1111 in #72
• feat(core): introduce merge tree writer by @zjw1111 in #82
• feat: add global index implementations by @lszskye in #73
• feat: add BTree global index core components by @lszskye in #74
• feat: Migrate split read modules by @lxy-9602 in #76
• feat: Migrate merge split read modules by @lxy-9602 in #77
• feat(core): add data file meta with versioned serializers and path factory by @lucasfang in #80
• feat: add file writer infrastructure and data/compact increment models by @lucasfang in #81
• feat: add read-ahead cache, serialization, stream and object utils by @dalingmeng in #75
• feat(manifest): add manifest and index manifest support by @lucasfang in #87
• feat: Migrate merge function module by @lxy-9602 in #83
• feat(format): introduce avro file format by @zjw1111 in #84
• feat: add type casting infrastructure for schema evolution by @lszskye in #79
• feat(manifest): add manifest entry, file metadata and serialization utilities by @lucasfang in #86
• feat: Migrate sort merge reader by @lxy-9602 in #85
• feat: add index file metadata and handler infrastructure by @lszskye in #78
• feat: Migrate pk compact rewriters by @lxy-9602 in #97
• feat: Migrate lookup file support for pk compact by @lxy-9602 in #96
• feat: Migrate aggregate merge functions by @lxy-9602 in #93
• feat: add file store commit and manifest merger support by @lucasfang in #95
• feat(core): add append-only/key-value file store write and compaction… by @lucasfang in #92
• feat: add file store write and restore utilities by @lucasfang in #91
• feat: introduce table scan, split and plan by @lszskye in #99
• feat(btree): add btree global index integration tests by @lszskye in #90
• feat(btree): add btree global index reader/writer/factory by @lszskye in #89
• feat(format): introduce blob file format by @zjw1111 in #98
• feat(format): introduce avro direct encoding by @zjw1111 in #94
• feat(format): introduce avro writer components by @zjw1111 in #88
• feat: introduce type casting executors for schema evolution by @lszskye in #100
• feat: introduce type casting executors for Binary, Date, and Timestam… by @lszskye in #101
• feat: implement TableRead and FallbackDataSplit by @lszskye in #111
• feat: Migrate lookup compact rewriter and remote manager by @lxy-9602 in #102
• test: Migrate read integration tests by @lxy-9602 in #106
• test: Migrate compaction integration tests by @lxy-9602 in #107
• test: add FileStoreCommitImpl test coverage by @lucasfang in #103
• feat(mergetree): add compaction core components and level management by @lucasfang in #104
• feat(mergetree): add compact manager, task abstractions and file rewr… by @lucasfang in #105
• feat(test): introduce clean and blob table integration tests by @zjw1111 in #108
• feat(testing): introduce mock testing utilities by @zjw1111 in #109
• feat(common): introduce arrow utilities by @zjw1111 in #110
• test: add integration tests for table scan functionality by @lszskye in #133
• test: add comprehensive tests for CastExecutor by @lszskye in #132
• feat: add table scan, split generation and read infrastructure by @lszskye in #131
• test: Migrate data evolution and global index integration tests by @lxy-9602 in #116
• feat: Migrate Lucene global index for fts by @lxy-9602 in #117
• feat: Migrate global index scan and file meta utilities by @lxy-9602 in #125
• feat(docs): introduce C++ documentation by @zjw1111 in #126
• feat(cmake): introduce build configuration by @zjw1111 in #127
• feat(operation): introduce cleanup operations by @zjw1111 in #128
• feat(append): add append-only writer, compact task and coordinator by @lucasfang in #112
• feat(append): add bucketed append compact manager and coordinator tests by @lucasfang in #113
• feat: add compact and postpone bucket writer utilities by @lucasfang in #114
• feat: add snapshot, snapshot_info, and table core utilities with tests by @lucasfang in #118
• feat(core): introduce core options by @zjw1111 in #115
• feat(build): introduce build scripts and public API headers by @zjw1111 in #129
• test(data): introduce avro orc and parquet fixtures by @zjw1111 in #139
• feat: add commit message and system table utilities by @lucasfang in #119
• feat(catalog): add catalog interfaces, file system catalog and snapshot by @lucasfang in #120
• feat(format): add ORC format adapter, stream implementations and builder utilities by @lucasfang in #121
• feat: Migrate blob file context and byte range utilities by @lxy-9602 in #130
• feat: Migrate audit log and binlog system tables by @lxy-9602 in #140
• fix: Move append compact files to core append directory by @lxy-9602 in #142
• test: add scan_and_read_inte_test by @lszskye in #134
• feat(orc): introduce ORC format infrastructure for statistics extraction, predicate pushdown, and read range generation by @lszskye in #135
• test(orc): add unit tests for orc format by @lszskye in #136
• feat: add ORC format reader/writer utilities by @lucasfang in #122
• test: add testing utilities for data generation, row conversion and r… by @lucasfang in #123
• test: add write integration test coverage by @lucasfang in #124
• feat: introduce FieldMapping and FieldMappingBuilder by @lszskye in #137
• fix: move append_only_writer files to core module by @lucasfang in #141
• fix: fix compile and format by @lucasfang in #143
• chore: add github workflows and update asf yaml by @lucasfang in #144
• chore: update homepage url links by @lucasfang in #146
• fix: wait for async tasks before early return in orphan files cleaner by @zjw1111 in #150
• chore: clean up test data README whitespace by @zjw1111 in #147
• feat(fs): introduce Jindo file system support by @zjw1111 in #148
• feat(index): introduce Lumina vector index and enable Lucene builds by @zjw1111 in #151
• chore: clean up license and notice attribution by @zjw1111 in #153
• fix: sync missing changes after repository migration by @zjw1111 in #154
• chore: prepare Apache Paimon C++ 0.2.3 release by @zjw1111 in #155
• chore: add and harden Apache release tooling by @zjw1111 in #156
• chore: enable rebase merges by @zjw1111 in #160
• feat: migrate Paimon C++ 0.3 development history by @lxy-9602 in #161
• chore: prepare the repository for v0.3.0 by @zjw1111 in #162
• docs: standardize Paimon C++ naming by @zjw1111 in #168
• feat(parquet): skip page headers of unselected pages via OffsetIndex-based direct read plan by @zhf999 in #167
• fix(release): backport reproducible source archives to 0.3 by @zjw1111 in #178

New Contributors

• @lucasfang made their first contribution in #3
• @dalingmeng made their first contribution in #5
• @slfan1989 made their first contribution in #9
• @leaves12138 made their first contribution in #29

Full Changelog: https://github.com/apache/paimon-cpp/commits/v0.3.0