Skip to content

v0.5.0

Latest
Latest

Choose a tag to compare

View all tags
@JingsongLi JingsongLi released this 13 Nov 07:21
· 96 commits to main since this release
v0.5.0
bb6389a

Apache Paimon Rust 0.5.0 is the source release approved from RC2, at commit bb6389a8b430e897e038ff7395b6a7763d785a7a.

Source release

Source archive, signature and SHA-512 checksum · KEYS

The attached source archive, signature and checksum are the exact artifacts approved in the release vote.

Highlights

  • Append, primary-key and data-evolution table compaction, including lookup changelog production and incremental primary-key compaction.
  • Prepared and captured mutation plans, native MERGE execution, distributed dynamic-bucket input execution, and expanded Arrow row-ID inputs for updates and deletes.
  • Native batch and streaming read improvements, REST query authorization filters and masks, catalog branch and version management, and external table engine time travel.
  • DataFusion 55.2 integration with expanded SQL functions, JSON operations, lateral collection generators, table and column comments, and tag, branch and consumer procedures.
  • Built-in full-text search, BTree-accelerated upsert key lookups, declared global-index maintenance, and scheduled snapshot-scoped hybrid search execution.
  • Broader Java-compatible schema evolution, nested Parquet projection, temporal and decimal casts, VARIANT and ROW reads, and manifest codec support.
  • Optional node-shared disk block caching, cached Parquet metadata, coalesced local-cache misses, and reuse of data-file, BTree and ROW sidecar metadata.
  • Expanded C commit-message transport and schema APIs, Python streaming SQL results, temporal matching kernels and compaction APIs.

Compatibility notes

  • The minimum supported Rust version is 1.95.0. This release uses DataFusion 55.2.0, Arrow 59.3.2 and Parquet 59.3.2 in its lockfile.
  • Lumina vector-index support has been removed. Use vindex DiskANN through create_global_index with index_type => 'diskann', and rebuild existing Lumina indexes in the vindex format.
  • Full-text search is now built in. Remove the former fulltext Cargo feature from downstream dependency feature lists.
  • Prebuilt Go Linux amd64 libraries require glibc 2.38 or newer, OpenSSL 3, and libffi.so.8; see the Go prerequisites.

Changes since 0.4.0

  • feat(blob): support custom URI readers in native writes (#1065)
  • feat(table): add prepared MERGE operations to core updates (#1068)
  • Support Paimon table sources in prepared MERGE (#1071)
  • chore: bump version to 0.5.0 (#1073)
  • feat(auth): apply query-auth row filters on read (#513)
  • feat(datafusion): support tag retention in create_tag (#1066)
  • feat(table): honor numeric incremental timestamp scan options (#1074)
  • [core] Extract shredded Variant fields from typed columns without row-wise decoding (#1069)
  • feat(core): add stateful StreamTableScan for Native streaming reads (#1075)
  • feat(index): support dry run when dropping global indexes (#1078)
  • [core] Reuse supplied REST data token during native table initialization (#1076)
  • [core] Align Parquet writer row group and page sizing options with Java (#1072)
  • fix(table): validate partial-update delete config and mixed batches (#957)
  • feat(core): align batch changelog planning and row-kind normalization (#1079)
  • feat(core): support sparse raw Blob column updates (#1081)
  • feat(core): support scoped Blob sources in table writes and updates (#1082)
  • python: expose SQL execution options (#1084)
  • feat(core): support Java scalar partition values in native table IO (#1083)
  • fix(core): preserve REST Blob view dependency context (#1085)
  • feat: plan source-backed primary-key sorted indexes (#1086)
  • feat(python): expose core vector search and scored results (#1087)
  • [core] Skip shredded Variant residuals that statistics prove null (#1080)
  • feat(auth): apply query-auth masks to non-BLOB columns (#1077)
  • feat(python): expose prepared sorted global index builds (#1088)
  • fix: repair native vector writes and data evolution read ranges (#1090)
  • feat(auth): support query-auth BLOB column masking (#1089)
  • Unify global index construction behind a common table builder (#1091)
  • Support sparse vindex builds with Java-compatible training semantics (#1093)
  • [Core] Honor parallelism in generic global index builds (#1094)
  • [Core] Add snapshot-pinned full-text search with scalar filters (#1095)
  • fix: preserve Java empty string predicates for search refinement (#1096)
  • feat(core): complete native local hybrid search (#1097)
  • [Core] Preserve captured snapshot metadata and empty read views (#1098)
  • feat: support native shard column updates in core (#1099)
  • feat: enable DV merge engine and postpone fixed bucket writes (#1101)
  • [core] Align numeric narrowing and timestamp schema evolution with Java (#1102)
  • [Core] Support caller snapshot properties in native commits (#1104)
  • [Core] Add predicate deletes to TableUpdate (#1105)
  • [core] Align BIGINT and timestamp schema evolution with Java (#1107)
  • feat(datafusion): route metadata tables to registered table engines (#1100)
  • feat(python): support streaming SQL results (#1103)
  • [core] Add confirmed snapshot commit callbacks (#1108)
  • [docs] Publish 0.4.0 release notes and Go Linux requirements (#1114)
  • format_table: fix partition extraction for reordered columns (#1111)
  • [core] Accept canonical REST identifier casing (#1110)
  • fix(auth): stop BLOB view predicate lookups at LIMIT under query-auth (#1092)
  • fix: support REST branches with distinct identities (#1109)
  • Align temporal string schema evolution with Java (#1115)
  • Align local-zone temporal schema evolution with Java (#1117)
  • feat(datafusion): support ALTER TABLE SET options (#1120)
  • Align decimal schema evolution with Java (#1118)
  • feat(datafusion): support snapshot time travel for external engines (#1113)
  • [io] Propagate User-Agent options to Jindo OSS (#1116)
  • [read] Support Java-compatible REST query authorization masks (#1122)
  • [read] Support Java calendar query authorization masks (#1123)
  • feat(datafusion): support timestamp time travel for external engines (#1121)
  • [read] Support transformed REST query authorization filters (#1124)
  • [read] Complete Java scalar REST authorization casts (#1125)
  • Complete Java-compatible schema evolution for native reads (#1128)
  • Support explicit partition selectors in native table scans (#1129)
  • feat(index): support partition-scoped global index drops (#1106)
  • fix(datafusion): resolve temporary objects first in SHOW CREATE TABLE (#1126)
  • Support REST authorized stream and combined incremental reads (#1130)
  • Support primary-key full-text results through scan and read (#1132)
  • Support supplied global-index results in batch scans (#1133)
  • Complete native global-index lifecycle and search planning (#1136)
  • fix: reject overlapping partial-update sequence groups (#1127)
  • fix(io): delete only the named directory in FileIO::delete_dir (#1135)
  • feat(datafusion): support SHOW CREATE TABLE for external engines (#1131)
  • feat(datafusion): support the delete_branch procedure (#941)
  • feat(core): add SystemRead for Native metadata reads (#1137)
  • [core] Support filesystem partitions in SystemRead (#1138)
  • [core] Improve fixed-bucket pruning and expose scan statistics (#1140)
  • [core] Support REST catalog branch management (#1141)
  • [core] Support REST catalog version management (#1142)
  • [core] Normalize Arrow layouts for upserts and data evolution updates (#1144)
  • [core] Support logical Arrow keys and selected upsert values (#1147)
  • [core] Extend Arrow input support for row-level updates (#1149)
  • [core] Preserve decimal predicates and equivalent Arrow match keys (#1150)
  • [core] Fix VARIANT timestamp projection and scalar conversions (#1151)
  • [core] Support additional Arrow inputs in row-level updates (#1152)
  • Support nested ROW predicates in native table reads (#1153)
  • [core] Build deletion-vector row ranges from roaring runs (#1148)
  • fix(core): preserve partial-update deletion state (#1139)
  • feat(datafusion): support create_tag_from_watermark (#1134)
  • Expand native table sources and nullable primary-key writes (#1155)
  • Complete authorized native read views and writer file planning (#1157)
  • Complete native FUSE writes, postpone planning and MERGE callbacks (#1165)
  • Support REST writes on query-authorized tables (#1168)
  • Expand native mutation inputs and partial-write support (#1169)
  • Complete manifest codec support and typed partition specs (#1170)
  • Fix primary-key ROW layout for Variant tables (#1171)
  • [core] Read s3.session-token with the same precedence as the access key (#1156)
  • [core] Reuse the key buffer and row list across sort-merge keys (#1162)
  • perf(variant): Fix slow shredded residual reads (#1166)
  • feat(datafusion): support reset_consumer (#1167)
  • [core] Read sparse bitmap matches from data evolution ROW sidecars (#1173)
  • [core] Align write validation and file metadata with Java (#1172)
  • [core] Align delete and update nullability with Java (#1179)
  • [ci] Remove unused ROW Variant fixture README (#1181)
  • Support append row kinds and historical row updates (#1180)
  • [core] Complete historical write, update and reader predicate parity with Java (#1182)
  • [core] Add bounded parallel table read execution (#1183)
  • feat(core): support structured-data local cache selection (#972)
  • feat(core): protect local metadata and global index cache blocks (#1184)
  • [core] Add opt-in node-shared disk block cache (#1175)
  • [core] Support copying stream scan state in Python bindings (#1187)
  • [core] Reduce shared disk cache lock contention (#1186)
  • [core] Support scheduled execution of snapshot-scoped search plans (#1191)
  • build: update TLS dependencies for RustSec advisories (#1194)
  • [core] Preserve append writer configuration per bucket across prepares (#1188)
  • [core] Downcast sequence and kind columns once per merge batch (#1193)
  • [core] Compare FLOAT/DOUBLE partition values like Java (#1190)
  • feat(core): support full-text prefilters on non-DE append tables (#1176)
  • [core] Support authorized raw file planning for native scans (#1197)
  • [core] Use BTree indexes for upsert key lookups (#1198)
  • [core] Expose postpone bucket plans through the write builder (#1200)
  • [core] Add captured mutation and index build plans with structured row-id conflicts (#1202)
  • [core] Share MERGE action evaluation for distributed execution (#1203)
  • [python] Use typed MERGE clauses and named execution factories (#1204)
  • [core] Expose phased hybrid search and pure score fusion (#1206)
  • [core] Add schema-based bucket routing and postpone input planning (#1207)
  • Support distributed dynamic bucket input execution (#1208)
  • Support captured row ID routing and read selection (#1209)
  • Support Arrow row ID inputs for deletion (#1210)
  • fix(table): move BinaryRow import into upsert tests (#1201)
  • Update locked ctutils and Tokio security patches (#1212)
  • Fix partition field mapping for vector search source plans (#1211)
  • perf(core): cache ROW sidecar size and reader metadata (#1196)
  • feat(datafusion): support clear_consumers (#1213)
  • fix(datafusion): preserve MERGE clause fallthrough for NULL predicates (#1220)
  • [core] Coalesce consecutive local cache misses (#1218)
  • [datafusion] Expose procedure execution for external integrations (#1217)
  • feat(datafusion): support replace_tag (#1214)
  • fix(ci): update zstd dependencies for security checks (#1229)
  • perf: skip source stat for complete shared-cache small files (#1228)
  • perf: reuse known BTree file sizes when opening cached readers (#1227)
  • [core] Support configurable sort order for partial-update sequence groups (#1226)
  • feat: share parsed Parquet metadata across read contexts (#1223)
  • fix(storage): retry temporary S3 failures (#1222)
  • fix: prevent concurrent snapshot overwrites (#1231)
  • [core] Remove Lumina vector index support (#1233)
  • test: replace Spark provisioning with golden fixtures (#1234)
  • feat: make full-text search a built-in capability (#1232)
  • perf: reuse known sizes when opening data file readers (#1235)
  • chore: raise minimum supported Rust version to 1.95 (#1237)
  • ci: run Python binding Rust unit tests (#1241)
  • perf: stream and spill scalar global-index sorting (#1244)
  • fix(read): honor read.batch-size for primary-key sort-merge output (#1239)
  • perf: certify full-text Top-K and reuse write buffer settings (#1245)
  • fix(table): create branches from tags with expired snapshots (#1221)
  • feat(table): add append and primary-key compaction APIs (#1247)
  • feat(parquet): project nested map value fields (#1246)
  • feat(read): complete nested projection pushdown for Parquet (#1248)
  • feat(table): support data evolution compaction (#1249)
  • feat(table): support lookup changelog production (#1252)
  • feat(python): add batch temporal matching kernels (#1238)
  • feat(table): support incremental deduplicate compaction (#1253)
  • feat(datafusion): add create_branch SQL procedure (#1250)
  • feat(table): cache lookup history for incremental compaction (#1254)
  • feat(table): compact primary-key tables with deletion vectors (#1255)
  • feat(table): extend incremental primary-key compaction (#1256)
  • feat(table): compact after writes when write-only=false (#1259)
  • fix(schema): list persisted schemas independently of catalog cache (#1260)
  • feat(table): maintain declared global indexes after writes (#1261)
  • fix(datafusion): preserve MERGE predicate grouping across clauses (#1263)
  • feat(c): support cross-process commit messages (#1257)
  • feat(table): unify index options and maintain PK/global indexes (#1264)
  • fix(datafusion): preserve string literals when rewriting MERGE references (#1266)
  • fix(table): align native vector index configuration semantics (#1270)
  • fix(rest): retry GET connection failures with bounded backoff (#1272)
  • feat(datafusion): add JSON text extraction and formatting functions (#1279)
  • fix(datafusion): resolve registered functions before REST lookup (#1277)
  • feat(datafusion): support scoped and filtered metadata listings (#1278)
  • feat(datafusion): add common SQL compatibility functions (#1280)
  • build(deps): bump astral-sh/setup-uv from 10.0.1 to 10.3.3 (#1295)
  • build(deps): bump pypa/gh-action-pypi-publish from 1.14.2 to 1.14.3 (#1294)
  • build(deps): bump taiki-e/install-action from 2.87.1 to 2.88.0 (#1293)
  • feat(datafusion): support table and column comment DDL (#1287)
  • feat(datafusion): evaluate immutable constant CALL arguments (#1290)
  • feat(datafusion): add common scalar compatibility functions (#1291)
  • feat(datafusion): support binary casts and byte widths (#1288)
  • feat(datafusion): support RLIKE and REGEXP operators (#1289)
  • feat(datafusion): support lateral collection generators (#1292)
  • build(deps): bump crossbeam-channel from 0.5.16 to 0.5.17 (#1296)
  • build(deps): bump twox-hash from 2.1.3 to 2.1.5 (#1300)
  • build(deps): bump lz4_flex from 0.13.1 to 0.14.0 (#1297)
  • build(deps): bump opendal-layer-retry from 0.58.2 to 0.59.5 (#1298)
  • feat(datafusion): extend aggregate, JSON and SQL dialect compatibility (#1304)
  • feat(datafusion): integrate upstream Spark-compatible functions (#1308)
  • feat(datafusion): support CALL sys.compact procedure (#1312)
  • feat(datafusion): support binary lengths, slices and split_index (#1311)
  • [datafusion] Upgrade to DataFusion 55.2.0 (#1236)
  • feat(core): support startup modes in StreamTableScan (#1313)
  • [datafusion] Expose range partitioning for table partition keys (#1314)
  • [core] Preserve zero-width binary rows in upsert selection (#1317)
  • [datafusion] Execute MERGE INTO through the native planner (#1315)
  • feat(rest): support server-resolved warehouses (#1305)
  • fix(table): move rows to their new bucket when UPDATE changes the bucket key (#1319)
  • feat(c): export a pinned table's Arrow schema (#1331)
  • feat(c): add bounded commit-message range export (#1330)
  • fix(datafusion): reject UPDATE and DELETE on temporary targets (#1325)
  • feat(python): expose table compaction through new_compact_builder (#1323)
  • [core] Read single-run primary-key sections without sort merge (#1164)
  • fix(datafusion): preserve quoted names in temporary views (#1333)
  • feat(c): expose TableWrite close for explicit cleanup (#1306)
  • [Bug] DROP TABLE deletes the wrong table when targets span multiple catalogs (#1339)
  • [Build] Upgrade paimon-full-text to 0.2.0 (#1344)
  • fix(datafusion): reject TRUNCATE on temporary targets (#1335)
  • feat(datafusion): support branch procedures on REST catalog tables (#1346)
  • fix: replace persisted video keyframe indexes with MP4 metadata (#1347)
  • fix(datafusion): preserve quoted table names in time-travel queries (#1360)
  • perf(core): index nested-key rows by their serialized BinaryRow key (#1354)
  • perf(core): keep rbm64 bitset containers as words through unions (#1350)
  • [core] Keep the winning cell by reference in pick-value aggregators (#1338)
  • fix(write): flush retained input before dictionary encoding exhausts its budget (#1329)

Full changelog: v0.4.0...v0.5.0

Assets 5
Loading