release-2.1.0

Source releases

Apache Paimon 2.1.0 source, signature and checksum

PyPaimon 2.1.0 source, signature and checksum

KEYS for signature verification

What's Changed

• Use repository secrets for PyPI publishing by @JingsongLi in #8951
• [core] Fix index source metadata compatibility by @JingsongLi in #8952
• [core] Fix error classification for invalid bucket key by @XiaoHongbo-Hope in #8974
• [python] Fix predicate pruning with incomplete statistics by @XiaoHongbo-Hope in #8978
• Bump brace-expansion from 1.1.16 to 1.1.18 in /docs by @dependabot[bot] in #8984
• [python] Fix benchmark package lint by @XiaoHongbo-Hope in #8986
• [core] Fix forced L0 compaction level selection by @JingsongLi in #8993
• [core][flink][spark] Introduce postpone fixed-bucket write builder by @JingsongLi in #8992
• [flink] Speed up lookup join tests by @JingsongLi in #8997
• [core] Avoid strict-mode conflicts for index-only compact by @LsomeYeah in #8996
• [format] Fix reads and predicate pushdown when Parquet files differ from the declared schema by @sundapeng in #8995
• [core] Fix chain table silent data loss for DV rows by @ArnavBalyan in #8990
• [flink] Support SQL DELETE for data evolution tables by @XiaoHongbo-Hope in #8988
• [orc] Support time type predicate push down for orc by @ArnavBalyan in #8983
• [python] Parse overrides in ConfigResponse and fix merge priority by @wenhaozhao011-cmd in #8989
• [format] Fix DELETE and UPDATE targeting wrong rows after Parquet predicate filtering by @XiaoHongbo-Hope in #8987
• [docs] Fix procedure parameter names by @lilei1128 in #8975
• [flink] Recover coordinator commit without intended failover by @ifndef-SleePy in #8970
• [filesystem][s3] Fix multipart part upload missing SSE-C encryption parameters by @thswlsqls in #8969
• [oss] Update bundled NOTICE for the aliyun-sdk-oss 3.17.4 upgrade by @thswlsqls in #8967
• [spark] Fix mergedRecordSize truncation to zero by @cxzl25 in #8965
• [core] Add conflict detection for concurrent merge-into & compaction by @steFaiz in #8964
• [format][python] Support additional MAP BLOB key types by @QuakeWang in #8963
• [obs] Add listFilesIterative implementation in HadoopCompliantFileIO by @thswlsqls in #8960
• [filesystem][obs] Remove access key and secret from required options in OBSLoader by @thswlsqls in #8958
• [docs] Add dls:// to the supported filesystems table by @thswlsqls in #8956
• [fs] Fix JindoFileIO to return actual file access time and owner by @thswlsqls in #8955
• [spark] Avoid scanning partition entries without partition_idle_time by @lilei1128 in #8932
• [core] Add privilege checks for partition and consumer ops by @cxzl25 in #8929
• [core] Limit index-only compact handling to data evolution by @LsomeYeah in #9001
• [flink] Expose sorted index build stream by @leaves12138 in #8998
• [core] Preserve index manifest compatibility by @JingsongLi in #9004
• [format] Avoid lossy Parquet predicate pushdown by @JingsongLi in #8999
• [vortex] Preserve physical row order in scans by @huwh in #9003
• [python] Expand native Rust scan planning by @XiaoHongbo-Hope in #9000
• [jindo] Disable directory timestamps for faster listings by @sundapeng in #9019
• [core] Narrow format table partition listing by name pattern by @sundapeng in #9025
• [spark] Make max_pt work on format tables by @sundapeng in #9024
• Bump undici from 7.28.0 to 7.29.0 in /docs by @dependabot[bot] in #9028
• [format] Reject incompatible annotated integers as BIGINT by @sundapeng in #9010
• [core][flink][spark][python][vector] Support vector indexes on DV tables by @XiaoHongbo-Hope in #8930
• [core] Support TopN pushdown for BTree global indexes by @discivigour in #8991
• [parquet] Support boolean predicate push down for parquet by @ArnavBalyan in #8971
• [core] Support full-text indexes on DV tables by @XiaoHongbo-Hope in #9031
• Bump fast-uri from 3.1.4 to 3.1.5 in /docs by @dependabot[bot] in #9029
• [python] Add Python 3.12, 3.13 to CI by @PyRSA in #9018
• [python] Add CI coverage for native split planning by @XiaoHongbo-Hope in #9011
• [format][python] Support TIME keys for MAP BLOB by @QuakeWang in #9009
• [core] Fix dropping stats in data evolution scan by @leaves12138 in #8994
• [python] Write postpone batches to fixed buckets by @XiaoHongbo-Hope in #8985
• [python] Reuse manifest merge results on commit retry by @XiaoHongbo-Hope in #9040
• [core] Support BLOB fields in partial-update tables. by @Stephen0421 in #8928
• [core] Remove versioned serializer abstraction by @JingsongLi in #9039
• [spark] Stage postpone fixed-bucket writes before rescale by @Zouxxyy in #9017
• [flink] Clone Paimon action supports add additional table options by @yuzelin in #9042
• [fs] Delegate tryToWriteAtomic through FileIO wrappers by @plusplusjiajia in #9034
• [spark] Support skipping expired partitions during compaction by @lilei1128 in #9021
• [core] Fix scan from tag on branch table. by @Stephen0421 in #9044
• [spark] Recompute bucket number for postpone overwrite by @Zouxxyy in #9051
• [core][python] Fix global index coverage for residual predicates by @XiaoHongbo-Hope in #9050
• [flink] Fix Flink version detection for key-only deletes on non-release versions by @thswlsqls in #9053
• [python] Plan data evolution row ranges with pypaimon-rust by @XiaoHongbo-Hope in #9038
• [python] Group write rows by partition/bucket in Arrow to avoid GIL-bound per-row extraction by @yugan95 in #9047
• [core][flink][spark] Refactor global index building and support incremental refresh by @leaves12138 in #8818
• [core] Cache projected schemas for data evolution stats by @leaves12138 in #9046
• [core] Optimize BTree negative predicate complement by @zhoulii in #8945
• [core] Centralize negative predicate complements in sorted index readers by @JingsongLi in #9060
• [spark] Cache blob field lookup in SparkInternalRow by @zhuxiangyi in #8943
• [core] Fix file operation thread limit by @huangxiaopingRD in #9082
• [hive] Support multiset type in hive writes by @thswlsqls in #9081
• [hive] Unescape partition path values when building Hive split predicate by @thswlsqls in #9080
• [hive] Fix ClassCastException when reading BLOB columns by @thswlsqls in #9079
• [hive] Fix Hive read failure for CHAR/VARCHAR columns exceeding Hive length limits by @thswlsqls in #9075
• [hive] Normalize column comments written to Hive metastore by @thswlsqls in #9073
• [fs] Delegate listFilesIterative through RESTTokenFileIO by @tonymtu in #9071
• [spark] Improve unresolved expression errors in MERGE INTO by @Zouxxyy in #9068
• [core][flink][spark] Support default bucket number for postpone writes by @Zouxxyy in #9067
• [avro] Fix ClassCastException when writing MULTISET with non-string element type by @thswlsqls in #9066
• [format] Skip the CSV header only in the split that contains it by @thswlsqls in #9065
• [docs] Fix full-text index option defaults and search query examples by @thswlsqls in #9062
• [flink] Report which required argument is missing in action factories by @thswlsqls in #9059
• [format][python] Support binary keys for MAP BLOB by @QuakeWang in #9057
• [cdc] Fix NPE when metadata columns are used with debezium-bson format by @thswlsqls in #9055
• [core] Tag diff query at postpone table should check bucket number by @yuzelin in #9048
• [core] Validate data types for max/min aggregate functions by @jackylee-ch in #9013
• [python] Align atomic write temporary file naming by @zhoulii in #9027
• [docs] Align scalar index search docs with code by @huangxiaopingRD in #9084
• [cdc] Support PostgreSQL jsonb type in postgres sync by @ArnavBalyan in #9020
• [core] Abort single file writers on unchecked close failures by @wombatu-kun in #9016
• [core] Fix wrong privilege type in NoPrivilegeException for assertCanCreateDatabase by @cxzl25 in #9015
• [spark][flink] Fix vector-field column conversion in catalogs by @jackylee-ch in #9014
• [python] Defer BLOB payload reads until after filtering by @XiaoHongbo-Hope in #8808
• [core] Share JDBC connection pool across catalog instances in same JVM by @nickdelnano in #8323
• [core] Fix SnapshotManager watermark binary searches with null watermarks by @u70b3 in #9037
• [python] Inherit watermark in PyPaimon commits by @JingsongLi in #9090
• [core][mosaic] Preserve projected raw reads by @jianguotian in #9086
• [spark] Rebase data evolution merge updates after compaction by @JingsongLi in #9091
• [spark] Remove upsert-key support by @JingsongLi in #9093
• [spark][python] Preserve rows when rebasing stale updates by @JingsongLi in #9096
• [core][flink][python] Support nullable primary keys by @JingsongLi in #9094
• [core] Reduce memory for update index by @leaves12138 in #9092
• [core] Support projected binary index manifest scans by @leaves12138 in #9095
• [python] Support DLF token path authentication by @liujiayi771 in #9089
• [python] Fix native-plan CI with updated paimon-rust by @XiaoHongbo-Hope in #9109
• [python][ray] Add row-id range processing API by @JingsongLi in #9112
• [python] Forward scan options to native planner by @XiaoHongbo-Hope in #9124
• [python] Avoid an extra file-size I/O for Blob reads by @XiaoHongbo-Hope in #9125
• Bump nanoid from 3.3.16 to 3.3.18 in /docs by @dependabot[bot] in #9110
• [core][python] Prune data evolution groups by merged stats by @XiaoHongbo-Hope in #9114
• [python] Align IN predicates with SQL null semantics by @JingsongLi in #9132
• [format] Fix RLE skipping at sparse row-range boundaries by @CrownChu in #9127
• [format] Fix cumulative DELTA_BINARY_PACKED read count by @leaves12138 in #9137
• [core] Add file index system table by @lilei1128 in #9138
• [core] Fix partition timestamp pattern replacement for prefix variable names by @juntaozhang in #9136
• [core] Honor query-level search-mode overrides by @CrownChu in #9128
• [core] Carry the row count and byte size a format table writer already counted by @sundapeng in #9121
• [core] Spell out what a negative partition statistic means by @sundapeng in #9120
• [flink] Fix nullable primary key test across Flink versions by @QuakeWang in #9118
• [rest] Do not replay a POST the server cannot absorb twice by @sundapeng in #9122
• [common] Fix concurrent remote stream leak in CachingSeekableInputStream by @wombatu-kun in #9098
• [core] Fix file index filtering with deletion vectors by @huangxiaopingRD in #9088
• [iceberg] Fix Iceberg metadata after a rollback by @plusplusjiajia in #9036
• [flink] Fix flaky test duplicate nullable PK IT by @ArnavBalyan in #9117
• Fix #9113 Exception when reading Paimon table via Hive proxy user in … by @tyoushinya in #9116
• [common] Fix timestamp partition values being written but not readable by @ArnavBalyan in #9115
• [format][python] Reject duplicate MAP BLOB keys by @QuakeWang in #9107
• [core] Reject Variant type for primary key and partition key by @ArnavBalyan in #9105
• [core] Fix stale default value in sort-compaction.range-strategy description by @jackylee-ch in #9104
• [lance] Reject unsupported element types nested in ARRAY by @jackylee-ch in #9102
• [python] Reuse streams for coalesced blob reads by @XiaoHongbo-Hope in #9133
• [python] Keep native planning for Blob descriptor reads by @XiaoHongbo-Hope in #9144
• [core] Remove configurable manifest format by @leaves12138 in #9154
• [lance] Reject unsupported VARIANT and BLOB types in LanceFileFormat schema validation by @thswlsqls in #9150
• [python] Fix row-id updates for Arrow columns over 2 GiB by @discivigour in #9146
• [spark][test] Use explicit Paimon provider in feature tests by @Zouxxyy in #9145
• [core] Add block-based manifest Avro reader by @JingsongLi in #9160
• [spark] Fix MERGE INTO silently skipping WHEN NOT MATCHED BY SOURCE actions by @kerwin-zk in #9155
• [python] Add efficient VARIANT path get and replace by @XiaoHongbo-Hope in #9147
• [flink] Fix flaky Lumina data evolution index refresh IT by @ArnavBalyan in #9153
• [python] Align exact VARIANT integer and decimal types by @XiaoHongbo-Hope in #9168
• [spark] Respect blob-compaction.enabled in Spark compact procedure by @jackylee-ch in #9165
• [format] Support delimiter as fallback key of csv.field-delimiter by @Zouxxyy in #9170
• [core] Compact adjacent overlapping blob ranges together by @leaves12138 in #9175
• [spark] Fix IndexOutOfBoundsException for vector_search with too few arguments by @jackylee-ch in #9164
• [core] Release every reader in SortMergeReaderWithMinHeap.close() by @PDGGK in #9163
• [core] Restore the interrupt status in the two JDBC catalog paths that drop it by @PDGGK in #9159
• [iceberg] Warn when the Iceberg metadata committer factory is missing from the classpath by @bube in #9158
• [iceberg] Omit unknown bounds in Iceberg manifest partition summary by @thswlsqls in #9152
• [core] Fix IS NAN predicate failing with file index by @ArnavBalyan in #9143
• [python] Fix locale-sensitive Date header and nonce replay on retry by @sundapeng in #9173
• [common] Finish the lookup-store close chain instead of stopping at the first failure by @PDGGK in #9172
• [spark] Improve compaction logs in CompactProcedure by @Zouxxyy in #9169
• [core] Unify manifest Avro readers by @leaves12138 in #9182
• [core][flink] Remove file-based privilege system by @JingsongLi in #9186
• [core] Fix level 1 inflation in universal compaction by @tsreaper in #9185
• [core] Fix invalid length when reading spilled data by @tsreaper in #9183
• [core] Parallelize primary-key full-text search by @QuakeWang in #9184
• [flink] Fix CSV delimiter fallback for format tables by @ArnavBalyan in #9178
• [core] Register every bucket in bucketList so overflow rows spread by @PDGGK in #9180
• [spark] Add Format Table partition operation tests and clearer errors by @sundapeng in #9119
• [core] Optimize data evolution compaction planning by @JingsongLi in #9177
• [test] Fix URI-based temporary paths by @JingsongLi in #9191
• [flink] Avoid overlapping sorted index key ranges by @leaves12138 in #9189
• [spark] Introduce reassign_row_id procedure by @zhuxiangyi in #9192
• [core] Add manifest Avro writer by @leaves12138 in #9194
• [core] Refactor conflict detection by table type by @JingsongLi in #9196
• [core] Allow first-row merge engine for primary-key managed BLOB tables. by @Stephen0421 in #9201
• [core][format] Support Arrow bundle writes by @jianguotian in #9100
• [spark] Retry data evolution compaction after merge conflicts by @JingsongLi in #9197
• [python] Fix pyjindo advanced config setting in pypaimon by @timmyyao in #9206
• [common] Fix like optimization for escaped patterns by @ArnavBalyan in #9205
• [common] Fix incorrect date for casting pre epoch time by @ArnavBalyan in #9203
• [docs] Clarify snapshot record count semantics by @wwj6591812 in #9202
• [spark] Do not display an unknown partition statistic as a number by @sundapeng in #9200
• [python] Support create_partitions in RESTCatalog by @plusplusjiajia in #9198
• [core] Fix dynamic bucket allocation across partitions by @ArnavBalyan in #9193
• [iceberg] Exclude NaN from manifest partition bounds by @ArnavBalyan in #9188
• [core] avoid reading stale blobs during fallback read. by @steFaiz in #9208
• [core] Avoid repeated sorted-index source lookups by @QuakeWang in #9187
• [flink] Avoid redundant HTTP BLOB existence probes by @wwj6591812 in #9181
• [core] Fix incorrect file indexes table range filtering by @ArnavBalyan in #9176
• [core] Support latest snapshot delta scan in batch by @wwj6591812 in #9142
• [common] Fix missing rows for null range bitmap indexes by @ArnavBalyan in #9140
• [spark] Prevent table deletion for self referencing RTAS by @ArnavBalyan in #9139
• [python] Avoid Arrow offset overflow when merging read batches by @XiaoHongbo-Hope in #9209
• Bump org.apache.logging.log4j:log4j-api from 2.25.4 to 2.25.5 by @dependabot[bot] in #9215
• [docs] Sync procedure documentation by @huangxiaopingRD in #9101
• [core] Fix stale file index cleanup by @huangxiaopingRD in #9085
• [common] Compare maps of different representations without throwing by @PDGGK in #9211
• [flink][spark] Support manifest sort in compact_manifest by @discivigour in #9083
• [core] Support arbitrary time granularity for chain table delta computation by @juntaozhang in #8185
• [core] Optimize ordinary manifest merging with raw Avro blocks by @leaves12138 in #9213
• [core] Speed up row-id manifest sorting by @leaves12138 in #9135
• [core] Fix incremental diff scan missing first snapshot by @ArnavBalyan in #9234
• [core] Parallelize primary-key sorted-index scan by @QuakeWang in #9231
• [core] Support DATE, TIMESTAMP and TIMESTAMP_NTZ in variant shredding schema by @juntaozhang in #9230
• [service] Fix KvQueryClient hang when query location lookup fails on retry by @thswlsqls in #9229
• [core] Close every writer and shut both executors down when one close fails by @PDGGK in #9227
• [flink] Fix scan partitions with partial partition spec by @ArnavBalyan in #9225
• [spark] Fix missing data for delete hiding remaining records by @ArnavBalyan in #9224
• [format] Validate blob file footer and index metadata by @Stephen0421 in #9223
• [spark] Add option for Hive-style dynamic partition writes by @kerwin-zk in #9222
• [core] Validate deletion vector mode when resetting table options by @ArnavBalyan in #9237
• [python] Update paimon-rust to speed up native-plan CI by @XiaoHongbo-Hope in #9235
• [python] Add tests for Data Evolution BLOB writes on branches by @XiaoHongbo-Hope in #9221
• [spark][flink] Support bucket-level compaction for fixed-bucket tables by @lilei1128 in #9219
• [python][ray] Fix update_by_row_id failure on empty datasets by @XiaoHongbo-Hope in #9217
• [spark] Fix bucketed scan reporting for unsupported timestamp precision by @zhuxiangyi in #9216
• [python] Fix GenericVariant UUID decoding for Java compatibility and support datetime/UUID by @juntaozhang in #9204
• [python] Stream scalar update_by_predicate assignments by file group by @XiaoHongbo-Hope in #9240
• [core] Support dynamic read batch sizing by @Gabriel39 in #9243
• [python] Support callable assignments in predicate updates by @XiaoHongbo-Hope in #9239
• [core] Refactor read batch sizing API by @JingsongLi in #9248
• [python] Preserve manifest order in Data Evolution splits by @XiaoHongbo-Hope in #9242
• [core] Support geospatial data types by @JingsongLi in #9251
• [core] Reuse projected entries in manifest run merge by @leaves12138 in #9241
• [ci][python] Fix lint-python native-plan CI failure by @XiaoHongbo-Hope in #9255
• [core] Close the data writer even when the changelog writer fails by @PDGGK in #9254
• [spark] Merge into query fails for is non null condition by @ArnavBalyan in #9252
• [python] Avoid concurrent import failures during split deserialization by @XiaoHongbo-Hope in #9250
• [core] Compare binary elements and keys by value in collect and merge_map by @PDGGK in #9249
• [oss] Add fs.oss.cname.enabled so non-Aliyun endpoints sign the bucket by @PDGGK in #9238
• [core] Take the stream position outside the finally so close always runs by @PDGGK in #9233
• [common] Fix ClassCastException when computing row size for blob fields by @ArnavBalyan in #9087
• [core] Fix file index pushdown for bitmap64 deletion vectors by @QuakeWang in #9141
• [core] Fix snapshots table failing IN filter for unknown id by @ArnavBalyan in #9134
• [core] Avoid external sort for fragmented manifests by @leaves12138 in #9257
• [python] Tighten lazy import comments by @XiaoHongbo-Hope in #9260
• [spark] Expose written columns for streaming micro-batches by @LsomeYeah in #9023
• [python] Avoid deleting files during distributed commit failures by @JingsongLi in #9269
• [python] Fix import deadlock: lazy-resolve package exports, drop custom locks by @XiaoHongbo-Hope in #9267
• [iceberg] Emit row-lineage metadata fields for Iceberg format version 3 by @vbabenkoru in #9244
• [flink] Fix unstable changelog compaction test by @QuakeWang in #9281
• [java][python] Add VARIANT path upsert via variant_set by @XiaoHongbo-Hope in #9253
• [core] Add Multivalue global index primitives by @JingsongLi in #9288
• [core] Add Multivalue global index for array membership by @JingsongLi in #9282
• [rest] Fix OpenAPI contract for client generation by @JingsongLi in #9289
• [core] Support array set predicates in multivalue index by @JingsongLi in #9294
• [core] Remove MultiValueIndexFileMeta by @JingsongLi in #9305
• [python] Add vectorized splice fast path for variant_set root inserts by @XiaoHongbo-Hope in #9259
• [spark] Push down array predicates to multivalue indexes by @JingsongLi in #9306
• [common] Avoid temporary long[] allocations when compressing blob footer index by @Stephen0421 in #9307
• [core] optimize column directives by @steFaiz in #9304
• [core] Fix cascading re-merge in BinaryExternalSortBuffer spill merge by @zhoulii in #9303
• [arrow] Fix nanosecond timestamp conversion for pre-epoch values by @jackylee-ch in #9299
• [core][rest] Report partition statistics when registering partitions by @sundapeng in #9295
• [core] Validate the carried-over default value when a column type changes by @PDGGK in #9292
• [core] Keep tied nulls in range bitmap TopN for multi-key sorts by @jackylee-ch in #9280
• [core] Match the lookup high level record by position, not by identity by @PDGGK in #9279
• [format] Escape the escape character when writing CSV by @PDGGK in #9278
• [core] Build expired partitions from their values instead of a joined string by @PDGGK in #9277
• [format] Stop aliasing the output vector when tracking the previous delta value by @PDGGK in #9275
• [python] Introduce hll_sketch aggregator function by @zhuxiangyi in #9274
• [core][python] Record full build version in snapshots by @leaves12138 in #9283
• [spark] Support consumer progress in streaming source by @LsomeYeah in #9273
• [hive] Fix chain table HMS partition drop skipped by fallback scan. by @Stephen0421 in #9272
• [api] Resume truncated HTTP GET response bodies by @wwj6591812 in #9271
• [core] Preserve failures during manifest rewrite cleanup by @QuakeWang in #9270
• [iceberg] Support custom table properties passthrough to Iceberg REST catalog by @JTaky in #9268
• [core] Keep the deletion-vector meta cache when a table is copied by @PDGGK in #9266
• [core] Read expire_tags older_than as a wall clock, not as an instant by @PDGGK in #9263
• [flink] Surface a failed asynchronous lookup refresh instead of dropping it by @PDGGK in #9262
• [format] Close the row-file input stream when the footer or index fails to parse by @PDGGK in #9258
• [common] Fix NullPointerExceptions on null transform inputs by @plusplusjiajia in #9214
• [iceberg] Assign manifest-level row lineage for Iceberg format version 3 by @vbabenkoru in #9245
• [python] Group row tracking commits by partition by @YannByron in #9312
• [flink] Fix literal conversion for BETWEEN predicates by @QuakeWang in #9310
• [core] Encapsulate split serialization protocols by @leaves12138 in #9316
• Bump org.eclipse.jetty:jetty-security from 12.1.8 to 12.1.10 in /paimon-iceberg by @dependabot[bot] in #9311
• [core] Add an optional positional long[] of per-column maximum sequence numbers to DataFileMeta by @leaves12138 in #9313
• [core] Avoid global index rebuilds for unrelated column compaction by @leaves12138 in #9069
• [core] Report partition statistics from format table commits by @sundapeng in #9296
• [spark] Fix timestamp precision in key dynamic writes by @Zouxxyy in #9322
• [python][ray] Push down self-merge matched conditions by @XiaoHongbo-Hope in #9314
• [core][spark] Support collecting format table partition statistics in MSCK REPAIR TABLE by @sundapeng in #9297
• [python] Support callable assignments in Ray self-merge by @XiaoHongbo-Hope in #9327
• [core] Optimize data evolution conflict scans by @leaves12138 in #9334
• [spark]to DE insert into path,the target table should also be target splits by @weijietong in #9333
• [core] Fix flaky column type update test by @ArnavBalyan in #9325
• [python][ray] Avoid routing shuffle in self-merge updates by @XiaoHongbo-Hope in #9335
• [python] Support target file row num for blob and vector writes by @YannByron in #9317
• [flink] Fix asynchronous partition refresh generation race by @QuakeWang in #9318
• [paimon-vector] Upgrade paimon-vector-index to 0.4.0 by @shyjsarah in #9341
• [core][spark][flink] Support TRUNCATE TABLE on format tables by @sundapeng in #9330
• [core][spark] Support ANALYZE TABLE on catalog-managed format tables by @sundapeng in #9298
• [core][spark] Use format partition row counts in scan statistics by @JingsongLi in #9351
• [core][spark] Fix format table INSERT OVERWRITE ignoring dynamic-partition-overwrite by @sundapeng in #9353
• [python][ray] Rebase self-merge updates after compaction by @XiaoHongbo-Hope in #9339
• [python][ray] Flatten large self-merge predicates by @XiaoHongbo-Hope in #9361
• [Spark] Support reading nested null fields for Parquet struct by @ArnavBalyan in #9364
• [format] Support numeric type widening when reading Avro files by @ArnavBalyan in #9362
• [format] Introduce support for Parquet byte stream split encoding by @ArnavBalyan in #9359
• [common] Fix wildcard matching for line terminators by @ArnavBalyan in #9358
• [spark] Fix MERGE INTO delete clause corrupting partial-update tables by @zhuxiangyi in #9356
• [spark] Forbid delete and update on first-row table (#6718) by @RunningDB in #9355
• [core] Fix null handling in ArrayContains by @QuakeWang in #9354
• [common] Correct local dependency for string conversion by @ArnavBalyan in #9346
• [format] Fix corrupt output by overlapping custom line delimiters by @ArnavBalyan in #9344
• [iceberg] Skip an already-deleted manifest list when expiring old metadata by @lucasgameiroborges in #9343
• [Fix] Support KEY_DYNAMIC bucket mode in CompactorSinkBuilder by @zhang-arvin in #9338
• [core] Fix incorrect substring result for unicode characters by @ArnavBalyan in #9328
• [common] Fix substring transform query failure for non int type by @ArnavBalyan in #9326
• [core] Fix scan version for fallback branch by @ArnavBalyan in #9323
• [flink] Avoid having QueryFileMonitor always busy by @thswlsqls in #9291
• [flink] Shut down KvQueryServer when QueryExecutorOperator closes by @thswlsqls in #9287
• [core] Fix flaky concurrent duplicate file discard test by @ArnavBalyan in #9285
• [flink] Support savepoint auto-tag on coordinator commit path by @ifndef-SleePy in #9309
• [vector] Expose IVF search tuning options by @shyjsarah in #9367
• [api] Support external Jackson for simple REST requests by @1fanwang in #9369
• [core] Reduce Variant buffer copies across Arrow and Parquet paths by @JingsongLi in #9371
• [api][spark] Treat an unreported partition statistic as unknown, not as zero by @sundapeng in #9379
• [spark] Resolve primary-key and bucket-key properties like other identifiers by @zhuxiangyi in #9372
• [core] Prune value columns when building lookup files for PersistPositionProcessor by @liangjie3138 in #9374
• [core][python] Fix partial ranges when updating newly added Blob columns by @zhoulii in #9388
• [core] Fix nullability parsing for nested types by @zhoulii in #9381
• [core] Quiesce manifest rewrite workers before cleanup by @QuakeWang in #9376
• [spark] Introduce CTAS support for partitioned tables by @ArnavBalyan in #9378
• [spark] Fix conditional query failing on variant casts by @ArnavBalyan in #9385
• [core][spark] Push down object table filters by @Stephen0421 in #9383
• [common] Follow Spark's SUBSTRING position semantics and count characters in TRIM by @plusplusjiajia in #9329
• [python] Fix quadratic cost of writing multiple batches by @yugan95 in #9375
• [common][python] Support SUBSTRING and TRIM in column masking by @plusplusjiajia in #9212
• [api] Add REST permission management by @JingsongLi in #9393
• [flink] Avoid slot deadlock when sorting normalized index keys by @leaves12138 in #9394
• [flink] Fix lookup output projection with internal key fields by @leaves12138 in #9392
• [core] Manifest read pool in expiration use scanManifestParallelism by @yuzelin in #9395
• [api] Add REST policy management by @JingsongLi in #9400
• [format] Fix Parquet delta length reads across vector batches by @ArnavBalyan in #9366
• [core][iceberg] Include table name in commit/Iceberg-commit exception by @JTaky in #9391
• [dcos] Supplementary document description by @lilei1128 in #9404
• [hive] Avoid binding to HiveConf.ConfVars constants renamed in Hive 4 by @leosanqing in #9403
• [docs] Add REST management OpenAPI by @JingsongLi in #9406
• [core] Add public binary serialization for BucketVectorSearchSplit by @JunRuiLee in #9386
• [flink] Do not shuffle to a negative channel when the partition hash is Integer.MIN_VALUE by @PDGGK in #9408
• [core] Expose bitmap index format APIs by @leaves12138 in #9414
• [core] Snapshot expiration deletes data and changelog files in batches by @yuzelin in #9409
• [core][test] Add REST management query authorization support by @JingsongLi in #9416
• [core][spark] Add REST management procedures by @JingsongLi in #9410
• [core][spark] Fix reading newly added BLOB columns after MERGE by @zhoulii in #9413
• [python][torch] Add batch-first streaming datasets by @XiaoHongbo-Hope in #9365
• [python][ray] Adapt default partitions to input size by @XiaoHongbo-Hope in #9357
• [core] Compare IcebergRef by value by @PDGGK in #9405
• [spark] Advance streaming source past empty full snapshots by @lilei1128 in #9401
• [common] Release the BTree index input when construction or close fails by @PDGGK in #9407
• [core] Fix shared decompression buffer in stable manifest blocks by @yuzelin in #9430
• [core] Improve row ID reassignment under concurrent writes by @JingsongLi in #9439
• [core] Add manifest file name in ManifestFileBlockMerger exception message for troubleshooting by @yuzelin in #9432
• [core] Parallelize row ID reassignment manifest rewrites by @JingsongLi in #9441
• [flink] Commit format table overwrites once by @sundapeng in #9433
• [spark] Reject SET LOCATION for format tables by @sundapeng in #9435
• [spark] Reject empty partition values for format tables by @sundapeng in #9436
• [python] Add HDF5 DataSource ingestion by @YannByron in #9411
• [core] Stream row ID reassignment manifest rewrites by @JingsongLi in #9443
• [python] Support MAP<K, BLOB> in multimodal BLOB reads by @XiaoHongbo-Hope in #9428
• [python] Default multimodal vector files to Parquet by @JingsongLi in #9444
• [core] Add exact partitioned FM global index by @JingsongLi in #9442
• [python] Fix BLOB CRC32 compatibility with Java by @liangjie3138 in #9426
• [spark] Fix min and max query failure on metadata columns by @ArnavBalyan in #9431
• [flink] Release both catalogs FlinkGenericCatalog wraps by @PDGGK in #9422
• [core] Reject resetting ignore-delete and ignore-update-before from true by @PDGGK in #9421
• [format] Fix reading Avro nullable unions when null value is in second branch by @ArnavBalyan in #9418
• [flink][cdc] Fix bucket-aware state recovery for multi-table CDC sink by @Stephen0421 in #9417
• [api][core] Parallelize format table overwrite commit by @sundapeng in #9397
• [core] Support FM indexes for primary-key tables by @JingsongLi in #9447
• [flink] Support unaligned checkpoints for coordinator commit by @JingsongLi in #9451
• [core] Harden and slim FM index storage by @JingsongLi in #9450
• [core][flink][spark] Treat non-positive row counts as unknown by @sundapeng in #9456
• [python] Add DataFusion extra for merge conditions by @XiaoHongbo-Hope in #9463
• [common] Release the input stream when FileIndexPredicate fails to open it by @PDGGK in #9462
• [core]rename FileStoreCommitImpl var name by @weijietong in #9457
• [iceberg] Refuse nanosecond timestamps while Iceberg metadata is enabled by @plusplusjiajia in #9455
• [api] Stop reading a server error message as a format string by @plusplusjiajia in #9454
• [format] Fix ORC zstd compression option inconsistent across writers by @ArnavBalyan in #9453
• [api] Fix race condition causing queued tasks to execute after SequentialBatchIterator close by @juntaozhang in #9452
• [metrics] Add totalFileCount and minAvgFileSize metrics to detect small file problem by @junmuz in #9449
• [hive] Fix query failure when dynamic filters are unavailable by @ArnavBalyan in #9448
• [python] Add RoboMIND AgileX HDF5 pipeline by @YannByron in #9445
• [format][python] Add packed video frame storage by @JingsongLi in #9461
• [flink][cdc] Fix missing primary keys for Debezium records by @ArnavBalyan in #9465
• [Fix] MongoDB CDC: handle null fullDocument in transaction to prevent NullNode exception (#9332) by @zhang-arvin in #9336
• [iceberg] Allow publishing VARIANT columns with Iceberg format version 3 by @vbabenkoru in #9246
• [core][python] Reuse PK sorted indexes with retired source files by @wangyong9999 in #9460
• [core] Fix column masking correctness in query-auth reads by @plusplusjiajia in #8570
• [format][python] Refactor video storage internals by @JingsongLi in #9470
• [flink] Fix decimal column analysis with null statistics by @ArnavBalyan in #9469
• [core] Read the table branch schemas when rewriting file indexes by @plusplusjiajia in #9468
• [iceberg] Refuse time precisions the mirror cannot publish by @plusplusjiajia in #9467
• [python][torch] Support distributed iterable dataset sharding by @XiaoHongbo-Hope in #9429
• [spark] Fix flaky row tracking concurrent compaction test by @JingsongLi in #9475
• [core][python] Support multiple packed video fields by @XiaoHongbo-Hope in #9473
• [multimodal] Support aligned multi-video file groups by @JingsongLi in #9480
• [python] Add LeRobot Dataset v3 import by @XiaoHongbo-Hope in #9446
• [spark] Clamp Format Table ANALYZE parallelism by @sundapeng in #9482
• [format] Support nested variant column pruning for shredded parquet files by @juntaozhang in #9389
• [core] Rename FM index identifier to fm by @JingsongLi in #9487
• [python][torch] Make map-style reads lazy by @XiaoHongbo-Hope in #9486
• [spark] Support parallel sorted index building across Paimon partitions by @liangjie3138 in #9491
• [python] Reject overlapping row-id update batches by @YannByron in #9484
• [spark] Support partition pruning for data evolution self-merge by @zhoulii in #9489
• [spark] Support timestamp_ntz and variant in generic row access by @shyjsarah in #9493
• [spark] Skip time-travel self-merge test on Spark 3.2 by @sundapeng in #9502
• [spark] Show format table partition statistics by @sundapeng in #9501
• [python] Add distributed RoboMIND action backfill by @YannByron in #9496
• [iceberg] Leave the nanosecond refusal to the validator that knows the table by @plusplusjiajia in #9503
• [python][ray] Cover conditional self-merge compaction rebase by @JingsongLi in #9516
• [core] Support changelog-producer.ignore-update-before and changelog-producer.ignore-delete options by @junmuz in #9531
• [python] Add ROSBag ingestion support by @YannByron in #9530
• [common][spark] Fix z-order boolean FALSE colliding with the null sentinel by @LuciferYang in #9527
• [common] Make BinaryRow.anyNull respect the row offset by @LuciferYang in #9525
• [common] Grow from one byte in MemorySliceOutput when the segment is empty by @LuciferYang in #9523
• [spark] Let MSCK see the null partition of a value-only format table by @sundapeng in #9521
• [common] Fix double position advance in DataOutputSerializer.writeBytes by @LuciferYang in #9520
• [common] Reject non-struct inner fields in variant shredding schema by @LuciferYang in #9518
• [common][spark] Fix boolean hilbert value colliding with the null sentinel by @LuciferYang in #9515
• [common] Fix partition getter indexing in InternalRowPartitionComputer by @LuciferYang in #9513
• [common] Fix permit accounting of SemaphoredDelegatingExecutor under interruption and rejection by @LuciferYang in #9511
• [test-util] Set accessible on @parameters provider and @parameter field by @thswlsqls in #9510
• [pvfs] Set working directory to the catalog root in PaimonVirtualFileSystem by @thswlsqls in #9507
• [docs] Fix stale paimon-vindex version in vector index documentation by @thswlsqls in #9505
• [Bug] Fix concurrent LocalTableQuery lookups returning null across files by @zhang-arvin in #9500
• [format] Introduce byte stream split for writing to parquet by @ArnavBalyan in #9499
• Bump browserslist from 4.28.2 to 4.28.8 in /docs by @dependabot[bot] in #9542
• [core]make stats model work while migrating by @weijietong in #9541
• [docs] Fix misleading DROP COLUMN doc for hive catalog by @sd4324530 in #9538
• [rest] Make RESTTokenFileIO cache maximum size configurable by @tonymtu in #9528
• [Bug] Fix debezium-json cannot infer primary keys from standard Debezium message key by @zhang-arvin in #9495
• [format] Preserve csv values matching the configured null literal by @ArnavBalyan in #9492
• [spark] Fix analyze failure for timestamp ntz columns by @ArnavBalyan in #9485
• [format] Prevent text tables from accepting unsupported schemas by @ArnavBalyan in #9479
• [flink] Fix schema inference for multi partition Kafka topics by @ArnavBalyan in #9478
• [format] Fix positional schema evolution for ORC reads by @ArnavBalyan in #9464
• [python] Expose Arrow batch reader for multimodal scans by @XiaoHongbo-Hope in #9537
• [spark] Support action predicate pruning for data evolution self-merge by @zhoulii in #9544
• Bump fast-uri from 3.1.5 to 3.1.7 in /docs by @dependabot[bot] in #9554
• [python] Cache decoded BLOB indexes by @YannByron in #9547
• [python] Infer multimodal vector columns by query dimension by @wangzhigang1999 in #9553
• [core][spark][flink] Support sub-field-level data evolution for nested columns by @zhuxiangyi in #8334
• [format] Give each row-format reader its own projected-row wrapper by @LuciferYang in #9561
• [python] Parse BlobDescriptor v1/v2 bytes without misclassifying inline payload by @Stephen0421 in #9539
• [core][spark][flink] Fix nested data evolution isolation by @JingsongLi in #9564
• [format] Support sep as fallback key of csv.field-delimiter by @jackylee-ch in #9565
• [format] Keep parsing remaining fields after a malformed one in PERMISSIVE CSV mode by @LuciferYang in #9556
• [format] Read BINARY-encoded decimals at scratch index 0 in parquet updater by @LuciferYang in #9563
• [vortex] Fix predicate pushdown literals for DATE and second-precision TIMESTAMP by @thswlsqls in #9558
• [format] Compare array wrapper-group names case-insensitively in ParquetReaderUtil by @LuciferYang in #9568
• [spark] Use the field index instead of the field id for TopN pushdown by @jackylee-ch in #9566
• [core][python][spark] Optimize data evolution write column metadata by @JingsongLi in #9574
• refactor(core): make SchemaManager an interface by @baiyangtx in #9174
• [api][core][spark] Support custom partition locations by @sundapeng in #9540
• [docs] Regenerate the configuration reference by @LuciferYang in #9592
• [python] Make multimodal table creation race-safe by @XiaoHongbo-Hope in #9590
• [python] Notify callbacks for successful retried commits by @XiaoHongbo-Hope in #9589
• [parquet] Push down StartsWith as Parquet binary range by @lilei1128 in #9594
• [python] Persist LeRobot dataset metadata in Paimon by @XiaoHongbo-Hope in #9529
• [common] Fail on a short remote read instead of caching zero padding by @jackylee-ch in #9593
• [api] Trim whitespace in the bucket-key option value by @jackylee-ch in #9591
• [format] Report unresolvable JSON string casts instead of a bare NPE by @LuciferYang in #9588
• [core] Clean up data file extra files by @zjw1111 in #9586
• [format] Build fresh ORC writer options per created writer by @LuciferYang in #9583
• [format] Close the text stream when line-reader construction fails by @LuciferYang in #9579
• [common] Fix pre-epoch timestamp casts truncating toward zero by @jackylee-ch in #9577
• [format] Close the ParquetFileReader when post-construction setup fails by @LuciferYang in #9576
• [format] Close the Avro stats extractor's stream on a corrupt file by @LuciferYang in #9573
• [format] Fail fast when skipping DELTA_LENGTH_BYTE_ARRAY data past end of page by @LuciferYang in #9571
• [spark] Reject variant extraction paths containing the metadata delimiter by @jackylee-ch in #9569
• [spark] Disable create table in default db by @junmuz in #9549
• [spark] Fix batch delete for partial update sequence groups by @ArnavBalyan in #9546
• [core] Avoid retaining rejected cache pages by @yuzelin in #9658
• [vortex] Honor write.batch-memory in the writer factory by @thswlsqls in #9605
• [docs] Document the VECTOR data type by @thswlsqls in #9603
• [core] Fix stale option key in clustering.strategy description by @thswlsqls in #9602
• [arrow] Fix second-precision timestamp conversion for pre-epoch values by @thswlsqls in #9600
• [spark] Fix inconsistent snapshots for data evolution V1 UPDATE by @zhoulii in #9597
• [Flink] Add configurable initial scan mode for streaming compaction by @lilei1128 in #9548
• [core] Unify block cache ownership and avoid redundant SST reads by @JingsongLi in #9663
• [core] Support reordered fields in manifest Avro files by @JingsongLi in #9683
• [python] Simplify LeRobot metadata tables and training tags by @YannByron in #9676
• [ci][python] Fix Python 3.6 CI after Bullseye EOL by @XiaoHongbo-Hope in #9684
• Bump js-yaml from 4.1.1 to 4.3.1 in /docs by @dependabot[bot] in #9685
• [python] Fix Arrow offset overflow in merge condition filtering by @XiaoHongbo-Hope in #9678
• [flink] Honor snapshot limit in exactly-once monitor source by @jianguotian in #9677
• [python][torch] Add lazy contiguous window dataset by @YannByron in #9580
• [python] Invalidate commit retry cache after overwrite by @XiaoHongbo-Hope in #9688
• [python] Add end-to-end test for row-id reassignment conflict by @XiaoHongbo-Hope in #9689
• feat(core): Support schema management over REST in RESTCatalog by @baiyangtx in #9673
• [python] Support snapshot properties in table commits by @YannByron in #9692
• [python][ray] Fix count metadata for row-level authorization by @QuakeWang in #9681
• Bump colord from 2.9.3 to 2.10.0 in /docs by @dependabot[bot] in #9698
• Bump svgo from 3.3.4 to 3.3.5 in /docs by @dependabot[bot] in #9701
• Bump js-yaml from 4.3.1 to 4.3.2 in /docs by @dependabot[bot] in #9699
• Bump joi from 17.13.4 to 17.13.7 in /docs by @dependabot[bot] in #9700
• [python] Add direct LeRobot capture writer by @YannByron in #9665
• [python] Add paired ACT storage benchmark by @YannByron in #9466
• [python] Remove duplicate BLOB validation during table creation by @wangzhigang1999 in #9704
• [core] Fix default NOT IN composition in file index reader by @lilei1128 in #9612
• [core] Fall back to renaming when multipart upload is unavailable by @Zouxxyy in #9610
• [python] Add Paimon LeRobot map-style dataset by @XiaoHongbo-Hope in #9498
• [core][spark] Let DROP PARTITION repair overlapping Format Table locations by @sundapeng in https://github.com/apache/paimon/pull/9712
• [docs] Explain Data Evolution design and file contracts with diagrams by @JingsongLi in https://github.com/apache/paimon/pull/9711
• [docs] Organize maintenance guides by task by @JingsongLi in https://github.com/apache/paimon/pull/9713
• [common] Default missing variant shredding fields to the unshredded index by @LuciferYang in https://github.com/apache/paimon/pull/9616
• [common] Always accept the first entry in bitmap index block packing by @LuciferYang in https://github.com/apache/paimon/pull/9614
• [common] Do not zero the HeapBytesVector buffer on reset by @LuciferYang in https://github.com/apache/paimon/pull/9644
• [common] Stop nesting row-tracking vector wrappers per batch by @LuciferYang in https://github.com/apache/paimon/pull/9634
• [common] Overflow-safe URI length check in VideoFrameDescriptor by @LuciferYang in https://github.com/apache/paimon/pull/9618
• [common] Keep escaped characters in StringToMapCastRule entry splitting by @LuciferYang in https://github.com/apache/paimon/pull/9620
• [common] Close the file-index stream when reader construction fails by @LuciferYang in https://github.com/apache/paimon/pull/9636
• [common] Propagate checked IOException from RESTTokenFileIO#fileIO by @LuciferYang in https://github.com/apache/paimon/pull/9638
• [format] Validate nested types for JSON tables by @LuciferYang in https://github.com/apache/paimon/pull/9585
• [common] Run the z-order sign flip at the width of the bit pattern by @LuciferYang in https://github.com/apache/paimon/pull/9628
• [core] Split large normal files in data evolution compaction by @JingsongLi in https://github.com/apache/paimon/pull/9709
• [api][core] Reset Format Table partition locations on overwrite by @sundapeng in https://github.com/apache/paimon/pull/9710
• [python] Preserve LeRobot recording metadata in direct writer by @YannByron in https://github.com/apache/paimon/pull/9708
• [docs] Restructure primary-key table documentation and diagrams by @JingsongLi in https://github.com/apache/paimon/pull/9718
• [docs] Reorganize append table guides and replace diagrams with SVGs by @JingsongLi in https://github.com/apache/paimon/pull/9719
• [docs] Reorganize multimodal table guides and diagrams by @JingsongLi in https://github.com/apache/paimon/pull/9721
• [docs] Restructure concepts documentation and diagrams by @JingsongLi in https://github.com/apache/paimon/pull/9722
• [docs] Reorganize Spark guides and diagrams by @JingsongLi in https://github.com/apache/paimon/pull/9724
• [docs] Reorganize Flink guides and add SVG diagrams by @JingsongLi in https://github.com/apache/paimon/pull/9725
• [docs] Restructure PyPaimon guides and add SVG diagrams by @JingsongLi in https://github.com/apache/paimon/pull/9726
• [python] Import LeRobot video datasets by @XiaoHongbo-Hope in https://github.com/apache/paimon/pull/9494
• [core][python] Add option to skip manifest merge in write-only mode by @JingsongLi in https://github.com/apache/paimon/pull/9727
• [filesystem] Isolate Jindo OSS presigning dependencies (#9856) by @jerry-024 in https://github.com/apache/paimon/pull/9887

New Contributors

• @wenhaozhao011-cmd made their first contribution in #8989
• @huwh made their first contribution in #9003
• @u70b3 made their first contribution in #9037
• @tyoushinya made their first contribution in #9116
• @Gabriel39 made their first contribution in #9243
• @vbabenkoru made their first contribution in #9244
• @lucasgameiroborges made their first contribution in #9343
• @1fanwang made their first contribution in #9369
• @leosanqing made their first contribution in #9403
• @wangyong9999 made their first contribution in #9460

Full Changelog: release-2.0.0...release-2.1.0