Skip to content

Apache Fluss 1.0.0

Latest
Latest

Choose a tag to compare

View all tags
@wuchong wuchong released this 22 Sep 14:00
· 114 commits to main since this release
v1.0.0
This tag was signed with the committer’s verified signature.
wuchong Jark Wu
GPG key ID: 85BACB5AEFAE3202
Verified
Learn about vigilant mode.
5c07f88
This commit was signed with the committer’s verified signature.
wuchong Jark Wu
GPG key ID: 85BACB5AEFAE3202
Verified
Learn about vigilant mode.

Apache Fluss 1.0.0 is the first major release since Fluss graduated to an Apache Top-Level Project. This release introduces the Fluss Gateway, brings native Rust, Python, and C++ clients into the main project, and expands storage, Lakehouse, Flink, Spark, and production operations capabilities.

📥 Downloads: https://fluss.apache.org/downloads/#apache-fluss-100

📖 Release blog: https://fluss.apache.org/blog/releases/1.0/

⚠️ Before upgrading from 0.9, read the 1.0 Upgrade Notes, especially the Lakehouse upgrade order, ACL changes, and AWS credentials provider migration.

🌐 Fluss Gateway

Features and Improvements

  • Introduce Fluss Gateway as a stateless Rust REST service, with HTTP lifecycle management, health checks, and graceful shutdown. (#3963)
  • Add cluster discovery and metadata APIs for databases, tables, and partitions. (#4078)
  • Add REST APIs for database, table, and partition DDL operations. (#4107)
  • Add schema-aware JSON decoding and batch append, upsert, partial update, and delete operations with per-row outcomes. (#4104, #4126)
  • Add configuration validation and secret redaction. (#3970)
  • Add Prometheus identity labels and improve REST access logs. (#4219, #4142, #4163)
  • Provide Gateway binary distributions and container release support. (#4098, #4306)

Bug Fixes

  • Reject duplicate MAP keys in JSON entry arrays. (#4205)

Gateway is a preview in 1.0. It supports metadata, DDL, and writes; record reads and HTTP caller authentication are not included. See the Gateway guide for deployment requirements and API limitations.

🗄️ Server / Coordinator / Storage

Features and Improvements

  • Add Coordinator high availability, leader election, and coordinator epoch fencing. (bbc445a, c49b848, #2781)
  • Keep standby Coordinators synchronized with dynamic configuration changes. (#3645)
  • Add a cluster health API for checking leaders and in-sync replicas during rolling upgrades. (#3400)
  • Support multiple local data directories on TabletServers. (#3030)
  • Add multiple remote data directory configuration and directory selection when creating tables and partitions. (#2757, #2763, #2769)
  • Add disk-usage write protection with a configurable recovery threshold; prevent KV leader promotion while disk writes are blocked. (#3340, #3612, #3539)
  • Add resource-based admission control for KV leader replicas. (#3614)
  • Add cooperative KV write backpressure based on RocksDB Level-0 file pressure. (#3463)
  • Share RocksDB block cache and write-buffer memory across KV tablets. (#2627, 059d7aa)
  • Add row-level TTL for primary key tables through table.kv.ttl. (#3621)
  • Support independent local log retention with table.log.local-ttl and dynamic updates to table.log.ttl. (#3778, #3233)
  • Support rolling expired active log segments on low-traffic tables, with dynamic configuration updates. (#3831, #3940)
  • Add V1 log batches with column statistics and server-side predicate pruning. (#2886, #2951)
  • Add server-side KV scans for reading primary key tables in batches. (#2976, #3151)
  • Support changing the default bucket count for future partitions while preserving existing partition layouts. (#3908, #4315)
  • Support adding aggregation columns to existing tables. (#3459)
  • Support standby replica election for primary key tables and table-level standby replica configuration; disable standby replicas by default for newly created primary key tables. (#2829, #3257, #4157)
  • Add table-drop throttling to protect the Coordinator event queue. (#3174)
  • Add per-tablet-server request retries and a dedicated executor for leader/ISR notifications. (#3391, #3647)
  • Batch leader/ISR updates during controlled shutdown and require the new leader's acknowledgment for leader-only rebalancing. (#3654, #3556)
  • Add timeouts for rebalance tasks and serialize leader-only rebalance operations. (#3429, #3071)
  • Improve remote log recovery with asynchronous prefetch and remote CDC segment downloads. (#3132, #2855)
  • Periodically checkpoint log recovery points. (#3547)
  • Speed up shutdown by closing KV tablets concurrently and skipping RocksDB flushes during TabletServer shutdown. (#3651, #3807)
  • Return ISR and bucket epoch information in bucket metadata. (#3785)
  • Support dynamic updates to auto-partition enablement, precreation count, and retention count; add the yyyy-MM-dd partition time format. (#3453, #3434, #2964, #3558)
  • Support altering database comments and custom properties. (#1172)
  • Support dynamic updates to the minimum ISR configuration and KV snapshot interval. (#2837, #2820)

Bug Fixes

  • Fix log recovery failures after unclean shutdown, orphan segments after full truncation, and writer recovery with sequence gaps. (#2942, #4013)
  • Recover log segments containing a single empty batch. (0ebc9f2)
  • Fix spurious ISR shrink and races during leader transitions. (#3702, #3920)
  • Fix ZooKeeper recovery thread safety in the rebalance manager and notification cleanup races. (#4232, #3752)
  • Fix missed table-creation events caused by asynchronous metadata cache updates. (#2710)
  • Fix KV snapshot restore fallback and restore the minimum retained offset after follower restarts. (#3790, #4060)
  • Keep flushed log offsets monotonic to preserve row counts after failover. (#3427)
  • Keep asynchronous KV flushes on WAL batch boundaries and detach flushed pre-write buffer entries. (b4b1fb1, dbf908a)
  • Fix overlapping remote log manifest ranges and expired remote log start offsets. (#3822, #3937, #3956)
  • Update remote log offsets before cleaning up local segments. (#4254)
  • Fix remote log index cache cleanup, cache-size overflow, and time-index memory accounting. (#3737, #3755, #3734, #3733)
  • Clean up stale KV directories and orphan files during TabletServer startup. (#4407, #3388)
  • Fall back to rack-unaware assignment when rack information is incomplete. (#3946)
  • Correctly remove partitions with multiple partition keys from the metadata cache. (#3882)
  • Fix periodic snapshot startup and avoid failing leader election on lake snapshot refresh errors. (#2978, #3922)
  • Treat full-schema partial writes as full-row writes. (#3325)
  • Fix CHAR min/max statistics and record-batch equality/hash-code contracts. (#4080, #3952)
  • Fix request-channel backpressure races and bound length-delimited RPC fields by readable bytes. (0bd725c, #3756)
  • Fix Netty buffer leaks on follower fetch timeouts. (#2926)
  • Prevent deletion of the default database and validate auto-partition timestamps. (#3312, #2940)

☕ Java Client

Features and Improvements

  • Add multi-table log scanners and writers. (#3140)
  • Add Scan.createBatchScanner() for whole-table batch reads and support server-side KV scans. (#2794, #3295)
  • Support consuming log data as native Arrow record batches. (#2995)
  • Extend the typed API with ARRAY, MAP, nested ROW, enum conversion, and column aliases. (#2814, #2900, #3447, #3444)
  • Add a remote-first log read preference. (#3606)
  • Reduce the default connection timeout to 15 seconds for faster metadata recovery. (#4095)
  • Support rack selection when adding or removing server tags through the Admin API. (#2890)
  • Cache synchronously fetched schemas and reduce per-record partition-routing overhead. (#3335, #3529)
  • Improve Arrow and Netty memory allocation and introduce lazy zero-copy RPC decoding. (#3026, #2803, #2948)

Bug Fixes

  • Resolve per-partition bucket counts before sending writes. (#4309)
  • Fix idempotent writer retry loops after lost responses; retry stale out-of-order responses without resetting the writer. (#2827, #4125)
  • Continue draining other buckets when one bucket is blocked. (#2953)
  • Abort pending write batches when a table is recreated with a different table ID. (#2747)
  • Clean up batches after synchronous send failures and isolate sender failures to the affected table. (#4028, ec517fe)
  • Fix missing records in primary key batch reads and initialize lake comparators before merge reads. (#3627, #3406)
  • Align fixed-schema reads by column ID across schema evolution. (#3771)
  • Keep UPDATE_BEFORE and UPDATE_AFTER together in a poll batch. (#3149)
  • Fix stale fetch detection across poll boundaries and release fetch permits when discarding results. (#3053, #3043)
  • Prevent EOF during concurrent remote log downloads and cancel pending downloads when closing readers. (#3816, #2786)
  • Refresh metadata and retry KV snapshot lease operations and read-only gateway requests. (#3661, #3390)
  • Retry bootstrap initialization on retriable errors. (#3758)
  • Fix memory leaks in limited scans, Arrow decompression, and completed Arrow batches. (#3003, #2647, #2728)
  • Fix lazy complex-type access after buffers are released. (#3221)
  • Fix tiering hangs caused by empty batches with the first_row merge engine. (#3242)
  • Correct writer resource cleanup ordering and lookup-client shutdown waiting. (#3102, #3328)

🦀 Rust / Python / C++ Clients

Features and Improvements

  • Bring the Rust client and language bindings into the main Fluss repository, with shared protocol generation, CI, documentation, and release tooling. (89eafb9, 400e132)
  • Ship native Rust, Python, and C++ APIs for log appends/scans, primary key upserts/deletes/lookups, partitioned tables, and column projection. (400e132)
  • Provide Python asynchronous scanning and context managers, with PyArrow batch ingestion and native primary key access. (f433709, 1257cc8, 9408a03, f7fb741)
  • Extend C++ complex-type support with MAP and nested ROW values. (4dc9ecc)
  • Add log filter pushdown, including predicate serialization and V1 batch statistics, to the Rust client. (#3847, #3900, #3971, #4011, #4074)
  • Expose log filter pushdown in Python and C++. (#4127, #4036)
  • Expose table Arrow schemas across Rust, Python, and C++. (#4148)
  • Add C++ bounded RecordBatch log readers and Rust into_stream() support for bounded readers. (#3988, #3664)
  • Add cooperative KV write backpressure support to the Rust client. (#3964)
  • Support comma-separated bootstrap servers. (#3711)
  • Support log-scanner schema evolution and hash-based routing by declared log-table bucket keys. (#3662, #3643)
  • Add Iceberg-format key encoding and preserve auto-increment metadata in schema serialization across language bindings. (#3934, #4171)
  • Avoid copying Arrow payloads during batch decoding. (#4154)
  • Check in generated Rust protobuf code so building the client no longer requires protoc. (938f8ff)
  • Extend Elixir bindings with ARRAY, MAP, ROW, and nullability support. (#3636)

Bug Fixes

  • Recover from stale TabletServer addresses and route non-partitioned metadata requests correctly. (#4393, #4394)
  • Reduce the default Rust connection timeout to 15 seconds. (#4392)
  • Fix stale out-of-order responses resetting the idempotent writer and close drained batches before sending. (#4252, #4258)
  • Stop retrying writes after a cached table has been dropped. (#4141)
  • Fix a lost wakeup in the log fetch buffer and tolerate missing partitions during scanner metadata refresh. (#3712, #3935)
  • Fix Arrow limited scans across schema evolution and append validation for NOT NULL columns. (#3821, #3899)
  • Validate Arrow batch column types and nullability before enqueueing writes. (#4147, 0bebe9e)
  • Preserve highest_field_id, validate nested ROW field names, and reject empty or duplicate primary key columns. (#4184, #4399, #4182)
  • Return errors for unsupported server API versions, invalid metadata JSON, and missing table descriptor schemas. (#4244, #4146, #4175)
  • Preserve C++ column type information during table creation and fix Bazel header declarations. (#3603, #4387)

🔄 Flink Connector / SQL

Features and Improvements

  • Add Flink 2.3 support. (01988d8)
  • Add built-in RoaringBitmap scalar and aggregate SQL functions, automatically registered by FlussCatalog. (#3319, #3398, #3492, #3777)
  • Integrate storage-layer filter pushdown. (#3032)
  • Add projection and filter pushdown for $changelog and $binlog virtual tables. (#3759)
  • Add batch scans for virtual log tables and bounded streaming reads. (#3925, #4022)
  • Add primary key batch reads using SST merge and integrate server-side KV scans. (#3677, a66a6e4)
  • Support batch UPDATE and DELETE with predicates that do not specify the full primary key. (#3863)
  • Add Union Read support to the DataStream API. (#3432)
  • Add watermark pushdown and ALTER TABLE support for modifying watermarks. (#2972, #2718)
  • Support bucket-based lookup shuffle for Flink 2.x. (#3675)
  • Introduce batched split assignment and expose partition details through sys.list_partition_infos. (#3288, f35260c)
  • Add the pendingRecords source metric and reduce default client-side bucket metric cardinality. (#4250, #3828)

Bug Fixes

  • Preserve lake offsets when partition splits are pruned. (#4326)
  • Preserve complete undo-recovery baselines, track consumed recovery progress, and route undo offsets to the correct sink subtask. (#3953, #3910, afc34bb)
  • Close writers immediately when a sink fails over. (#3817)
  • Mark lake and KV snapshots as finished when EOF is reached before the log phase. (#4042)
  • Fix batch reads with empty buckets and empty Lakehouse tables. (#4007, #3208, #2958)
  • Fix mixed split types in primary key batch fallback and preserve log-only source restore mode. (#3296, #3355)
  • Read newly discovered partitions from the earliest available offset. (#3548)
  • Fix partition pruning for non-string partition keys. (#3322)
  • Reject COUNT(column) pushdown for nullable columns. (#3271)
  • Fix partition listing for $changelog and $binlog, and validate database existence when resolving catalog functions. (#3641, #3906)
  • Fix LakeSource copy isolation and gate single-row lookup pushdown on partition liveness. (#4059, #3442)
  • Fix missing RoaringBitmap classes in the shaded connector JAR. (#3981)
  • Respect multiple Flink I/O temporary directories. (#3617)

⚡ Spark Connector

Features and Improvements

  • Add batch Union Read for lake-enabled log and primary key tables. (#2956, #3042, #3142)
  • Add time-range incremental batch reads. (#3883)
  • Add storage-layer filter pushdown for log tables and predicate pushdown into lake tables. (#3116, #3238)
  • Add partition filter pushdown for non-lake log and primary key tables. (#3240)
  • Add LIMIT pushdown and improve partition/non-partition predicate handling. (#3346, #3397)
  • Add scan.maxRecordsPerPartition to split log-table input partitions. (#3260)
  • Support MAP data types and Scala 2.13 builds. (#2740, #3041)
  • Add a numRowsRead metric and clearer scan operator names in the Spark UI. (#3196)
  • Allow lake plugins to override table defaults with catalog configuration and support Spark job-level Paimon DLF credentials. (#3364, 32f66f3)

Bug Fixes

  • Fix empty projections in COUNT(*) and COUNT(1) queries. (#3227)
  • Fix column projection in log and primary key reads. (#3057)
  • Use primary key field positions correctly when comparing rows in merge reads. (#2987)
  • Use KV snapshots when falling back from lake batch reads. (#3317)

🌊 Lakehouse Integrations

Apache Paimon

  • Support primary key lookups against historical partitions retained in Paimon. (#3630)
  • Support historical partition writes, including primary key updates, and tier the changes back to the lake. (#4001, #4120)
  • Add local cache capacity management and dynamic worker/queue configuration for historical lookups. (#3864, #4034)
  • Support clean lake table schemas containing only user columns while retaining compatibility with existing legacy schemas. (#3982)
  • Support custom Paimon storage and table paths. (#3523, #3476)
  • Use Arrow record batches for lake tiering reads and writes. (#3430)
  • Support altering lake table properties and relax compatibility checks for comments and options. (#2799, #3890)
  • Upgrade Paimon to 2.0.0. (#4093)
  • Fix deletion-vector readable snapshot and offset selection. (#3519)
  • Fix false misses during concurrent local lookups and remove the global lookup lock. (#4158, #4050)
  • Preserve Arrow column name case and Paimon split state across class relocation. (#4123, #3825)
  • Keep lakestream.enabled synchronized with lake acceleration state. (#4118)
  • Fix partition expiration in tiering-managed tables and avoid aborting a Paimon commit after failure. (#2862, #3370)
  • Fix primary key comparators and eagerly resolve partitions in the lake writer. (#2980, #3023)

Apache Iceberg

  • Support clean and legacy lake table schemas. (#4019)
  • Add ADD COLUMN schema evolution. (#2593)
  • Reuse an existing compatible lake table and ignore Fluss-only properties during compatibility checks. (#3601, #4220)
  • Validate partition specifications before tiering and fail fast on unsupported multi-key table creation. (#3639, #2499)
  • Fix predicate pushdown for TINYINT and SMALLINT, and read only the column statistics required by filters. (#3565, #3237, #2842)
  • Fix local-time conversion overflow and listener leaks in the lake committer. (#3535, #3542)

Apache Hudi

  • Introduce Hudi Lakehouse integration, including catalog management and bucket assignment. (#3256, #3395, #3316)
  • Add Hudi source split planning, source readers, and predicate pushdown. (#3456, #3499, #3501)
  • Add Hudi lake writers, committers, and tiering compaction. (#3507, #3509, #3518)
  • Support Flink Union Read for Hudi log and primary key tables. (#3528, #3541)
  • Use clean lake table schemas. (#4077)

Lance

  • Add nested ROW support. (#2578)
  • Write array columns as Arrow FixedSizeList to support native vector search. (#2707)

🔁 Lake Tiering / Maintenance

Features and Improvements

  • Add the remove_orphan_files Flink action for remote storage cleanup, with supporting manifest and snapshot listing RPCs. (#3404, #3480)
  • Allow tables created before cluster-level Lakehouse enablement to enable lake tiering later. (#2924, #2973)
  • Allow datalake auto-compaction settings to be changed before lake tiering is enabled. (#3458)
  • Add lake.tiering.io.tmpdirs and improve temporary-directory fallback behavior. (#3569, #2859)
  • Allow tiering jobs to override the remote-first read preference. (#3886)
  • Use an exponential-delay restart strategy for the Flink tiering job. (#3683)

Bug Fixes

  • Fix duplicate writes in Flink tiering. (#3608)
  • Persist tiering progress with empty commits for Hudi and Lance, including buckets that finish without data. (#3941, #3893)
  • Fix lake writer leaks and avoid creating writers for empty splits. (#3720, #3291)
  • Avoid chaining the tiering source with the committer. (#3802)
  • Handle Coordinator disconnects in the tiering source enumerator. (#3107)
  • Reject lake snapshot commits for dropped tables. (#3494)
  • Fix duplicate rows in Union Read for non-string partition columns. (#3500)
  • Avoid altering lake catalogs for tables that never enabled Lakehouse storage. (#3302)

🔐 Security / Configuration

  • Support centralized, dynamically updated multi-user SASL/PLAIN credentials. (#3047)
  • Require ALL permission to create or drop ACLs. (#3680)
  • Add authorization checks to database/table existence and offset lookup APIs. (#3078, #3298)
  • Restrict superuser credential changes to super.users. (#4229)
  • Add configuration providers for resolving secrets, including Helm integration. (#3658, #3685)
  • Redact sensitive configuration values and S3 delegation credentials in logs. (#3486, #3421)
  • Support plugin discovery for client authentication. (#3474)
  • Restrict client.security.sasl.jaas.config to PlainLoginModule. (#3425)
  • Preserve RPC/catalog exception causes and report configuration errors accurately. (#3329, #3281, #2857)

📊 Metrics / Observability

  • Add per-error request metrics and request queue size metrics. (#4213, #3232)
  • Add metric filtering and Prometheus label character filtering. (#4135, #3212)
  • Add an InfluxDB metrics reporter and Basic Auth for Prometheus PushGateway. (#2839, #3552)
  • Add per-table tiering metrics and Coordinator tiering scheduling metrics. (#2454, #2815)
  • Add tiering freshness, pending time, lag, last-successful-tiering timestamp, and bytes-read metrics. (#3076, #3469, #3678, #2933)
  • Add bucket/partition counts, log start offsets, and local physical log size metrics. (#2800, #3229, #3841)
  • Expose KV WAL memory pool metrics. (0b72a7a)
  • Correct pending-record counts before tiering starts and table counts after dropping partitioned tables. (#3059, #3482)
  • Fix Coordinator metric updates during idle periods and snapshot metric access during cleanup. (#3148, 44e1cdd)

☁️ Filesystems / Cloud Storage

  • Add Tencent Cloud COS filesystem support. (#2836)
  • Support configurable AWS S3 credentials providers. (bfaaffb)
  • Align Hadoop dependencies to 3.4.3 and migrate S3 credentials providers to AWS SDK v2. (#3699)
  • Fix client.fs.* configuration forwarding. (#2997)
  • Fix Azure delegation-token authentication configuration. (#2947)
  • Relocate shared dependencies in S3, Google Cloud Storage, and Azure plugins to avoid Spark conflicts; exclude conflicting Jetty dependencies from Azure. (#3960, #3508)

⛵ Helm / Deployment

  • Add configuration-provider integration for secrets. (#3685)
  • Support annotations on StatefulSets and Services. (#3687)
  • Improve chart configuration validation. (#2902)
  • Disable the Log4j shutdown hook in foreground mode. (#3634)

📦 Build / Dependencies / Distribution

  • Upgrade the Rust Arrow stack to version 59. (#3600)
  • Add Trusted Publishing for Rust and Python clients. (#4333)
  • Give Python release candidates distinct versions and use the ASF-approved Rust build action. (#4412, #4411)
  • Fix Quickstart JAR selection for release builds. (#4366)
  • Include LICENSE and NOTICE files in Rust and Python artifacts and verify packaged wheel licenses. (#3803, #4090, #4405)
  • Include licenses and notices for statically linked Rust dependencies and bundled Java dependencies. (97b213a, 9561288)
  • Update release metadata and remove Incubator branding following graduation to a Top-Level Project. (#3750, #3791)

📚 Documentation

  • Add Gateway usage, deployment, and HTTP-to-Lakehouse quickstart guides. (#4162, #4210, #4231)
  • Add a real-time user profile tutorial and RoaringBitmap SQL examples. (#2669, #3823)
  • Document Java multi-table scanners and writers. (#4192)
  • Document non-full-primary-key batch UPDATE and DELETE in Flink. (#4191)
  • Document Coordinator HA, resource protection, and orphan-file cleanup. (#4111, #3701, #4323)
  • Expand retention/TTL guidance, including local log TTL and active segment rolling. (#3944)
  • Document clean lake schemas and the required rolling-upgrade sequence. (#4112)
  • Add Hudi, Apache Polaris/Iceberg REST catalog, and Gravitino integration guides. (#3549, #3515, #3669)
  • Update the multi-language client support matrix and multi-user SASL/PLAIN configuration guidance. (#4385, #4168)

⚠️ Upgrade Notes

  • Lakehouse schemas: Newly created Paimon and Iceberg lake tables use clean schemas without __bucket, __offset, or __timestamp. Existing legacy tables retain their layout. Upgrade lake-reading Flink connectors and their lake plugins first, then the tiering service, and finally the Fluss cluster. Older readers and tiering services cannot handle clean tables safely.
  • ACL management: Principals that create or drop ACLs now need ALL permission on the target resource.
  • S3 credentials: Custom or explicitly configured AWS SDK v1 credentials providers must be migrated to AWS SDK v2. Static access keys and the default credentials provider chain do not require configuration changes.
  • Active segment rolling: Keep log.retention.roll-active-segment.enabled disabled throughout a rolling upgrade from 0.9. Enable it only after all Coordinators and TabletServers have been upgraded.
  • Bucket rescaling: Bucket-count changes apply to future partitions. Existing partitions keep their bucket counts. Upgrade relevant clients and connectors before using this feature; aggregation tables and tables with historical partition access are not supported.

See the complete upgrade notes for configuration changes and compatibility details.

🔗 Full Changelog

v0.9.1-incubating...v1.0.0

Assets 2
Loading