Search before asking
Motivation
Fluss currently lacks an efficient way to load a large initial dataset or
backfill a primary-key table. Writing records through the ordinary online
write path is expensive for bounded batch workloads.
This capability is also a foundation for promoting lake-resident data into
Fluss's real-time serving path. Typical use cases include upgrading a lake
table into a real-time table and promoting a cold lake partition into a
real-time partition, without replaying the complete dataset through the
ordinary online write path.
The target capability should build ordinary KV snapshots outside TabletServers,
install them through a server-managed transaction, and make the complete target
visible atomically.
Production use also requires scoped temporary write authorization. A workflow
that succeeds only when clients have unrestricted storage credentials is not
sufficient for production batch insert or lake-to-real-time promotion.
Solution
Scope
The initial implementation supports:
- Empty primary-key tables and partitions.
- Offline construction of ordinary Fluss KV snapshots.
- Server-managed Begin, Commit, and Abort transactions.
- Target fencing while a bulk load is in progress.
- Atomic publication after every bucket and replica is ready.
- Coordinator failover and assignment changes during transaction processing.
- A public client SDK for building and committing bulk-load files.
- Scoped temporary write authorization for build sessions.
- Flink batch insert as the first engine integration.
Sub Tasks
The tasks are expected to be delivered in the listed order.
Anything else?
Completion criteria
This umbrella issue is complete when:
- An externally built KV snapshot can be validated, adopted, and restored by Fluss.
- The server provides a complete and recoverable Begin/Commit/Abort lifecycle.
- The client SDK exposes the complete bulk-load workflow.
- Build sessions can obtain scoped temporary write authorization without
serializing credentials into build contexts or engine checkpoints.
- Flink batch insert can load an empty primary-key table or partition and expose
the result atomically.
- Failure, abort, Coordinator failover, TabletServer restart, and assignment
change paths have appropriate regression coverage.
- Compatibility behavior and user documentation are complete.
Non-goals
- Streaming bulk load.
- Loading into a non-empty target.
- Non-primary-key tables.
- Atomic commit across multiple independent targets.
- Spark integration.
- A complete general-purpose authentication or authorization system.
Willingness to contribute
Search before asking
Motivation
Fluss currently lacks an efficient way to load a large initial dataset or
backfill a primary-key table. Writing records through the ordinary online
write path is expensive for bounded batch workloads.
This capability is also a foundation for promoting lake-resident data into
Fluss's real-time serving path. Typical use cases include upgrading a lake
table into a real-time table and promoting a cold lake partition into a
real-time partition, without replaying the complete dataset through the
ordinary online write path.
The target capability should build ordinary KV snapshots outside TabletServers,
install them through a server-managed transaction, and make the complete target
visible atomically.
Production use also requires scoped temporary write authorization. A workflow
that succeeds only when clients have unrestricted storage credentials is not
sufficient for production batch insert or lake-to-real-time promotion.
Solution
Scope
The initial implementation supports:
Sub Tasks
The tasks are expected to be delivered in the listed order.
Anything else?
Completion criteria
This umbrella issue is complete when:
serializing credentials into build contexts or engine checkpoints.
the result atomically.
change paths have appropriate regression coverage.
Non-goals
Willingness to contribute