Where a tiering round spends its time: measurements of the Iceberg and Paimon lake writers (and a GPU writer experiment)

[mikepapadim]

I measured what a tiering round spends its time on for an append-only ARROW log table, on Fluss main (1.1-SNAPSHOT). I'm sharing the numbers because they point to the lake writer rather than fetch or commit. They come from one machine, and the limits are listed at the end. Proposals that follow from them: #5222 / #5223 (a lake declares record batch support) and #5226 (a pluggable Arrow file writer for Iceberg tiering).

Setup

• i9-13900K (32 threads), local NVMe, JDK 11, idle machine; 3 runs each, median shown (spread < 5%).
• Data: TPC-H lineitem-like, 16 columns (3 BIGINT, 1 INT, 4 DOUBLE, 3 DATE, 5 STRING), 138 Arrow bytes per row. Sizes below are Arrow bytes per round (8 MB = 49K rows, 512 MB = 3.9M rows). Output: ZSTD Parquet (the default for both lakes).
• Round profile: TieringSplitReader driven as the Flink tiering source drives it, against an in-process cluster (one tablet server, one bucket). The rows are appended first (not timed), then one log split covers them. The lake writer is wrapped with timers, so the round splits into writer (write + complete) and other (fetch, Arrow decode, row conversion). The commit is timed separately.
• Writer-only: the same rows fed straight to IcebergLakeWriter / PaimonLakeWriter, so that encode cost is measured alone.

A tiering round

Arrow MB	Iceberg: round (writer / other / commit)	writer share	Paimon batch path: round	writer share	Paimon row path: round	writer share
8	162 ms (116 / 46 / 59)	52%	84 ms (54 / 30 / 4)	61%	87 ms (48 / 39 / 4)	53%
32	593 ms (459 / 134 / 58)	71%	257 ms (202 / 55 / 3)	77%	297 ms (176 / 121 / 4)	59%
128	2234 ms (1805 / 434 / 59)	79%	950 ms (795 / 155 / 3)	83%	1137 ms (699 / 437 / 3)	61%
512	8876 ms (7192 / 1683 / 57)	81%	3657 ms (3156 / 501 / 3)	86%	4420 ms (2746 / 1624 / 3)	63%

1. The lake writer is 52-86% of a round here. Fetch and commit are small on a local setup.
2. Iceberg tiering is ~2.4x slower than Paimon on the same data (8.9 s vs 3.7 s at 512 MB). Part of the gap is the row conversion that Iceberg can't skip today: other is 1.7 s, versus 0.5 s for Paimon's batch path. [lake] Let the lake tiering factory declare record batch write support #5223 plus [lake/iceberg] Implement SupportsRecordBatchWrite for IcebergLakeWriter #4048 would address that.
3. Paimon's batch path speeds up the round (−17% at 512 MB) by skipping row decode, not by encoding faster. Writer-only, Paimon encodes 512 MB in 3.08 s from rows and 3.16 s from Arrow batches. So a columnar path removes the conversion, but parquet-java's encode and compression stay: about 165 MB/s per writer for Paimon and 70 MB/s for Iceberg.

Writer-only: CPU vs a GPU writer

As an experiment, I wrote the same rows (with Fluss's __bucket / __offset / __timestamp columns) as one ZSTD Parquet file with libcudf on an RTX 4090. I called it from Java through TornadoVM, and the timing includes the copy out of Arrow-layout buffers and the transfer to the device. I checked the output with Spark: row count, sums and string lengths all match.

Arrow MB	Iceberg writer	Paimon writer (batch)	GPU (cuDF)
8	124 ms	53 ms	33 ms
32	477 ms	202 ms	60 ms
128	1836 ms	802 ms	91 ms
512	7308 ms (10 CPU-s)	3160 ms (3.4 CPU-s)	224 ms (0.22 CPU-s)

Small rounds gain little (fixed costs dominate at 8 MB). From ~32 MB up, encode time and CPU drop by an order of magnitude. Tiering runs one writer per bucket in parallel, though, so on a 32-thread host the CPU writers together outrun one GPU. What the GPU really offers is CPU seconds freed per GB tiered, not a faster round on an idle machine.

None of this needs a GPU dependency in Fluss. #5226 proposes a device-neutral extension point so that alternative Parquet writers (an Arrow-native CPU writer once arrow-java#904 lands, or a GPU writer) can live outside the tree.

Limits

• One machine, local disk, loopback. There's no object store or remote fetch, and only one bucket and one table. With S3 and remote reads, other and commit grow and the writer's share falls.
• [tiering] Tiering Job Performance: Read-Write Pipeline Optimization #2915 reports low CPU utilisation in tiering with many buckets. That is about parallelism and pipelining, which this setup doesn't reproduce. The two findings are complementary.
• One schema. Wider string-heavy tables or larger decimals will shift the numbers.

I'm happy to share the harness, for example as a fluss-jmh benchmark if that would be useful, and to rerun with a different schema or on object storage. I'd welcome pointers to real tiering workloads to measure.

[mikepapadim]

Follow-up: the #4048 author pointed to apache/parquet-java#3734, an ArrowParquetWriter that writes Arrow VectorSchemaRoots straight to Parquet pages. It is PLAIN only, with no dictionary encoding yet. I ran it on the same data: 19 columns, 16384-row batches, ZSTD. PR head 1856d5b8 was built against parquet 1.18.2, and its own tests pass.

Arrow MB	Iceberg writer (rows)	Paimon writer	#3734 ArrowParquetWriter	GPU (cuDF)
8	124 ms	53 ms	42 ms	33 ms
128	1836 ms	802 ms	675 ms	91 ms
512	7308 ms (68 MB out)	3160 ms (74 MB)	2571 ms (108 MB)	224 ms (71 MB)

• At 512 MB, [server] prevent remote log cache size overflow #3734 is 2.8x faster than the Iceberg row path and about 1.2x faster than Paimon. Its files are about 50% larger, though, because there is no dictionary encoding. Uncompressed, it takes 1.38 s, so about half its time is ZSTD over the larger PLAIN pages.
• That makes [server] prevent remote log cache size overflow #3734 the natural CPU implementation for the extension point proposed in [lake/iceberg] Pluggable Arrow file writer for tiering append-only tables #5226. Dictionary encoding is what would bring its output size in line.