[tiering][iceberg] Validate partition spec before lake tiering

[litiliu]

Search before asking

• I searched in the issues and found nothing similar.

Fluss version

main (development)

Please describe the bug 🐞

Problem

The Iceberg lake tiering writer does not validate that the current Iceberg partition spec is still compatible with the partitioning definition used by Fluss.

An external Iceberg client can evolve the partition spec after lake tiering has been enabled. A newly created or restarted tiering writer may then load the evolved Iceberg spec but continue using partition values calculated according to the Fluss table's original bucket definition.

This problem already applies to the existing explicit bucket partitioning and does not depend on implicit/hidden partition support.

Reproduction example

1. Create a Fluss primary-key table distributed by id into 8 buckets and enable Iceberg lake tiering.
2. Wait for the Iceberg table to be created and verify that its current partition spec contains bucket[8](id).
3. Stop or pause the tiering job so that its lake writer will be recreated later.
4. Using an external Iceberg client, evolve the Iceberg table's current partition spec from bucket[8](id) to bucket[16](id). The Fluss table remains configured with 8 buckets.
5. Restart the tiering job, causing a new Iceberg lake writer to load the current bucket[16](id) spec.
6. Write a record whose bucket[8](id) value differs from its bucket[16](id) value, and wait for it to be tiered.
7. Inspect the committed data file's partition metadata or query the table with a predicate on that id.

Current behavior

The writer creates a PartitionKey using the current Iceberg spec, but directly sets the Fluss table bucket ID into its bucket field. It does not apply the current Iceberg bucket transform to the original id value.

For example, if bucket[8](id) = 3 while bucket[16](id) = 11, the writer records 3 as the value of the bucket[16](id) partition field. Because both values are integers, this mismatch may not cause a type or commit error.

The file can therefore be committed with incorrect partition metadata. Iceberg readers may project the predicate using bucket[16](id) and prune that file, causing the record to be missing from filtered queries even though the data file was committed successfully.

Expected behavior

When creating the Iceberg lake writer, Fluss should verify that the current Iceberg partition spec is semantically equivalent to the partitioning definition expected from the Fluss table. If it is incompatible, writer initialization should fail with a clear error before any incorrectly partitioned file can be committed.

Solution

Add an Iceberg partition-spec compatibility validator to the lake writer.

The validation should compare the expected Fluss-to-Iceberg mapping with the current Iceberg spec, including:

• source columns;
• transform types;
• transform parameters such as the bucket count;
• partition field order where it affects the produced partition row.

Run the validation when the Iceberg writer is initialized. If table metadata can be refreshed while the writer remains active, cache the validated Iceberg spec ID and validate again when the current spec ID changes.

Reject incompatible specs with a non-retriable, descriptive exception that includes both the expected and current partition definitions.

Are you willing to submit a PR?

• I'm willing to submit a PR!