Skip to content

DynamicWriteResultSerializer version not bumped after format change #17314

Description

@wangyum

Apache Iceberg version

1.11.0 (latest release)

Query engine

Flink

Please describe the bug 🐞

DynamicWriteResultSerializer was modified in #14810 to add a specId field to the serialized format, but getVersion() was not bumped from 1. This breaks the SimpleVersionedSerializer versioning contract: if a checkpoint from a job running the pre-#14810 code (V1, no specId) is restored on the current code (still claiming V1 but expecting specId), the readInt() for specId will consume the first four bytes of the WriteResult payload, producing a corrupted WriteResult and silent data loss.

Steps to Reproduce

  1. Start a Flink job using the dynamic Iceberg sink on a version before Flink: Refactor WriteResult aggregation in DynamicIcebergSink #14810 (serializes WriteTarget + WriteResult bytes as V1).
  2. Take a checkpoint.
  3. Upgrade the job to a version with Flink: Refactor WriteResult aggregation in DynamicIcebergSink #14810 (serializes TableKey + specId + WriteResult bytes, but still claims V1).
  4. Restore from the checkpoint.
  5. deserialize(1, ...) runs the new V1 code path, which calls TableKey.deserializeFrom(view) (reads only tableName + branch) then view.readInt() for specId — but the old V1 bytes have schemaId (an int) immediately after branch, so readInt() consumes schemaId as specId, and the remaining bytes (starting from the real specId field) are passed to WriteResultSerializer.deserialize(), producing a corrupt WriteResult.

Expected Behavior

The serializer should use distinct version numbers for the two formats so that old V1 bytes are deserialized using the old V1 layout (WriteTarget + WriteResult) and new V2 bytes are deserialized using the new V2 layout (TableKey + specId + WriteResult).

Actual Behavior

Both formats claim version 1, so the deserializer cannot distinguish them, leading to silent corruption on upgrade.

Willingness to contribute

  • I can contribute a fix for this bug independently
  • I would be willing to contribute a fix for this bug with guidance from the Iceberg community
  • I cannot contribute a fix for this bug at this time

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions