You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
DynamicWriteResultSerializer was modified in #14810 to add a specId field to the serialized format, but getVersion() was not bumped from 1. This breaks the SimpleVersionedSerializer versioning contract: if a checkpoint from a job running the pre-#14810 code (V1, no specId) is restored on the current code (still claiming V1 but expecting specId), the readInt() for specId will consume the first four bytes of the WriteResult payload, producing a corrupted WriteResult and silent data loss.
deserialize(1, ...) runs the new V1 code path, which calls TableKey.deserializeFrom(view) (reads only tableName + branch) then view.readInt() for specId — but the old V1 bytes have schemaId (an int) immediately after branch, so readInt() consumes schemaId as specId, and the remaining bytes (starting from the real specId field) are passed to WriteResultSerializer.deserialize(), producing a corrupt WriteResult.
Expected Behavior
The serializer should use distinct version numbers for the two formats so that old V1 bytes are deserialized using the old V1 layout (WriteTarget + WriteResult) and new V2 bytes are deserialized using the new V2 layout (TableKey + specId + WriteResult).
Actual Behavior
Both formats claim version 1, so the deserializer cannot distinguish them, leading to silent corruption on upgrade.
Willingness to contribute
I can contribute a fix for this bug independently
I would be willing to contribute a fix for this bug with guidance from the Iceberg community
I cannot contribute a fix for this bug at this time
Apache Iceberg version
1.11.0 (latest release)
Query engine
Flink
Please describe the bug 🐞
DynamicWriteResultSerializerwas modified in #14810 to add aspecIdfield to the serialized format, butgetVersion()was not bumped from1. This breaks theSimpleVersionedSerializerversioning contract: if a checkpoint from a job running the pre-#14810 code (V1, nospecId) is restored on the current code (still claiming V1 but expectingspecId), thereadInt()forspecIdwill consume the first four bytes of theWriteResultpayload, producing a corruptedWriteResultand silent data loss.Steps to Reproduce
WriteTarget+WriteResultbytes as V1).TableKey+specId+WriteResultbytes, but still claims V1).deserialize(1, ...)runs the new V1 code path, which callsTableKey.deserializeFrom(view)(reads onlytableName+branch) thenview.readInt()forspecId— but the old V1 bytes haveschemaId(an int) immediately afterbranch, soreadInt()consumesschemaIdasspecId, and the remaining bytes (starting from the realspecIdfield) are passed toWriteResultSerializer.deserialize(), producing a corruptWriteResult.Expected Behavior
The serializer should use distinct version numbers for the two formats so that old V1 bytes are deserialized using the old V1 layout (
WriteTarget+WriteResult) and new V2 bytes are deserialized using the new V2 layout (TableKey+specId+WriteResult).Actual Behavior
Both formats claim version 1, so the deserializer cannot distinguish them, leading to silent corruption on upgrade.
Willingness to contribute