Flink: Support Variant type in DynamicIcebergSink - #17900
Open
fmorillo7694 wants to merge 1 commit into
Open
Conversation
…nd evolution
The dynamic sink's CompareSchemasVisitor and EvolveSchemaVisitor extend
SchemaWithPartnerVisitor but do not override variant(), so any DynamicRecord
whose schema contains a Variant field fails with
UnsupportedOperationException("Unsupported type: variant") during schema
resolution in TableMetadataCache - even when the target table already exists
with an exactly matching schema. DataConverter.get() also lacks a VARIANT
case for the conversion path.
The static IcebergSink writes Variant fields correctly (FlinkSchemaUtil,
FlinkParquetWriters/Readers all support it since 1.11.0), and
VariantAvroDynamicTableRecordGenerator already produces Variant-bearing
DynamicRecords, so the dynamic sink pipeline is currently unable to process
records its own generator emits.
Add variant() overrides to both visitors (same-type: SAME, otherwise
SCHEMA_UPDATE_NEEDED) and an identity VARIANT case to DataConverter, for
the v2.1, v2.2, and v2.3 modules. v1.20 is excluded: Flink 1.20 has no
VariantType, so the code path is unreachable there.
Fixes apache#17615
Co-authored-by: water <672684719@qq.com>
This was referenced Aug 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #17615
Problem
DynamicIcebergSinkfails on anyDynamicRecordwhose schema contains avariantfield:CompareSchemasVisitorandEvolveSchemaVisitorextendSchemaWithPartnerVisitorbut do not overridevariant(), so they inherit the base implementation which throws unconditionally. SinceTableMetadataCache.schema()runsCompareSchemasVisitoron every incoming record's schema, the failure occurs during schema resolution — before any writer is created, and even when the target table already exists with an exactly matching schema.DataConverter.get()also lacks aVARIANTcase on the conversion path.The rest of the stack already supports variant:
FlinkSchemaUtilmapsVariantTypein both directions,FlinkParquetWriters/FlinkParquetReadersread and write it, and the staticIcebergSinkwrites variant fields end-to-end (verified: identical schema + row data succeeds throughIcebergSink.forRowDataagainst the same table where the dynamic sink throws). Notably,VariantAvroDynamicTableRecordGenerator(#16450) already produces variant-bearingDynamicRecords that the pipeline is currently unable to process.Changes
CompareSchemasVisitor: addvariant()override —SAMEwhen the table field is also variant,SCHEMA_UPDATE_NEEDEDotherwise (variant has no widening/conversion semantics).EvolveSchemaVisitor: addvariant()override, consistent withprimitive().DataConverter: add identitycase VARIANT.v2.1,v2.2, andv2.3modules.v1.20is intentionally excluded: Flink 1.20 has noVariantType/LogicalTypeRoot.VARIANT, so the dynamic sink cannot receive variant data there and acase VARIANT:inDataConverterwould not compile against Flink 1.20.Tests
Per module (v2.1/v2.2/v2.3):
TestCompareSchemasVisitor: same variant schema →SAME; variant vs non-variant table field →SCHEMA_UPDATE_NEEDED; variant field missing from table →SCHEMA_UPDATE_NEEDED.TestEvolveSchemaVisitor: adding a top-level variant field via schema evolution; identical variant schema is a no-op.TestRowDataConverter: variant value passes throughDataConverterunchanged.:iceberg-flink:iceberg-flink-{2.1,2.2,2.3}:testfor the three test classes: 79 tests per module, 0 failures. Spotless clean.Relationship to #17631
This supersedes #17631 (thank you @waterWang for the original fix — credited via
Co-authored-byon the commit). That PR targets theflink/v2.0module which no longer exists onmain, and itsv1.20DataConverterchange cannot compile against Flink 1.20; this PR retargets the same fix to the current module layout (v2.1/v2.2/v2.3) and adds test coverage.