Skip to content

Flink: Support Variant type in DynamicIcebergSink - #17900

Open
fmorillo7694 wants to merge 1 commit into
apache:mainfrom
fmorillo7694:flink-dynamic-sink-variant
Open

Flink: Support Variant type in DynamicIcebergSink#17900
fmorillo7694 wants to merge 1 commit into
apache:mainfrom
fmorillo7694:flink-dynamic-sink-variant

Conversation

@fmorillo7694

Copy link
Copy Markdown

Fixes #17615

Problem

DynamicIcebergSink fails on any DynamicRecord whose schema contains a variant field:

java.lang.UnsupportedOperationException: Unsupported type: variant
	at org.apache.iceberg.schema.SchemaWithPartnerVisitor.variant(SchemaWithPartnerVisitor.java:167)
	at org.apache.iceberg.schema.SchemaWithPartnerVisitor.visit(SchemaWithPartnerVisitor.java:111)
	at org.apache.iceberg.schema.SchemaWithPartnerVisitor.visit(SchemaWithPartnerVisitor.java:62)
	at org.apache.iceberg.schema.SchemaWithPartnerVisitor.visit(SchemaWithPartnerVisitor.java:45)
	at org.apache.iceberg.flink.sink.dynamic.CompareSchemasVisitor.visit(CompareSchemasVisitor.java:60)
	at org.apache.iceberg.flink.sink.dynamic.TableMetadataCache.schema(TableMetadataCache.java:161)
	at org.apache.iceberg.flink.sink.dynamic.TableMetadataCache.schema(TableMetadataCache.java:112)
	at org.apache.iceberg.flink.sink.dynamic.DynamicRecordProcessor.collect(DynamicRecordProcessor.java:138)

CompareSchemasVisitor and EvolveSchemaVisitor extend SchemaWithPartnerVisitor but do not override variant(), so they inherit the base implementation which throws unconditionally. Since TableMetadataCache.schema() runs CompareSchemasVisitor on every incoming record's schema, the failure occurs during schema resolution — before any writer is created, and even when the target table already exists with an exactly matching schema. DataConverter.get() also lacks a VARIANT case on the conversion path.

The rest of the stack already supports variant: FlinkSchemaUtil maps VariantType in both directions, FlinkParquetWriters/FlinkParquetReaders read and write it, and the static IcebergSink writes variant fields end-to-end (verified: identical schema + row data succeeds through IcebergSink.forRowData against the same table where the dynamic sink throws). Notably, VariantAvroDynamicTableRecordGenerator (#16450) already produces variant-bearing DynamicRecords that the pipeline is currently unable to process.

Changes

  • CompareSchemasVisitor: add variant() override — SAME when the table field is also variant, SCHEMA_UPDATE_NEEDED otherwise (variant has no widening/conversion semantics).
  • EvolveSchemaVisitor: add variant() override, consistent with primitive().
  • DataConverter: add identity case VARIANT.
  • Applied to the v2.1, v2.2, and v2.3 modules. v1.20 is intentionally excluded: Flink 1.20 has no VariantType / LogicalTypeRoot.VARIANT, so the dynamic sink cannot receive variant data there and a case VARIANT: in DataConverter would not compile against Flink 1.20.

Tests

Per module (v2.1/v2.2/v2.3):

  • TestCompareSchemasVisitor: same variant schema → SAME; variant vs non-variant table field → SCHEMA_UPDATE_NEEDED; variant field missing from table → SCHEMA_UPDATE_NEEDED.
  • TestEvolveSchemaVisitor: adding a top-level variant field via schema evolution; identical variant schema is a no-op.
  • TestRowDataConverter: variant value passes through DataConverter unchanged.

:iceberg-flink:iceberg-flink-{2.1,2.2,2.3}:test for the three test classes: 79 tests per module, 0 failures. Spotless clean.

Relationship to #17631

This supersedes #17631 (thank you @waterWang for the original fix — credited via Co-authored-by on the commit). That PR targets the flink/v2.0 module which no longer exists on main, and its v1.20 DataConverter change cannot compile against Flink 1.20; this PR retargets the same fix to the current module layout (v2.1/v2.2/v2.3) and adds test coverage.

…nd evolution

The dynamic sink's CompareSchemasVisitor and EvolveSchemaVisitor extend
SchemaWithPartnerVisitor but do not override variant(), so any DynamicRecord
whose schema contains a Variant field fails with
UnsupportedOperationException("Unsupported type: variant") during schema
resolution in TableMetadataCache - even when the target table already exists
with an exactly matching schema. DataConverter.get() also lacks a VARIANT
case for the conversion path.

The static IcebergSink writes Variant fields correctly (FlinkSchemaUtil,
FlinkParquetWriters/Readers all support it since 1.11.0), and
VariantAvroDynamicTableRecordGenerator already produces Variant-bearing
DynamicRecords, so the dynamic sink pipeline is currently unable to process
records its own generator emits.

Add variant() overrides to both visitors (same-type: SAME, otherwise
SCHEMA_UPDATE_NEEDED) and an identity VARIANT case to DataConverter, for
the v2.1, v2.2, and v2.3 modules. v1.20 is excluded: Flink 1.20 has no
VariantType, so the code path is unreachable there.

Fixes apache#17615

Co-authored-by: water <672684719@qq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flink: DynamicIcebergSink throws UnsupportedOperationException for VARIANT columns

1 participant