[FLINK-40490][table] Add runtime serialization and codegen for the UUID type by raminqaf · Pull Request #29087 · apache/flink

raminqaf · 2026-09-03T11:33:06Z

What is the purpose of the change

Second subtask of FLIP-604, building directly on FLINK-40486 (merged as #29041), which introduced the UUID logical type at the type-system layer only. After that change a UUID value still could not be materialized, serialized, or returned at runtime. This change wires UUID through the table runtime, code generation, and the DataStream type stack so UUID
columns work end to end.

Internally a UUID is stored as its canonical 16-byte big-endian encoding, reusing them byte[] representation and getBinary/writeBinary access already used by BINARY/VARBINARY. The external/default conversion class is java.util.UUID.

Brief change log

Internal representation and serialization: LogicalTypeUtils.toInternalConversionClass returns byte[]; InternalSerializers, BinaryWriter, BinaryArrayWriter, the interpreted RowData/ArrayData field getters, and BinaryArrayData sizing reuse the BINARY/VARBINARY path.
Code generation: CodeGenUtils (type term, hashing, row read/write) and GenerateUtils literal handling emit the byte[] path for UUID; RexLiteralUtil converts a java.util.UUID literal to 16 big-endian bytes.
External conversion: new UuidUuidConverter (byte[] <-> java.util.UUID), registered in DataStructureConverters.
DataStream API: new UuidSerializer and UuidComparator, plus Types.UUID. The comparator uses unsigned big-endian byte order (the type's defined order), not the signed java.util.UUID.compareTo.
Reflective extraction: UUID is auto-extracted for Table/SQL via ClassDataTypeConverter but intentionally not auto-registered for the DataStream API (BasicTypeInfo.TYPES); see below.
Planner plumbing: FlinkRelMdSize (16-byte estimate) and ExpressionReducer (skip constant folding, as for VARIANT).

Verifying this change

This change added tests and can be verified as follows:

UuidSerializerTest / UuidComparatorTest (flink-core): serialization round-trip and unsigned ordering, including normalized-key consistency; the sorted test data spans the signed-long boundary (0x7FFF... vs 0x8000...).
DataStructureConvertersTest: byte[] <-> java.util.UUID round-trip.
ClassDataTypeConverterTest / DataTypeExtractorTest: Table/SQL auto-extraction of a java.util.UUID field to DataTypes.UUID().
UuidITCase (flink-table-planner): end-to-end SELECT, ARRAY[...], multi-row VALUES, UNION/CASE, MAP value, and nested ROW, collecting java.util.UUID back through the full plan/codegen/execution stack.

Design notes / trade-offs for reviewers

DataStream extraction is opt-in. Registering java.util.UUID for automatic reflective extraction in the DataStream API would silently switch the serializer of existing java.util.UUID fields from Kryo to UuidSerializer, breaking savepoint compatibility on upgrade. DataStream users therefore opt in via Types.UUID; automatic extraction is planned for the next major version. Table/SQL does auto-extract, since a plain UUID field there otherwise requires an explicit RAW annotation, so there is no silent change.
Comparator order is unsigned big-endian to match the type's canonical order and the SQL side, deliberately differing from java.util.UUID.compareTo (signed per half).

Related tickets (FLIP-604)

Depends on: FLINK-40486 — Introduce the UUID logical type (merged, [FLINK-40486][table] Introduce the UUID logical type #29041).
Follow-ups, handled in separate subtasks:
- FLINK-40487 — casts UUID <-> STRING / BINARY(16) (note: .print() depends on the
  to-string cast).
- FLINK-40488 — comparison and ordering in SQL (ORDER BY / GROUP BY / joins);
  TypeCheckUtils.isComparable still excludes UUID in this PR.
- FLINK-40489 — UUID_V4 / UUID_V7 functions.
- FLINK-40494 — documentation.
Not addressed here: compiled-plan RexNode literal serde, left as a follow-up as it was for VARIANT.

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): yes — adds
Types.UUID and BasicTypeInfo.UUID_TYPE_INFO
The serializers: yes — new UuidSerializer; UUID internal state uses the byte-array serializer
The runtime per-record code paths (performance sensitive): yes — per-record field getters/
writers and generated field access for UUID
Anything that affects deployment or recovery: JobManager (and its components),
Checkpointing, Kubernetes/Yarn, ZooKeeper: no (new type; no state migration for existing jobs)
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? yes (as part of FLIP-604)
If yes, how is the feature documented? not documented in this PR; user-facing docs are
handled in a separate FLIP-604 subtask (FLINK-40494)

flinkbot · 2026-09-03T11:49:25Z

CI report:

2d875bc Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot run azure re-run the last Azure build

…ID type Wire the UUID logical type through the table runtime and code generation so UUID values can be materialized, serialized, and returned to users. Internally a UUID is stored as its canonical 16-byte big-endian encoding, reusing the byte[] representation and getBinary/writeBinary access of BINARY/VARBINARY; the external conversion class is java.util.UUID. This covers InternalSerializers, the binary row and array writers and getters (including the interpreted RowData and ArrayData field getters), CodeGenUtils/GenerateUtils, RexLiteral conversion, the data structure converter, planner size estimation, and constant reduction handling. For the DataStream API this adds UuidSerializer and UuidComparator plus Types.UUID. The comparator orders values by unsigned big-endian bytes, matching the type's defined order rather than the signed java.util.UUID.compareTo. UUID is intentionally not registered for automatic reflective extraction in the DataStream API so that the serializer of existing java.util.UUID fields does not change; users opt in via Types.UUID. Table and SQL do auto-extract UUID via ClassDataTypeConverter, since a plain field otherwise requires an explicit RAW annotation. Casts (FLINK-40487) and comparison and ordering in SQL (FLINK-40488) are handled in separate subtasks.

snuyanzin · 2026-09-03T14:20:48Z

+        // UUID is intentionally not registered here. Automatic reflective extraction of
+        // java.util.UUID to UUID_TYPE_INFO in the DataStream API stays opt-in via Types.UUID to
+        // avoid silently changing the serializer of existing java.util.UUID fields (currently
+        // handled by Kryo). This is planned to change in the next major Flink version.


who and how will remember it when next major Flink version starts?
Should we better have a jira with label something like Flink-3.0?

Created: issues.apache.org/jira/projects/FLINK/issues/FLINK-40546

snuyanzin · 2026-09-03T14:25:58Z

        addDefaultDataType(
                java.time.Period.class, DataTypes.INTERVAL(DataTypes.YEAR(4), DataTypes.MONTH()));
        addDefaultDataType(ColumnList.class, DataTypes.DESCRIPTOR());
+        addDefaultDataType(java.util.UUID.class, DataTypes.UUID());


any specific reason for absolute name here?

Not sure what the convention here Is. most of the java classes here have the absolute name

twalthr

Please also update:

JavaDoc of RowData
Every logical type should also support it's internal representation. So byte[] should be a supported conversion class in UuidType.

twalthr · 2026-09-03T14:19:14Z

+        // UUID is intentionally not registered here. Automatic reflective extraction of
+        // java.util.UUID to UUID_TYPE_INFO in the DataStream API stays opt-in via Types.UUID to
+        // avoid silently changing the serializer of existing java.util.UUID fields (currently
+        // handled by Kryo). This is planned to change in the next major Flink version.


Please open a ticket for this with Fixed Version 3.0

Created: https://issues.apache.org/jira/projects/FLINK/issues/FLINK-40546

twalthr · 2026-09-03T14:24:15Z

    /** Returns type information for {@link java.time.Instant}. Supports a null value. */
    public static final TypeInformation<Instant> INSTANT = BasicTypeInfo.INSTANT_TYPE_INFO;

+    /** Returns type information for {@link java.util.UUID}. Supports a null value. */


Suggested change

/** Returns type information for {@link java.util.UUID}. Supports a null value. */

/** Returns type information for {@link java.util.UUID}. */

the serializer code says:

so there is no reserved value for {@code null}; nullability is handled by wrapping serializers

twalthr · 2026-09-03T14:31:57Z

+import static org.junit.jupiter.params.provider.Arguments.of;
+
+/** Runtime tests for the {@code UUID} type through the full plan/codegen/execution stack. */
+class UuidITCase {


Never just implement plain ITCases, usually they should extend from a test base. Otherwise a full Flink local cluster is being used instead of performant testing clusters. I would suggest to use SemanticTestBase here or BuiltInFunctionTestBase?

Removed and converted into semantic tests

twalthr · 2026-09-03T14:34:24Z

        putConverter(LogicalTypeRoot.STRUCTURED_TYPE, RowData.class, identity());
        putConverter(LogicalTypeRoot.RAW, byte[].class, RawByteArrayConverter::create);
        putConverter(LogicalTypeRoot.RAW, RawValueData.class, identity());
+        putConverter(LogicalTypeRoot.UUID, UUID.class, constructor(UuidUuidConverter::new));


Suggested change

putConverter(LogicalTypeRoot.UUID, UUID.class, constructor(UuidUuidConverter::new));

putConverter(LogicalTypeRoot.UUID, UUID.class, constructor(UuidUuidConverter::new));

putConverter(LogicalTypeRoot.UUID, byte[].class, identity());

Added byte[] to the inputOutputConvertion

- Document the UUID internal representation (byte[]) in the RowData javadoc. - Support byte[] as a conversion class for UUID (its internal representation) and register the identity data structure converter. - Remove the inaccurate "supports a null value" note from Types.UUID, since the serializer has no reserved null value. - Annotate UuidSerializerSnapshot with @internal to satisfy the API annotation architecture rule. - Replace the plain UUID ITCase with a SemanticTestBase based UuidSemanticTest and move the test programs into UuidTestPrograms.

twalthr

LGTM, thank @raminqaf

raminqaf force-pushed the FLINK-40490-uuid-codegen branch from bbacb85 to 2fc07d2 Compare September 3, 2026 13:15

snuyanzin reviewed Sep 3, 2026

View reviewed changes

twalthr reviewed Sep 3, 2026

View reviewed changes

twalthr approved these changes Sep 3, 2026

View reviewed changes

	/** Returns type information for {@link java.util.UUID}. Supports a null value. */
	/** Returns type information for {@link java.util.UUID}. */

	putConverter(LogicalTypeRoot.UUID, UUID.class, constructor(UuidUuidConverter::new));
	putConverter(LogicalTypeRoot.UUID, UUID.class, constructor(UuidUuidConverter::new));
	putConverter(LogicalTypeRoot.UUID, byte[].class, identity());

Uh oh!

Conversation

raminqaf commented Sep 3, 2026

What is the purpose of the change

Brief change log

Verifying this change

Design notes / trade-offs for reviewers

Related tickets (FLIP-604)

Does this pull request potentially affect one of the following parts:

Documentation

Uh oh!

flinkbot commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI report:

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

twalthr left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

twalthr left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

flinkbot commented Sep 3, 2026 •

edited

Loading

twalthr left a comment •

edited

Loading