From e4ca3b329fe58d8ee3272af6c344cad766e901dc Mon Sep 17 00:00:00 2001 From: Ramin Gharib Date: Thu, 21 May 2026 17:57:55 +0200 Subject: [PATCH 1/8] [FLINK-39636][table] Add produces_full_deletes parameter in TO_CHANGELOG Adds an optional `produces_full_deletes` boolean parameter to the TO_CHANGELOG built-in PTF. When the caller asserts the input emits full DELETE rows, the function passes them through unchanged and the planner can skip ChangelogNormalize via the REQUIRE_FULL_DELETE conditional trait. Otherwise the function emits partial DELETE rows that preserve identifying columns (partition keys via the framework, upsert keys via the function) and null the rest. Rejects produces_full_deletes=true when the input changelog never emits DELETE rows (insert-only mode or op_mapping strips DELETE). --- .../docs/sql/reference/queries/changelog.md | 58 ++++- flink-python/pyflink/table/table.py | 18 +- .../flink/table/api/PartitionedTable.java | 10 +- .../org/apache/flink/table/api/Table.java | 10 +- .../functions/BuiltInFunctionDefinitions.java | 40 ++-- .../types/inference/BuiltInCondition.java | 1 + .../table/types/inference/TraitCondition.java | 11 + .../strategies/ToChangelogTypeStrategy.java | 16 +- .../exec/stream/ToChangelogSemanticTests.java | 10 +- .../exec/stream/ToChangelogTestPrograms.java | 216 +++++++++++++++++- .../plan/stream/sql/ToChangelogTest.java | 38 +++ .../plan/stream/sql/ToChangelogTest.xml | 66 +++++- .../plan/to-changelog-retract-restore.json | 7 +- .../functions/ptf/ToChangelogFunction.java | 72 +++++- 14 files changed, 527 insertions(+), 46 deletions(-) diff --git a/docs/content/docs/sql/reference/queries/changelog.md b/docs/content/docs/sql/reference/queries/changelog.md index 78f61e1ce107e..7a1598ee850fb 100644 --- a/docs/content/docs/sql/reference/queries/changelog.md +++ b/docs/content/docs/sql/reference/queries/changelog.md @@ -296,17 +296,19 @@ This is useful when you need to materialize changelog events into a downstream s SELECT * FROM TO_CHANGELOG( input => TABLE source_table [PARTITION BY key_col], [op => DESCRIPTOR(op_column_name),] - [op_mapping => MAP['INSERT', 'I', 'DELETE', 'D', ...]] + [op_mapping => MAP['INSERT', 'I', 'DELETE', 'D', ...],] + [produces_full_deletes => BOOLEAN] ) ``` ### Parameters -| Parameter | Required | Description | -|:-------------|:---------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `input` | Yes | The input table. With `PARTITION BY`, rows with the same key are co-located and run in the same operator instance. Without `PARTITION BY`, each row is processed independently. Accepts insert-only, retract, and upsert tables. For upsert tables, the provided `PARTITION BY` key should match or be a subset of the upsert key of the subquery. | -| `op` | No | A `DESCRIPTOR` with a single column name for the operation code column. Defaults to `op`. | -| `op_mapping` | No | A `MAP` mapping change operation names to custom output codes. Keys can contain comma-separated names to map multiple operations to the same code (e.g., `'INSERT, UPDATE_AFTER'`). When provided, only mapped operations are forwarded - unmapped events are dropped. Each change operation may appear at most once across all entries. | +| Parameter | Required | Description | +|:------------------------|:---------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `input` | Yes | The input table. With `PARTITION BY`, rows with the same key are co-located and run in the same operator instance. Without `PARTITION BY`, each row is processed independently. Accepts insert-only, retract, and upsert tables. For upsert tables, the provided `PARTITION BY` key should match or be a subset of the upsert key of the subquery. | +| `op` | No | A `DESCRIPTOR` with a single column name for the operation code column. Defaults to `op`. | +| `op_mapping` | No | A `MAP` mapping change operation names to custom output codes. Keys can contain comma-separated names to map multiple operations to the same code (e.g., `'INSERT, UPDATE_AFTER'`). When provided, only mapped operations are forwarded - unmapped events are dropped. Each change operation may appear at most once across all entries. | +| `produces_full_deletes` | No | A `BOOLEAN` literal that controls how DELETE rows are emitted. When `true`, the function requires fully-populated DELETE rows from the input. The planner inserts a `ChangelogNormalize` operator for upsert sources that emit key-only deletes, so downstream sees the full pre-image on DELETE. When `false` (default), no full-delete requirement is enforced. Partial DELETE rows from the input pass through unchanged. With `PARTITION BY` (set semantics), the function additionally nulls non-partition-key columns on DELETE rows even when the input row is fully populated, so the output always carries only the key on DELETE. | #### Default op_mapping @@ -397,6 +399,42 @@ SELECT * FROM TO_CHANGELOG( -- UPDATE_BEFORE is dropped (not in the mapping) ``` +#### Delete handling + +The `produces_full_deletes` argument controls how DELETE rows are emitted and what the planner requires from the input. The behavior depends on whether `PARTITION BY` is used (set semantics) or not (row semantics). + +**With `produces_full_deletes => true`.** The planner requires the input to produce DELETE rows with all columns populated. For upsert sources, a `ChangelogNormalize` operator is inserted to materialize the full pre-image from state. The function then emits fully-populated DELETE rows downstream. + +```sql +-- Upsert source: -D[id:5] (key-only). +-- ChangelogNormalize materializes the full pre-image from state. +-- Output: +I[op:'DELETE', id:5, name:'Alice'] +SELECT * FROM TO_CHANGELOG( + input => TABLE upsert_source, + produces_full_deletes => true +) +``` + +**With `produces_full_deletes => false` (default).** The planner does not require fully-populated DELETE rows on the input. For upsert sources that emit key-only deletes (e.g. Kafka compacted topics), this avoids the stateful `ChangelogNormalize` operator that would otherwise materialize the full pre-image of each deleted row. + +In **row semantics** (no `PARTITION BY`) the function passes the input row through unchanged. If the source emits partial DELETE rows they remain partial downstream; if it emits full DELETE rows they remain full. + +```sql +-- Source emits -D[id:5] (key-only). +-- Output: +I[op:'DELETE', id:5, name:null] +SELECT * FROM TO_CHANGELOG(input => TABLE upsert_source) +``` + +In **set semantics** (`PARTITION BY`) the function additionally nulls every non-partition-key column on DELETE rows. This forces the output to carry only the partition key on DELETE even when the input row was fully populated, which matches the shape expected by upsert sinks and Kafka compacted topics. + +```sql +-- Source emits -D[id:5, name:'Alice'] (full pre-image, e.g. from a retract source). +-- Output: +I[id:5, op:'DELETE', name:null] +SELECT * FROM TO_CHANGELOG(input => TABLE retract_source PARTITION BY id) +``` + +There is no way to derive a partial DELETE in row semantics when the input emits a full pre-image, since the function has no key column to preserve. Use `PARTITION BY` for that case. + #### Partitioning by a key ```sql @@ -434,6 +472,14 @@ Table result = myTable.toChangelog( map("INSERT, UPDATE_AFTER", "false", "DELETE", "true").asArgument("op_mapping") ); +// Require fully-populated DELETE rows from the input (inserts a ChangelogNormalize for +// upsert sources). When false (default), no full-delete requirement is enforced; in row +// semantics the input passes through unchanged, in set semantics non-partition-key columns +// are nulled on DELETE. +Table result = myTable.toChangelog( + lit(true).asArgument("produces_full_deletes") +); + // Set semantics: co-locate rows with the same key in the same parallel operator instance. // Equivalent to PARTITION BY in SQL. The partition keys are prepended to the output columns. Table result = myTable.partitionBy($("id")).toChangelog(); diff --git a/flink-python/pyflink/table/table.py b/flink-python/pyflink/table/table.py index 583167fa47838..bb06642db9e12 100644 --- a/flink-python/pyflink/table/table.py +++ b/flink-python/pyflink/table/table.py @@ -1196,10 +1196,18 @@ def to_changelog(self, *arguments: Expression) -> 'Table': INSERT-only row with a string ``op`` column indicating the original operation (INSERT, UPDATE_BEFORE, UPDATE_AFTER, DELETE). + The optional ``produces_full_deletes`` boolean controls how DELETE rows are + emitted. When ``True``, the planner inserts a ``ChangelogNormalize`` operator + for upsert sources that emit key-only deletes so the function emits fully + populated DELETE rows downstream. When ``False`` (default), no full-delete + requirement is enforced. In row semantics the input is passed through unchanged, + and in set semantics (``PARTITION BY``) non-partition-key columns are nulled on + DELETE rows. + Example: :: - >>> from pyflink.table.expressions import descriptor, map_ + >>> from pyflink.table.expressions import descriptor, map_, lit, col >>> # Default: adds 'op' column with standard change operation names >>> result = table.to_changelog() >>> # Custom op column name and mapping @@ -1213,8 +1221,14 @@ def to_changelog(self, *arguments: Expression) -> 'Table': ... map_("INSERT, UPDATE_AFTER", "false", ... "DELETE", "true").as_argument("op_mapping") ... ) + >>> # Require fully populated DELETE rows from the input. Inserts a + >>> # ChangelogNormalize for upsert sources. + >>> result = table.to_changelog( + ... lit(True).as_argument("produces_full_deletes") + ... ) - :param arguments: Optional named arguments for ``op`` and ``op_mapping``. + :param arguments: Optional named arguments for ``op``, ``op_mapping``, and + ``produces_full_deletes``. :return: An append-only :class:`~pyflink.table.Table` with an ``op`` column prepended to the input columns. """ diff --git a/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/PartitionedTable.java b/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/PartitionedTable.java index 51086f1edfe22..074bdfdbaeff3 100644 --- a/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/PartitionedTable.java +++ b/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/PartitionedTable.java @@ -204,9 +204,17 @@ public interface PartitionedTable { * descriptor("deleted").asArgument("op"), * map("INSERT, UPDATE_AFTER", "false", "DELETE", "true").asArgument("op_mapping") * ); + * + * // Require fully-populated DELETE rows from the input (inserts a ChangelogNormalize for + * // upsert sources). When false (default), DELETE rows on upsert inputs may omit non-key + * // columns, which avoids the stateful normalization operator upstream. + * Table result = table + * .partitionBy($("id")) + * .toChangelog(lit(true).asArgument("produces_full_deletes")); * } * - * @param arguments optional named arguments for {@code op} and {@code op_mapping} + * @param arguments optional named arguments for {@code op}, {@code op_mapping}, and {@code + * produces_full_deletes} * @return an append-only {@link Table} with output schema {@code [partition_keys, op, * non_partition_input_columns]} * @see Table#toChangelog(Expression...) diff --git a/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/Table.java b/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/Table.java index 0ffb90c5fcc02..de6ce9d3e5ecf 100644 --- a/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/Table.java +++ b/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/Table.java @@ -1459,9 +1459,17 @@ default TableResult executeInsert( * descriptor("deleted").asArgument("op"), * map("INSERT, UPDATE_AFTER", "false", "DELETE", "true").asArgument("op_mapping") * ); + * + * // Require fully-populated DELETE rows from the input (inserts a ChangelogNormalize for + * // upsert sources). When false (default), no full-delete requirement is enforced and partial + * // DELETE rows from the input pass through unchanged. + * Table result = table.toChangelog( + * lit(true).asArgument("produces_full_deletes") + * ); * } * - * @param arguments optional named arguments for {@code op} and {@code op_mapping} + * @param arguments optional named arguments for {@code op}, {@code op_mapping}, and {@code + * produces_full_deletes} * @return an append-only {@link Table} with an {@code op} column prepended to the input columns */ Table toChangelog(Expression... arguments); diff --git a/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/BuiltInFunctionDefinitions.java b/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/BuiltInFunctionDefinitions.java index e3d30a5c8da6b..7083b8c355b85 100644 --- a/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/BuiltInFunctionDefinitions.java +++ b/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/BuiltInFunctionDefinitions.java @@ -840,25 +840,35 @@ ANY, and(logical(LogicalTypeRoot.BOOLEAN), LITERAL) "UPDATE_BEFORE")))) .withConditionalTrait( StaticArgumentTrait.REQUIRE_FULL_DELETE, - TraitCondition.or( - // op_mapping omitted: default mapping includes - // DELETE. - TraitCondition.not( - TraitCondition.argIsPresent( - "op_mapping")), - TraitCondition.argMatches( - "op_mapping", - Map.class, - mapping -> - opMappingContainsKey( - (Map) - mapping, - "DELETE")))), + // Require full deletes only when the user explicitly + // asks for them via produces_full_deletes=TRUE *and* + // the active op_mapping includes DELETE. Otherwise the + // planner can skip ChangelogNormalize for upsert + // sources that emit key-only deletes. + TraitCondition.and( + TraitCondition.argIsEqualTo( + "produces_full_deletes", Boolean.TRUE), + TraitCondition.or( + TraitCondition.not( + TraitCondition.argIsPresent( + "op_mapping")), + TraitCondition.argMatches( + "op_mapping", + Map.class, + mapping -> + opMappingContainsKey( + (Map< + String, + String>) + mapping, + "DELETE"))))), StaticArgument.scalar("op", DataTypes.DESCRIPTOR(), true), StaticArgument.scalar( "op_mapping", DataTypes.MAP(DataTypes.STRING(), DataTypes.STRING()), - true)) + true), + StaticArgument.scalar( + "produces_full_deletes", DataTypes.BOOLEAN(), true)) .inputTypeStrategy(TO_CHANGELOG_INPUT_TYPE_STRATEGY) .outputTypeStrategy(TO_CHANGELOG_OUTPUT_TYPE_STRATEGY) .runtimeClass( diff --git a/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/BuiltInCondition.java b/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/BuiltInCondition.java index ba87cc0559dd6..4c0149f6533b1 100644 --- a/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/BuiltInCondition.java +++ b/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/BuiltInCondition.java @@ -40,6 +40,7 @@ enum Kind { ARG_IS_PRESENT, NOT, OR, + AND, ARG_MATCHES } diff --git a/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/TraitCondition.java b/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/TraitCondition.java index 14734db11f4b6..8b86b37cc9fb8 100644 --- a/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/TraitCondition.java +++ b/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/TraitCondition.java @@ -84,6 +84,17 @@ static TraitCondition or(final TraitCondition left, final TraitCondition right) ctx -> left.test(ctx) || right.test(ctx)); } + /** + * True when both the {@code left} and the {@code right} {@link TraitCondition} evaluate to + * true. + */ + static TraitCondition and(final TraitCondition left, final TraitCondition right) { + return new BuiltInCondition( + BuiltInCondition.Kind.AND, + List.of(left, right), + ctx -> left.test(ctx) && right.test(ctx)); + } + /** True when the named scalar argument was provided by the caller. */ static TraitCondition argIsPresent(final String argName) { return new BuiltInCondition( diff --git a/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/ToChangelogTypeStrategy.java b/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/ToChangelogTypeStrategy.java index d977deab4a93b..8ff8101dc5e5e 100644 --- a/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/ToChangelogTypeStrategy.java +++ b/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/ToChangelogTypeStrategy.java @@ -59,7 +59,7 @@ public final class ToChangelogTypeStrategy { new InputTypeStrategy() { @Override public ArgumentCount getArgumentCount() { - return ConstantArgumentCount.between(1, 3); + return ConstantArgumentCount.between(1, 4); } @Override @@ -77,7 +77,12 @@ public List getExpectedSignatures(final FunctionDefinition definition Signature.of( Argument.of("input", "TABLE"), Argument.of("op", "DESCRIPTOR"), - Argument.of("op_mapping", "MAP"))); + Argument.of("op_mapping", "MAP")), + Signature.of( + Argument.of("input", "TABLE"), + Argument.of("op", "DESCRIPTOR"), + Argument.of("op_mapping", "MAP"), + Argument.of("produces_full_deletes", "BOOLEAN"))); } }; @@ -144,6 +149,13 @@ private static Optional> validateInputs( } } + final boolean hasProducesFullDeletesArg = !callContext.isArgumentNull(3); + if (hasProducesFullDeletesArg && !callContext.isArgumentLiteral(3)) { + return callContext.fail( + throwOnFailure, + "The 'produces_full_deletes' argument must be a constant BOOLEAN literal."); + } + return Optional.of(callContext.getArgumentDataTypes()); } diff --git a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ToChangelogSemanticTests.java b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ToChangelogSemanticTests.java index bfffa8bd67d04..762e08b353895 100644 --- a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ToChangelogSemanticTests.java +++ b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ToChangelogSemanticTests.java @@ -55,6 +55,14 @@ public List programs() { ToChangelogTestPrograms.INVALID_DESCRIPTOR, ToChangelogTestPrograms.INVALID_OP_MAPPING, ToChangelogTestPrograms.OP_MAPPING_REFERENCES_UNSUPPORTED_KIND, - ToChangelogTestPrograms.DUPLICATE_ROW_KIND); + ToChangelogTestPrograms.DUPLICATE_ROW_KIND, + ToChangelogTestPrograms.PRODUCES_FULL_DELETES_ON_APPEND_ONLY_INPUT, + ToChangelogTestPrograms.PRODUCES_FULL_DELETES_WITHOUT_DELETE_IN_OP_MAPPING, + ToChangelogTestPrograms.ROW_SEM_PARTIAL_DELETES, + ToChangelogTestPrograms.ROW_SEM_FORCE_FULL_DELETES, + ToChangelogTestPrograms.SET_SEM_PARTIAL_DELETES, + ToChangelogTestPrograms.SET_SEM_FULL_DELETES, + ToChangelogTestPrograms.SET_SEM_FORCE_FULL_DELETES, + ToChangelogTestPrograms.SET_SEM_FORCE_PARTIAL_DELETES); } } diff --git a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ToChangelogTestPrograms.java b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ToChangelogTestPrograms.java index 250ad1f1d8fbe..f04d30d1d9945 100644 --- a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ToChangelogTestPrograms.java +++ b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ToChangelogTestPrograms.java @@ -157,7 +157,7 @@ public class ToChangelogTestPrograms { public static final TableTestProgram UPSERT = TableTestProgram.of( "to-changelog-upsert-input", - "upsert input gets ChangelogNormalize for UPDATE_BEFORE and full deletes") + "upsert input in row semantics gets ChangelogNormalize for UPDATE_BEFORE and emits full deletes") .setupTableSource( SourceTestStep.newBuilder("t") .addSchema( @@ -187,7 +187,8 @@ public class ToChangelogTestPrograms { public static final TableTestProgram UPSERT_PARTITION_BY = TableTestProgram.of( "to-changelog-upsert-partition-by", - "PARTITION BY upsert key + mapping without UB skips ChangelogNormalize") + "PARTITION BY upsert key + mapping without UB skips ChangelogNormalize; " + + "default produces_full_deletes=false nulls non-key columns on DELETE") .setupTableSource( SourceTestStep.newBuilder("t") .addSchema( @@ -206,7 +207,7 @@ public class ToChangelogTestPrograms { "+I[Alice, C, 10]", "+I[Bob, C, 20]", "+I[Alice, C, 30]", - "+I[Bob, D, 20]") + "+I[Bob, D, null]") .build()) .runSql( "INSERT INTO sink SELECT * FROM TO_CHANGELOG(" @@ -598,4 +599,213 @@ public class ToChangelogTestPrograms { ValidationException.class, "Duplicate change operation: 'DELETE'") .build(); + + public static final TableTestProgram PRODUCES_FULL_DELETES_ON_APPEND_ONLY_INPUT = + TableTestProgram.of( + "to-changelog-produces-full-deletes-on-append-only-input", + "fails when produces_full_deletes=true on an input that never emits DELETE rows") + .setupTableSource(SIMPLE_SOURCE) + .runFailingSql( + "SELECT * FROM TO_CHANGELOG(" + + "input => TABLE t, " + + "produces_full_deletes => true)", + ValidationException.class, + "the input table only produces [INSERT] and never emits DELETE rows") + .build(); + + public static final TableTestProgram PRODUCES_FULL_DELETES_WITHOUT_DELETE_IN_OP_MAPPING = + TableTestProgram.of( + "to-changelog-produces-full-deletes-without-delete-in-op-mapping", + "fails when produces_full_deletes=true but the active op_mapping strips DELETE") + .setupTableSource( + SourceTestStep.newBuilder("t") + .addSchema( + "name STRING PRIMARY KEY NOT ENFORCED", "score BIGINT") + .addMode(ChangelogMode.all()) + .producedValues(Row.ofKind(RowKind.INSERT, "Alice", 10L)) + .build()) + .runFailingSql( + "SELECT * FROM TO_CHANGELOG(" + + "input => TABLE t, " + + "op_mapping => MAP['INSERT, UPDATE_AFTER', 'X'], " + + "produces_full_deletes => true)", + ValidationException.class, + "the active 'op_mapping' does not map DELETE rows") + .build(); + + // -------------------------------------------------------------------------------------------- + // Row semantics x delete handling matrix + // -------------------------------------------------------------------------------------------- + + public static final TableTestProgram ROW_SEM_PARTIAL_DELETES = + TableTestProgram.of( + "to-changelog-row-sem-partial-deletes", + "row semantics: produces_full_deletes=false skips ChangelogNormalize and a partial DELETE row from the input passes through unchanged") + .setupTableSource( + SourceTestStep.newBuilder("t") + .addSchema( + "name STRING PRIMARY KEY NOT ENFORCED", "score BIGINT") + .addMode(ChangelogMode.all()) + .producedValues( + Row.ofKind(RowKind.INSERT, "Alice", 10L), + Row.ofKind(RowKind.INSERT, "Bob", 20L), + Row.ofKind(RowKind.UPDATE_BEFORE, "Alice", 10L), + Row.ofKind(RowKind.UPDATE_AFTER, "Alice", 30L), + Row.ofKind(RowKind.DELETE, "Bob", null)) + .build()) + .setupTableSink( + SinkTestStep.newBuilder("sink") + .addSchema("op STRING", "name STRING", "score BIGINT") + .consumedValues( + "+I[INSERT, Alice, 10]", + "+I[INSERT, Bob, 20]", + "+I[UPDATE_BEFORE, Alice, 10]", + "+I[UPDATE_AFTER, Alice, 30]", + "+I[DELETE, Bob, null]") + .build()) + .runSql( + "INSERT INTO sink SELECT * FROM TO_CHANGELOG(" + + "input => TABLE t, " + + "produces_full_deletes => false)") + .build(); + + public static final TableTestProgram ROW_SEM_FORCE_FULL_DELETES = + TableTestProgram.of( + "to-changelog-row-sem-force-full-deletes", + "row semantics: produces_full_deletes=true forces ChangelogNormalize to materialize the full DELETE row from an upsert source emitting key-only deletes") + .setupTableSource( + SourceTestStep.newBuilder("t") + .addSchema( + "name STRING PRIMARY KEY NOT ENFORCED", "score BIGINT") + .addMode(ChangelogMode.upsert()) + .producedValues( + Row.ofKind(RowKind.INSERT, "Alice", 10L), + // Key-only delete: ChangelogNormalize fills the row. + Row.ofKind(RowKind.DELETE, "Alice", null)) + .build()) + .setupTableSink( + SinkTestStep.newBuilder("sink") + .addSchema("op STRING", "name STRING", "score BIGINT") + .consumedValues( + "+I[INSERT, Alice, 10]", "+I[DELETE, Alice, 10]") + .build()) + .runSql( + "INSERT INTO sink SELECT * FROM TO_CHANGELOG(" + + "input => TABLE t, " + + "produces_full_deletes => true)") + .build(); + + // -------------------------------------------------------------------------------------------- + // Set semantics x delete handling matrix + // -------------------------------------------------------------------------------------------- + + public static final TableTestProgram SET_SEM_FORCE_PARTIAL_DELETES = + TableTestProgram.of( + "to-changelog-set-sem-force-partial-deletes", + "set semantics: produces_full_deletes=false nulls non-partition-key columns on DELETE even when the input row is fully populated") + .setupTableSource( + SourceTestStep.newBuilder("t") + .addSchema( + "name STRING PRIMARY KEY NOT ENFORCED", "score BIGINT") + .addMode(ChangelogMode.all()) + .producedValues( + Row.ofKind(RowKind.INSERT, "Alice", 10L), + Row.ofKind(RowKind.INSERT, "Bob", 20L), + Row.ofKind(RowKind.UPDATE_AFTER, "Alice", 30L), + Row.ofKind(RowKind.DELETE, "Bob", 20L)) + .build()) + .setupTableSink( + SinkTestStep.newBuilder("sink") + .addSchema("name STRING", "op STRING", "score BIGINT") + .consumedValues( + "+I[Alice, INSERT, 10]", + "+I[Bob, INSERT, 20]", + "+I[Alice, UPDATE_AFTER, 30]", + "+I[Bob, DELETE, null]") + .build()) + .runSql( + "INSERT INTO sink SELECT * FROM TO_CHANGELOG(" + + "input => TABLE t PARTITION BY name," + + "produces_full_deletes => false)") + .build(); + + public static final TableTestProgram SET_SEM_PARTIAL_DELETES = + TableTestProgram.of( + "to-changelog-set-sem-partial-deletes", + "set semantics: produces_full_deletes=false (default) lets a partial DELETE row from the input pass through with non-partition-key columns null") + .setupTableSource( + SourceTestStep.newBuilder("t") + .addSchema( + "name STRING PRIMARY KEY NOT ENFORCED", "score BIGINT") + .addMode(ChangelogMode.all()) + .producedValues( + Row.ofKind(RowKind.INSERT, "Alice", 10L), + Row.ofKind(RowKind.DELETE, "Alice", null)) + .build()) + .setupTableSink( + SinkTestStep.newBuilder("sink") + .addSchema("name STRING", "op STRING", "score BIGINT") + .consumedValues( + "+I[Alice, INSERT, 10]", "+I[Alice, DELETE, null]") + .build()) + .runSql( + "INSERT INTO sink SELECT * FROM TO_CHANGELOG(" + + "input => TABLE t PARTITION BY name)") + .build(); + + public static final TableTestProgram SET_SEM_FULL_DELETES = + TableTestProgram.of( + "to-changelog-set-sem-full-deletes", + "set semantics: produces_full_deletes=true on an input that already emits full deletes is a no-op for the planner and the full DELETE row reaches the output") + .setupTableSource( + SourceTestStep.newBuilder("t") + .addSchema( + "name STRING PRIMARY KEY NOT ENFORCED", "score BIGINT") + .addMode(ChangelogMode.all()) + .producedValues( + Row.ofKind(RowKind.INSERT, "Alice", 10L), + Row.ofKind(RowKind.DELETE, "Alice", 10L)) + .build()) + .setupTableSink( + SinkTestStep.newBuilder("sink") + .addSchema("name STRING", "op STRING", "score BIGINT") + .consumedValues( + "+I[Alice, INSERT, 10]", "+I[Alice, DELETE, 10]") + .build()) + .runSql( + "INSERT INTO sink SELECT * FROM TO_CHANGELOG(" + + "input => TABLE t PARTITION BY name, " + + "produces_full_deletes => true)") + .build(); + + public static final TableTestProgram SET_SEM_FORCE_FULL_DELETES = + TableTestProgram.of( + "to-changelog-set-sem-force-full-deletes", + "set semantics: produces_full_deletes=true forces ChangelogNormalize to materialize the full DELETE row from an upsert source emitting key-only deletes") + .setupTableSource( + SourceTestStep.newBuilder("t") + .addSchema( + "name STRING PRIMARY KEY NOT ENFORCED", "score BIGINT") + .addMode(ChangelogMode.upsert()) + .producedValues( + Row.ofKind(RowKind.INSERT, "Alice", 10L), + Row.ofKind(RowKind.INSERT, "Bob", 20L), + Row.ofKind(RowKind.UPDATE_AFTER, "Alice", 30L), + Row.ofKind(RowKind.DELETE, "Bob", null)) + .build()) + .setupTableSink( + SinkTestStep.newBuilder("sink") + .addSchema("name STRING", "op STRING", "score BIGINT") + .consumedValues( + "+I[Alice, INSERT, 10]", + "+I[Bob, INSERT, 20]", + "+I[Alice, UPDATE_BEFORE, 10]", + "+I[Alice, UPDATE_AFTER, 30]", + "+I[Bob, DELETE, 20]") + .build()) + .runSql( + "INSERT INTO sink SELECT * FROM TO_CHANGELOG(" + + "input => TABLE t PARTITION BY name, " + + "produces_full_deletes => true)") + .build(); } diff --git a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/stream/sql/ToChangelogTest.java b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/stream/sql/ToChangelogTest.java index 7f6f93849c282..1f7347b027e7f 100644 --- a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/stream/sql/ToChangelogTest.java +++ b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/stream/sql/ToChangelogTest.java @@ -144,4 +144,42 @@ void testUpsertPartitionByNoUpdateBeforeAndDelete() { + "op_mapping => MAP['INSERT,UPDATE_AFTER', 'C'])", CHANGELOG_MODE); } + + @Test + void testUpsertSourceProducesFullDeletes() { + util.tableEnv() + .executeSql( + "CREATE TABLE upsert_source (" + + " id INT," + + " name STRING," + + " PRIMARY KEY (id) NOT ENFORCED" + + ") WITH (" + + " 'connector' = 'values'," + + " 'changelog-mode' = 'I,UA,D'" + + ")"); + util.verifyRelPlan( + "SELECT * FROM TO_CHANGELOG(" + + "input => TABLE upsert_source, " + + "produces_full_deletes => true)", + CHANGELOG_MODE); + } + + @Test + void testUpsertSourceKeyOnlyDeletes() { + util.tableEnv() + .executeSql( + "CREATE TABLE upsert_source (" + + " id INT," + + " name STRING," + + " PRIMARY KEY (id) NOT ENFORCED" + + ") WITH (" + + " 'connector' = 'values'," + + " 'changelog-mode' = 'I,UA,D'" + + ")"); + util.verifyRelPlan( + "SELECT * FROM TO_CHANGELOG(" + + "input => TABLE upsert_source, " + + "produces_full_deletes => false)", + CHANGELOG_MODE); + } } diff --git a/flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/plan/stream/sql/ToChangelogTest.xml b/flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/plan/stream/sql/ToChangelogTest.xml index 7cea1058e5c82..174addbf41890 100644 --- a/flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/plan/stream/sql/ToChangelogTest.xml +++ b/flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/plan/stream/sql/ToChangelogTest.xml @@ -23,14 +23,14 @@ limitations under the License. @@ -42,14 +42,14 @@ ProcessTableFunction(invocation=[TO_CHANGELOG(TABLE(#0), DEFAULT(), DEFAULT(), D @@ -62,14 +62,14 @@ ProcessTableFunction(invocation=[TO_CHANGELOG(TABLE(#0) PARTITION BY($0), DEFAUL @@ -81,14 +81,14 @@ ProcessTableFunction(invocation=[TO_CHANGELOG(TABLE(#0), DEFAULT(), DEFAULT(), D @@ -101,14 +101,14 @@ ProcessTableFunction(invocation=[TO_CHANGELOG(TABLE(#0) PARTITION BY($0), DESCRI @@ -121,14 +121,56 @@ ProcessTableFunction(invocation=[TO_CHANGELOG(TABLE(#0) PARTITION BY($0), DESCRI + + + + + TABLE upsert_source, produces_full_deletes => false)]]> + + + + + + + + + + + TABLE upsert_source, produces_full_deletes => true)]]> + + + + + + ", - "description" : "ProcessTableFunction(invocation=[TO_CHANGELOG(TABLE(#0), DEFAULT(), DEFAULT(), DEFAULT(), DEFAULT())], uid=[null], select=[op,name,score], rowType=[RecordType(VARCHAR(2147483647) op, VARCHAR(2147483647) name, BIGINT score)])", + "description" : "ProcessTableFunction(invocation=[TO_CHANGELOG(TABLE(#0), DEFAULT(), DEFAULT(), DEFAULT(), DEFAULT(), DEFAULT())], uid=[null], select=[op,name,score], rowType=[RecordType(VARCHAR(2147483647) op, VARCHAR(2147483647) name, BIGINT score)])", "uid" : null, "functionCall" : { "kind" : "CALL", @@ -59,6 +59,11 @@ "syntax" : "SPECIAL", "internalName" : "$DEFAULT$1", "type" : "MAP" + }, { + "kind" : "CALL", + "syntax" : "SPECIAL", + "internalName" : "$DEFAULT$1", + "type" : "BOOLEAN" }, { "kind" : "CALL", "syntax" : "SPECIAL", diff --git a/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/ptf/ToChangelogFunction.java b/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/ptf/ToChangelogFunction.java index 0b6c6ca55b88e..0110475ef8d1d 100644 --- a/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/ptf/ToChangelogFunction.java +++ b/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/ptf/ToChangelogFunction.java @@ -65,11 +65,13 @@ public class ToChangelogFunction extends BuiltInProcessTableFunction { private final Map rawOpMap; private final int[] outputIndices; + private final boolean producesFullDelete; private transient Map opMap; private transient GenericRowData opRow; private transient JoinedRowData output; private transient ProjectedRowData projectedOutput; + private transient GenericRowData nullPayloadRow; @SuppressWarnings("unchecked") public ToChangelogFunction(final SpecializedContext context) { @@ -84,7 +86,30 @@ public ToChangelogFunction(final SpecializedContext context) { if (opMapping != null) { validateOpMap(this.rawOpMap, tableSemantics); } + final boolean producesFullDeletesArg = + callContext.getArgumentValue(3, Boolean.class).orElse(false); + validateProducesFullDeletes(producesFullDeletesArg, this.rawOpMap, tableSemantics); + this.outputIndices = ChangelogTypeStrategyUtils.computeOutputIndices(tableSemantics); + this.producesFullDelete = resolveProducesFullDelete(producesFullDeletesArg, tableSemantics); + } + + /** + * Decides whether this function emits full DELETE rows (input passed through unchanged) or + * partial DELETE rows (only identifying columns preserved, rest nulled). + * + *

The framework prepends partition-key columns to the output without consulting this + * function, so in set semantics partition keys are preserved on DELETE rows for free. In row + * semantics there is no key column to preserve, so the function passes the input through + * unchanged regardless of {@code produces_full_deletes}. + */ + private static boolean resolveProducesFullDelete( + final boolean producesFullDeletesArg, final TableSemantics tableSemantics) { + if (producesFullDeletesArg) { + return true; + } + final boolean hasPartitionBy = tableSemantics.partitionByColumns().length > 0; + return !hasPartitionBy; } @Override @@ -95,6 +120,7 @@ public void open(final FunctionContext context) throws Exception { opRow = new GenericRowData(1); output = new JoinedRowData(); projectedOutput = ProjectedRowData.from(outputIndices); + nullPayloadRow = new GenericRowData(outputIndices.length); } /** @@ -145,17 +171,59 @@ private static void validateOpMap( } } + /** + * Rejects {@code produces_full_deletes=true} when the input changelog cannot produce DELETE + * rows (either the input mode does not contain DELETE, or the active {@code op_mapping} strips + * it). The parameter is then dead and likely a user mistake. + * + *

Lives here rather than in the input type strategy because {@link + * TableSemantics#changelogMode()} returns empty during type inference and is only populated at + * specialization time, which is when this constructor runs. + */ + private static void validateProducesFullDeletes( + final boolean producesFullDeletesArg, + final Map mapping, + final TableSemantics tableSemantics) { + if (!producesFullDeletesArg) { + return; + } + final ChangelogMode inputMode = tableSemantics.changelogMode().orElse(null); + if (inputMode == null) { + return; + } + if (!inputMode.contains(RowKind.DELETE)) { + throw new ValidationException( + String.format( + "Invalid 'produces_full_deletes' for TO_CHANGELOG: the input table " + + "only produces %s and never emits DELETE rows. Remove the " + + "'produces_full_deletes' argument.", + inputMode.getContainedKinds())); + } + if (!mapping.containsKey(RowKind.DELETE)) { + throw new ValidationException( + "Invalid 'produces_full_deletes' for TO_CHANGELOG: the active 'op_mapping' " + + "does not map DELETE rows, so no DELETE rows are emitted. Remove " + + "the 'produces_full_deletes' argument or add a DELETE entry to " + + "'op_mapping'."); + } + } + public void eval( final Context ctx, final RowData input, @Nullable final ColumnList op, - @Nullable final MapData opMapping) { + @Nullable final MapData opMapping, + @Nullable final Boolean producesFullDeletes) { final StringData opCode = opMap.get(input.getRowKind()); if (opCode == null) { return; } opRow.setField(0, opCode); - collect(output.replace(opRow, projectedOutput.replaceRow(input))); + final RowData payload = + (input.getRowKind() == RowKind.DELETE && !producesFullDelete) + ? nullPayloadRow + : projectedOutput.replaceRow(input); + collect(output.replace(opRow, payload)); } } From e08cdd21d165e9eec47fc31f07e8e2c6efba0e0a Mon Sep 17 00:00:00 2001 From: Ramin Gharib Date: Fri, 22 May 2026 17:06:54 +0200 Subject: [PATCH 2/8] [FLINK-39636][table] Refactor ToChangelogTypeStrategy to match FromChangelogTypeStrategy Aligns the structure of ToChangelogTypeStrategy with its sibling FromChangelogTypeStrategy: introduces positional argument-index constants, switches the input type strategy to ValidationOnlyInputTypeStrategy so signature/argument-count come from the StaticArguments in the function definition, splits the monolithic validateInputs into per-argument helpers, and reuses ChangelogTypeStrategyUtils.resolveOpColumnName instead of the local duplicate. --- .../strategies/ToChangelogTypeStrategy.java | 182 +++++++++++------- .../ToChangelogInputTypeStrategyTest.java | 146 ++++++++++++++ .../exec/stream/ToChangelogSemanticTests.java | 1 - .../exec/stream/ToChangelogTestPrograms.java | 19 -- .../functions/ptf/ToChangelogFunction.java | 20 +- 5 files changed, 267 insertions(+), 101 deletions(-) create mode 100644 flink-table/flink-table-common/src/test/java/org/apache/flink/table/types/inference/strategies/ToChangelogInputTypeStrategyTest.java diff --git a/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/ToChangelogTypeStrategy.java b/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/ToChangelogTypeStrategy.java index 8ff8101dc5e5e..d406875aa2fec 100644 --- a/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/ToChangelogTypeStrategy.java +++ b/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/ToChangelogTypeStrategy.java @@ -22,15 +22,10 @@ import org.apache.flink.table.api.DataTypes; import org.apache.flink.table.api.DataTypes.Field; import org.apache.flink.table.api.ValidationException; -import org.apache.flink.table.functions.FunctionDefinition; import org.apache.flink.table.functions.TableSemantics; import org.apache.flink.table.types.DataType; -import org.apache.flink.table.types.inference.ArgumentCount; import org.apache.flink.table.types.inference.CallContext; -import org.apache.flink.table.types.inference.ConstantArgumentCount; import org.apache.flink.table.types.inference.InputTypeStrategy; -import org.apache.flink.table.types.inference.Signature; -import org.apache.flink.table.types.inference.Signature.Argument; import org.apache.flink.table.types.inference.TypeStrategy; import org.apache.flink.types.ColumnList; @@ -46,7 +41,13 @@ @Internal public final class ToChangelogTypeStrategy { - private static final String DEFAULT_OP_COLUMN_NAME = "op"; + // Positional argument indexes for TO_CHANGELOG. Must match the order of StaticArguments + // registered in BuiltInFunctionDefinitions#TO_CHANGELOG; changing one without the other + // silently breaks argument resolution. + public static final int ARG_TABLE = 0; + public static final int ARG_OP = 1; + public static final int ARG_OP_MAPPING = 2; + public static final int ARG_PRODUCES_FULL_DELETES = 3; private static final Set VALID_ROW_KIND_NAMES = Set.of("INSERT", "UPDATE_BEFORE", "UPDATE_AFTER", "DELETE"); @@ -56,34 +57,12 @@ public final class ToChangelogTypeStrategy { // -------------------------------------------------------------------------------------------- public static final InputTypeStrategy INPUT_TYPE_STRATEGY = - new InputTypeStrategy() { - @Override - public ArgumentCount getArgumentCount() { - return ConstantArgumentCount.between(1, 4); - } - + new ValidationOnlyInputTypeStrategy() { @Override public Optional> inferInputTypes( final CallContext callContext, final boolean throwOnFailure) { return validateInputs(callContext, throwOnFailure); } - - @Override - public List getExpectedSignatures(final FunctionDefinition definition) { - return List.of( - Signature.of(Argument.of("input", "TABLE")), - Signature.of( - Argument.of("input", "TABLE"), Argument.of("op", "DESCRIPTOR")), - Signature.of( - Argument.of("input", "TABLE"), - Argument.of("op", "DESCRIPTOR"), - Argument.of("op_mapping", "MAP")), - Signature.of( - Argument.of("input", "TABLE"), - Argument.of("op", "DESCRIPTOR"), - Argument.of("op_mapping", "MAP"), - Argument.of("produces_full_deletes", "BOOLEAN"))); - } }; // -------------------------------------------------------------------------------------------- @@ -94,20 +73,16 @@ public List getExpectedSignatures(final FunctionDefinition definition callContext -> { final TableSemantics semantics = callContext - .getTableSemantics(0) + .getTableSemantics(ARG_TABLE) .orElseThrow( () -> new ValidationException( "First argument must be a table for TO_CHANGELOG.")); - final String opColumnName = resolveOpColumnName(callContext); - final List inputFields = DataType.getFields(semantics.dataType()); - final int[] outputIndices = - ChangelogTypeStrategyUtils.computeOutputIndices(semantics); + final String opColumnName = + ChangelogTypeStrategyUtils.resolveOpColumnName(callContext); - final List outputFields = new ArrayList<>(); - outputFields.add(DataTypes.FIELD(opColumnName, DataTypes.STRING())); - Arrays.stream(outputIndices).mapToObj(inputFields::get).forEach(outputFields::add); + final List outputFields = buildOutputFields(semantics, opColumnName); return Optional.of(DataTypes.ROW(outputFields).notNull()); }; @@ -118,45 +93,68 @@ public List getExpectedSignatures(final FunctionDefinition definition private static Optional> validateInputs( final CallContext callContext, final boolean throwOnFailure) { - final boolean isMissingTableArg = callContext.getTableSemantics(0).isEmpty(); - if (isMissingTableArg) { + Optional> error; + + error = validateTableArg(callContext, throwOnFailure); + if (error.isPresent()) { + return error; + } + + error = validateOpDescriptor(callContext, throwOnFailure); + if (error.isPresent()) { + return error; + } + + error = validateOpMapping(callContext, throwOnFailure); + if (error.isPresent()) { + return error; + } + + error = validateProducesFullDeletes(callContext, throwOnFailure); + if (error.isPresent()) { + return error; + } + + return Optional.of(callContext.getArgumentDataTypes()); + } + + private static Optional> validateTableArg( + final CallContext callContext, final boolean throwOnFailure) { + if (callContext.getTableSemantics(ARG_TABLE).isEmpty()) { return callContext.fail( throwOnFailure, "First argument must be a table for TO_CHANGELOG."); } + return Optional.empty(); + } - final Optional opDescriptor = callContext.getArgumentValue(1, ColumnList.class); - final boolean hasInvalidOpDescriptor = - opDescriptor.isPresent() && opDescriptor.get().getNames().size() != 1; - if (hasInvalidOpDescriptor) { + private static Optional> validateOpDescriptor( + final CallContext callContext, final boolean throwOnFailure) { + final Optional opDescriptor = + callContext.getArgumentValue(ARG_OP, ColumnList.class); + if (opDescriptor.isPresent() && opDescriptor.get().getNames().size() != 1) { return callContext.fail( throwOnFailure, "The descriptor for argument 'op' must contain exactly one column name."); } + return Optional.empty(); + } - final boolean hasMappingArgProvided = !callContext.isArgumentNull(2); - final boolean isMappingArgLiteral = callContext.isArgumentLiteral(2); + /** Validates op_mapping is a constant literal and that its keys are well-formed. */ + @SuppressWarnings("rawtypes") + private static Optional> validateOpMapping( + final CallContext callContext, final boolean throwOnFailure) { + final boolean hasMappingArgProvided = !callContext.isArgumentNull(ARG_OP_MAPPING); + final boolean isMappingArgLiteral = callContext.isArgumentLiteral(ARG_OP_MAPPING); if (hasMappingArgProvided && !isMappingArgLiteral) { return callContext.fail( throwOnFailure, "The 'op_mapping' argument must be a constant MAP literal."); } - final Optional opMapping = callContext.getArgumentValue(2, Map.class); + final Optional opMapping = callContext.getArgumentValue(ARG_OP_MAPPING, Map.class); if (opMapping.isPresent()) { - final Optional> validationError = - validateOpMappingKeys(callContext, opMapping.get(), throwOnFailure); - if (validationError.isPresent()) { - return validationError; - } + return validateOpMappingKeys(callContext, opMapping.get(), throwOnFailure); } - - final boolean hasProducesFullDeletesArg = !callContext.isArgumentNull(3); - if (hasProducesFullDeletesArg && !callContext.isArgumentLiteral(3)) { - return callContext.fail( - throwOnFailure, - "The 'produces_full_deletes' argument must be a constant BOOLEAN literal."); - } - - return Optional.of(callContext.getArgumentDataTypes()); + return Optional.empty(); } /** @@ -199,12 +197,64 @@ private static Optional> validateOpMappingKeys( return Optional.empty(); } - private static String resolveOpColumnName(final CallContext callContext) { - return callContext - .getArgumentValue(1, ColumnList.class) - .filter(cl -> !cl.getNames().isEmpty()) - .map(cl -> cl.getNames().get(0)) - .orElse(DEFAULT_OP_COLUMN_NAME); + @SuppressWarnings("rawtypes") + private static Optional> validateProducesFullDeletes( + final CallContext callContext, final boolean throwOnFailure) { + final boolean hasArgProvided = !callContext.isArgumentNull(ARG_PRODUCES_FULL_DELETES); + if (hasArgProvided && !callContext.isArgumentLiteral(ARG_PRODUCES_FULL_DELETES)) { + return callContext.fail( + throwOnFailure, + "The 'produces_full_deletes' argument must be a constant BOOLEAN literal."); + } + final boolean producesFullDeletes = + callContext + .getArgumentValue(ARG_PRODUCES_FULL_DELETES, Boolean.class) + .orElse(false); + if (!producesFullDeletes) { + return Optional.empty(); + } + // The check against the input changelog mode lives in the function constructor since + // TableSemantics#changelogMode() returns empty here at type-inference time. The mapping + // check below only needs the literal op_mapping argument, so it lives here. + final Optional opMapping = callContext.getArgumentValue(ARG_OP_MAPPING, Map.class); + if (opMapping.isPresent() && !mapsDelete(opMapping.get())) { + return callContext.fail( + throwOnFailure, + "Invalid 'produces_full_deletes' for TO_CHANGELOG: the active 'op_mapping' " + + "does not map DELETE rows, so no DELETE rows are emitted. Remove " + + "the 'produces_full_deletes' argument or add a DELETE entry to " + + "'op_mapping'."); + } + return Optional.empty(); + } + + /** + * Returns {@code true} when at least one {@code op_mapping} key references {@code DELETE}. + * Keys may be comma-separated (e.g., {@code "INSERT, DELETE"}) per the user-facing contract. + */ + @SuppressWarnings("rawtypes") + private static boolean mapsDelete(final Map opMapping) { + for (final Object key : opMapping.keySet()) { + if (!(key instanceof String)) { + continue; + } + for (final String rawName : ((String) key).split(",")) { + if ("DELETE".equals(rawName.trim())) { + return true; + } + } + } + return false; + } + + private static List buildOutputFields( + final TableSemantics semantics, final String opColumnName) { + final List inputFields = DataType.getFields(semantics.dataType()); + final int[] outputIndices = ChangelogTypeStrategyUtils.computeOutputIndices(semantics); + final List outputFields = new ArrayList<>(); + outputFields.add(DataTypes.FIELD(opColumnName, DataTypes.STRING())); + Arrays.stream(outputIndices).mapToObj(inputFields::get).forEach(outputFields::add); + return outputFields; } private ToChangelogTypeStrategy() {} diff --git a/flink-table/flink-table-common/src/test/java/org/apache/flink/table/types/inference/strategies/ToChangelogInputTypeStrategyTest.java b/flink-table/flink-table-common/src/test/java/org/apache/flink/table/types/inference/strategies/ToChangelogInputTypeStrategyTest.java new file mode 100644 index 0000000000000..e9b20a7a5ba02 --- /dev/null +++ b/flink-table/flink-table-common/src/test/java/org/apache/flink/table/types/inference/strategies/ToChangelogInputTypeStrategyTest.java @@ -0,0 +1,146 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.table.types.inference.strategies; + +import org.apache.flink.table.api.DataTypes; +import org.apache.flink.table.types.DataType; +import org.apache.flink.table.types.inference.InputTypeStrategiesTestBase; +import org.apache.flink.table.types.inference.utils.TableSemanticsMock; +import org.apache.flink.types.ColumnList; + +import java.util.Map; +import java.util.stream.Stream; + +import static org.apache.flink.table.types.inference.strategies.SpecificInputTypeStrategies.TO_CHANGELOG_INPUT_TYPE_STRATEGY; + +/** Tests for {@link ToChangelogTypeStrategy#INPUT_TYPE_STRATEGY}. */ +class ToChangelogInputTypeStrategyTest extends InputTypeStrategiesTestBase { + + private static final DataType TABLE_TYPE = + DataTypes.ROW( + DataTypes.FIELD("name", DataTypes.STRING()), + DataTypes.FIELD("score", DataTypes.BIGINT())); + + private static final DataType DESCRIPTOR_TYPE = DataTypes.DESCRIPTOR(); + + private static final DataType MAP_TYPE = DataTypes.MAP(DataTypes.STRING(), DataTypes.STRING()); + + private static final DataType BOOLEAN_TYPE = DataTypes.BOOLEAN(); + + @Override + protected Stream testData() { + return Stream.of( + // Valid: produces_full_deletes=true with default op_mapping (includes DELETE) + TestSpec.forStrategy( + "Valid produces_full_deletes=true with default mapping", + TO_CHANGELOG_INPUT_TYPE_STRATEGY) + .calledWithArgumentTypes( + TABLE_TYPE, DESCRIPTOR_TYPE, MAP_TYPE, BOOLEAN_TYPE) + .calledWithTableSemanticsAt(0, new TableSemanticsMock(TABLE_TYPE)) + .calledWithLiteralAt(1, ColumnList.of("op")) + .calledWithLiteralAt(2, null) + .calledWithLiteralAt(3, true) + .expectArgumentTypes( + TABLE_TYPE, DESCRIPTOR_TYPE, MAP_TYPE, BOOLEAN_TYPE), + + // Valid: produces_full_deletes=true with op_mapping that includes DELETE + TestSpec.forStrategy( + "Valid produces_full_deletes=true with explicit DELETE mapping", + TO_CHANGELOG_INPUT_TYPE_STRATEGY) + .calledWithArgumentTypes( + TABLE_TYPE, DESCRIPTOR_TYPE, MAP_TYPE, BOOLEAN_TYPE) + .calledWithTableSemanticsAt(0, new TableSemanticsMock(TABLE_TYPE)) + .calledWithLiteralAt(1, ColumnList.of("op")) + .calledWithLiteralAt(2, Map.of("INSERT", "I", "DELETE", "D")) + .calledWithLiteralAt(3, true) + .expectArgumentTypes( + TABLE_TYPE, DESCRIPTOR_TYPE, MAP_TYPE, BOOLEAN_TYPE), + + // Valid: produces_full_deletes=true with comma-separated DELETE key + TestSpec.forStrategy( + "Valid produces_full_deletes=true with comma-separated DELETE", + TO_CHANGELOG_INPUT_TYPE_STRATEGY) + .calledWithArgumentTypes( + TABLE_TYPE, DESCRIPTOR_TYPE, MAP_TYPE, BOOLEAN_TYPE) + .calledWithTableSemanticsAt(0, new TableSemanticsMock(TABLE_TYPE)) + .calledWithLiteralAt(1, ColumnList.of("op")) + .calledWithLiteralAt(2, Map.of("INSERT, DELETE", "X")) + .calledWithLiteralAt(3, true) + .expectArgumentTypes( + TABLE_TYPE, DESCRIPTOR_TYPE, MAP_TYPE, BOOLEAN_TYPE), + + // Valid: produces_full_deletes=false with op_mapping that omits DELETE + TestSpec.forStrategy( + "Valid produces_full_deletes=false with no DELETE in mapping", + TO_CHANGELOG_INPUT_TYPE_STRATEGY) + .calledWithArgumentTypes( + TABLE_TYPE, DESCRIPTOR_TYPE, MAP_TYPE, BOOLEAN_TYPE) + .calledWithTableSemanticsAt(0, new TableSemanticsMock(TABLE_TYPE)) + .calledWithLiteralAt(1, ColumnList.of("op")) + .calledWithLiteralAt(2, Map.of("INSERT, UPDATE_AFTER", "X")) + .calledWithLiteralAt(3, false) + .expectArgumentTypes( + TABLE_TYPE, DESCRIPTOR_TYPE, MAP_TYPE, BOOLEAN_TYPE), + + // Error: produces_full_deletes=true with op_mapping that strips DELETE + TestSpec.forStrategy( + "produces_full_deletes=true rejected when op_mapping omits DELETE", + TO_CHANGELOG_INPUT_TYPE_STRATEGY) + .calledWithArgumentTypes( + TABLE_TYPE, DESCRIPTOR_TYPE, MAP_TYPE, BOOLEAN_TYPE) + .calledWithTableSemanticsAt(0, new TableSemanticsMock(TABLE_TYPE)) + .calledWithLiteralAt(1, ColumnList.of("op")) + .calledWithLiteralAt(2, Map.of("INSERT, UPDATE_AFTER", "X")) + .calledWithLiteralAt(3, true) + .expectErrorMessage( + "Invalid 'produces_full_deletes' for TO_CHANGELOG: the active " + + "'op_mapping' does not map DELETE rows"), + + // Error: multi-column descriptor + TestSpec.forStrategy( + "Descriptor with multiple columns", + TO_CHANGELOG_INPUT_TYPE_STRATEGY) + .calledWithArgumentTypes( + TABLE_TYPE, DESCRIPTOR_TYPE, MAP_TYPE, BOOLEAN_TYPE) + .calledWithTableSemanticsAt(0, new TableSemanticsMock(TABLE_TYPE)) + .calledWithLiteralAt(1, ColumnList.of("a", "b")) + .expectErrorMessage("must contain exactly one column name"), + + // Error: invalid RowKind in op_mapping key + TestSpec.forStrategy( + "Invalid RowKind in mapping key", TO_CHANGELOG_INPUT_TYPE_STRATEGY) + .calledWithArgumentTypes( + TABLE_TYPE, DESCRIPTOR_TYPE, MAP_TYPE, BOOLEAN_TYPE) + .calledWithTableSemanticsAt(0, new TableSemanticsMock(TABLE_TYPE)) + .calledWithLiteralAt(1, ColumnList.of("op")) + .calledWithLiteralAt(2, Map.of("INVALID_KIND", "X")) + .expectErrorMessage("Unknown change operation: 'INVALID_KIND'"), + + // Error: duplicate RowKind across entries + TestSpec.forStrategy( + "Duplicate RowKind in mapping keys", + TO_CHANGELOG_INPUT_TYPE_STRATEGY) + .calledWithArgumentTypes( + TABLE_TYPE, DESCRIPTOR_TYPE, MAP_TYPE, BOOLEAN_TYPE) + .calledWithTableSemanticsAt(0, new TableSemanticsMock(TABLE_TYPE)) + .calledWithLiteralAt(1, ColumnList.of("op")) + .calledWithLiteralAt(2, Map.of("INSERT, DELETE", "A", "DELETE", "B")) + .expectErrorMessage("Duplicate change operation: 'DELETE'")); + } +} diff --git a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ToChangelogSemanticTests.java b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ToChangelogSemanticTests.java index 762e08b353895..b3376b3a34677 100644 --- a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ToChangelogSemanticTests.java +++ b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ToChangelogSemanticTests.java @@ -57,7 +57,6 @@ public List programs() { ToChangelogTestPrograms.OP_MAPPING_REFERENCES_UNSUPPORTED_KIND, ToChangelogTestPrograms.DUPLICATE_ROW_KIND, ToChangelogTestPrograms.PRODUCES_FULL_DELETES_ON_APPEND_ONLY_INPUT, - ToChangelogTestPrograms.PRODUCES_FULL_DELETES_WITHOUT_DELETE_IN_OP_MAPPING, ToChangelogTestPrograms.ROW_SEM_PARTIAL_DELETES, ToChangelogTestPrograms.ROW_SEM_FORCE_FULL_DELETES, ToChangelogTestPrograms.SET_SEM_PARTIAL_DELETES, diff --git a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ToChangelogTestPrograms.java b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ToChangelogTestPrograms.java index f04d30d1d9945..5126014b9a36a 100644 --- a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ToChangelogTestPrograms.java +++ b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ToChangelogTestPrograms.java @@ -613,25 +613,6 @@ public class ToChangelogTestPrograms { "the input table only produces [INSERT] and never emits DELETE rows") .build(); - public static final TableTestProgram PRODUCES_FULL_DELETES_WITHOUT_DELETE_IN_OP_MAPPING = - TableTestProgram.of( - "to-changelog-produces-full-deletes-without-delete-in-op-mapping", - "fails when produces_full_deletes=true but the active op_mapping strips DELETE") - .setupTableSource( - SourceTestStep.newBuilder("t") - .addSchema( - "name STRING PRIMARY KEY NOT ENFORCED", "score BIGINT") - .addMode(ChangelogMode.all()) - .producedValues(Row.ofKind(RowKind.INSERT, "Alice", 10L)) - .build()) - .runFailingSql( - "SELECT * FROM TO_CHANGELOG(" - + "input => TABLE t, " - + "op_mapping => MAP['INSERT, UPDATE_AFTER', 'X'], " - + "produces_full_deletes => true)", - ValidationException.class, - "the active 'op_mapping' does not map DELETE rows") - .build(); // -------------------------------------------------------------------------------------------- // Row semantics x delete handling matrix diff --git a/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/ptf/ToChangelogFunction.java b/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/ptf/ToChangelogFunction.java index 0110475ef8d1d..d8874a2be1695 100644 --- a/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/ptf/ToChangelogFunction.java +++ b/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/ptf/ToChangelogFunction.java @@ -88,7 +88,7 @@ public ToChangelogFunction(final SpecializedContext context) { } final boolean producesFullDeletesArg = callContext.getArgumentValue(3, Boolean.class).orElse(false); - validateProducesFullDeletes(producesFullDeletesArg, this.rawOpMap, tableSemantics); + validateProducesFullDeletes(producesFullDeletesArg, tableSemantics); this.outputIndices = ChangelogTypeStrategyUtils.computeOutputIndices(tableSemantics); this.producesFullDelete = resolveProducesFullDelete(producesFullDeletesArg, tableSemantics); @@ -172,18 +172,15 @@ private static void validateOpMap( } /** - * Rejects {@code produces_full_deletes=true} when the input changelog cannot produce DELETE - * rows (either the input mode does not contain DELETE, or the active {@code op_mapping} strips - * it). The parameter is then dead and likely a user mistake. + * Rejects {@code produces_full_deletes=true} when the input changelog never emits DELETE rows. * *

Lives here rather than in the input type strategy because {@link * TableSemantics#changelogMode()} returns empty during type inference and is only populated at - * specialization time, which is when this constructor runs. + * specialization time. The complementary check against the literal {@code op_mapping} + * argument runs earlier in {@code ToChangelogTypeStrategy}. */ private static void validateProducesFullDeletes( - final boolean producesFullDeletesArg, - final Map mapping, - final TableSemantics tableSemantics) { + final boolean producesFullDeletesArg, final TableSemantics tableSemantics) { if (!producesFullDeletesArg) { return; } @@ -199,13 +196,6 @@ private static void validateProducesFullDeletes( + "'produces_full_deletes' argument.", inputMode.getContainedKinds())); } - if (!mapping.containsKey(RowKind.DELETE)) { - throw new ValidationException( - "Invalid 'produces_full_deletes' for TO_CHANGELOG: the active 'op_mapping' " - + "does not map DELETE rows, so no DELETE rows are emitted. Remove " - + "the 'produces_full_deletes' argument or add a DELETE entry to " - + "'op_mapping'."); - } } public void eval( From 0dfb78c6ed126cbd81904104554570dbae81c96b Mon Sep 17 00:00:00 2001 From: Ramin Gharib Date: Tue, 26 May 2026 10:56:29 +0200 Subject: [PATCH 3/8] [FLINK-39735][table] Expose input upsert key on TableSemantics Adds upsertKeyColumns() to TableSemantics, populated by the planner via FlinkRelMetadataQuery.getUpsertKeys. ProcessTableFunctions can now read the upsert key of each input table at specialization time without having to re-derive it from a RelNode or require the caller to repeat the key via PARTITION BY. Plumbs the value end-to-end: StreamPhysicalProcessTableFunction queries the metadata, StreamExecProcessTableFunction persists it as @JsonProperty inputUpsertKeys (one entry per table input) so compiled plans round-trip, and OperatorBindingCallContext / BridgingSqlFunction / ProcessTableRunnerGenerator thread it to the specialized function constructor. --- .../rules/ResolveCallByArgumentsRule.java | 5 ++ .../flink/table/functions/TableSemantics.java | 16 +++++++ .../inference/utils/TableSemanticsMock.java | 17 +++++++ .../bridging/BridgingSqlFunction.java | 19 +++++++- .../inference/CallBindingCallContext.java | 5 ++ .../inference/OperatorBindingCallContext.java | 47 +++++++++++++++++-- .../StreamExecProcessTableFunction.java | 36 +++++++++++--- .../StreamPhysicalProcessTableFunction.java | 14 +++++- .../codegen/ProcessTableRunnerGenerator.scala | 10 +++- .../plan/to-changelog-retract-restore.json | 3 +- .../process/RuntimeTableSemantics.java | 10 +++- ...tTableOperatorInterruptibleTimersTest.java | 3 +- .../functions/TestHarnessTableSemantics.java | 11 +++++ 13 files changed, 177 insertions(+), 19 deletions(-) diff --git a/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/expressions/resolver/rules/ResolveCallByArgumentsRule.java b/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/expressions/resolver/rules/ResolveCallByArgumentsRule.java index be8e578deea4d..5680cb6521887 100644 --- a/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/expressions/resolver/rules/ResolveCallByArgumentsRule.java +++ b/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/expressions/resolver/rules/ResolveCallByArgumentsRule.java @@ -781,6 +781,11 @@ public Optional changelogMode() { return Optional.empty(); } + @Override + public int[] upsertKeyColumns() { + return new int[0]; + } + private PartitionQueryOperation findPartitionOperation(QueryOperation op) { if (op instanceof PartitionQueryOperation) { return (PartitionQueryOperation) op; diff --git a/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/TableSemantics.java b/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/TableSemantics.java index f63566befce7d..808a5e57a5231 100644 --- a/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/TableSemantics.java +++ b/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/TableSemantics.java @@ -128,6 +128,22 @@ public interface TableSemantics { */ Optional changelogMode(); + /** + * Upsert key columns derived from the passed table's metadata. + * + *

Returns 0-based column indices that uniquely identify a row for upsert semantics. This is + * distinct from {@link #partitionByColumns()}: partition keys describe distribution and + * co-location, upsert keys describe row identity. Useful for functions that need to emit + * key-only deletes, match UPDATE_BEFORE / UPDATE_AFTER pairs, or route CDC events without + * forcing the caller to repeat the key via {@code PARTITION BY}. + * + *

Returns an empty array when no upsert key is derivable (e.g., a pure append-only source) + * or when the planner has not yet computed metadata (during type inference). + * + * @return Indices of the upsert key columns of the passed table, or an empty array if none. + */ + int[] upsertKeyColumns(); + /** The sort direction for ORDER BY columns in table arguments with set semantics. */ @PublicEvolving enum SortDirection { diff --git a/flink-table/flink-table-common/src/test/java/org/apache/flink/table/types/inference/utils/TableSemanticsMock.java b/flink-table/flink-table-common/src/test/java/org/apache/flink/table/types/inference/utils/TableSemanticsMock.java index fe881f8fd1fc2..bd0ead2a3180b 100644 --- a/flink-table/flink-table-common/src/test/java/org/apache/flink/table/types/inference/utils/TableSemanticsMock.java +++ b/flink-table/flink-table-common/src/test/java/org/apache/flink/table/types/inference/utils/TableSemanticsMock.java @@ -35,6 +35,7 @@ public class TableSemanticsMock implements TableSemantics { private final SortDirection[] orderByDirections; private final int timeColumn; private final ChangelogMode changelogMode; + private final int[] upsertKeyColumns; public TableSemanticsMock(DataType dataType) { this(dataType, new int[0], new int[0], -1, null); @@ -46,6 +47,16 @@ public TableSemanticsMock( int[] orderByColumns, int timeColumn, @Nullable ChangelogMode changelogMode) { + this(dataType, partitionByColumns, orderByColumns, timeColumn, changelogMode, new int[0]); + } + + public TableSemanticsMock( + DataType dataType, + int[] partitionByColumns, + int[] orderByColumns, + int timeColumn, + @Nullable ChangelogMode changelogMode, + int[] upsertKeyColumns) { this.dataType = dataType; this.partitionByColumns = partitionByColumns; this.orderByColumns = orderByColumns; @@ -55,6 +66,7 @@ public TableSemanticsMock( } this.timeColumn = timeColumn; this.changelogMode = changelogMode; + this.upsertKeyColumns = upsertKeyColumns; } @Override @@ -86,4 +98,9 @@ public int timeColumn() { public Optional changelogMode() { return Optional.ofNullable(changelogMode); } + + @Override + public int[] upsertKeyColumns() { + return upsertKeyColumns; + } } diff --git a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/bridging/BridgingSqlFunction.java b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/bridging/BridgingSqlFunction.java index 2c393adf875ec..c38aa3eca85a5 100644 --- a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/bridging/BridgingSqlFunction.java +++ b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/bridging/BridgingSqlFunction.java @@ -335,7 +335,7 @@ public boolean hasScalarArgument(String name) { * scalar arguments through the same coercion path as validation. */ public CallContext toCallContext(RexCall call) { - return toCallContext(call, null, null, null); + return toCallContext(call, null, null, null, null); } /** @@ -348,6 +348,20 @@ public CallContext toCallContext( @Nullable List inputTimeColumns, @Nullable List inputChangelogModes, @Nullable ChangelogMode outputChangelogMode) { + return toCallContext( + call, inputTimeColumns, inputChangelogModes, outputChangelogMode, null); + } + + /** + * Variant that additionally exposes the call's input upsert keys. Used by the streaming codegen + * path so PTFs can specialize themselves on the input's row-identity information. + */ + public CallContext toCallContext( + RexCall call, + @Nullable List inputTimeColumns, + @Nullable List inputChangelogModes, + @Nullable ChangelogMode outputChangelogMode, + @Nullable List inputUpsertKeys) { return new OperatorBindingCallContext( dataTypeFactory, getDefinition(), @@ -355,7 +369,8 @@ public CallContext toCallContext( call.getType(), inputTimeColumns, inputChangelogModes, - outputChangelogMode); + outputChangelogMode, + inputUpsertKeys); } /** diff --git a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/inference/CallBindingCallContext.java b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/inference/CallBindingCallContext.java index 065a30335458f..e5615273f3109 100644 --- a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/inference/CallBindingCallContext.java +++ b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/inference/CallBindingCallContext.java @@ -312,6 +312,11 @@ public int timeColumn() { public Optional changelogMode() { return Optional.empty(); } + + @Override + public int[] upsertKeyColumns() { + return new int[0]; + } } // -------------------------------------------------------------------------------------------- diff --git a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/inference/OperatorBindingCallContext.java b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/inference/OperatorBindingCallContext.java index f31406dad196d..f69de54c54d82 100644 --- a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/inference/OperatorBindingCallContext.java +++ b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/inference/OperatorBindingCallContext.java @@ -64,13 +64,14 @@ public final class OperatorBindingCallContext extends AbstractSqlCallContext { private final @Nullable List inputTimeColumns; private final @Nullable List inputChangelogModes; private final @Nullable ChangelogMode outputChangelogMode; + private final @Nullable List inputUpsertKeys; public OperatorBindingCallContext( DataTypeFactory dataTypeFactory, FunctionDefinition definition, SqlOperatorBinding binding, RelDataType returnRelDataType) { - this(dataTypeFactory, definition, binding, returnRelDataType, null, null, null); + this(dataTypeFactory, definition, binding, returnRelDataType, null, null, null, null); } public OperatorBindingCallContext( @@ -81,6 +82,26 @@ public OperatorBindingCallContext( @Nullable List inputTimeColumns, @Nullable List inputChangelogModes, @Nullable ChangelogMode outputChangelogMode) { + this( + dataTypeFactory, + definition, + binding, + returnRelDataType, + inputTimeColumns, + inputChangelogModes, + outputChangelogMode, + null); + } + + public OperatorBindingCallContext( + DataTypeFactory dataTypeFactory, + FunctionDefinition definition, + SqlOperatorBinding binding, + RelDataType returnRelDataType, + @Nullable List inputTimeColumns, + @Nullable List inputChangelogModes, + @Nullable ChangelogMode outputChangelogMode, + @Nullable List inputUpsertKeys) { super( dataTypeFactory, definition, @@ -109,6 +130,7 @@ public int size() { this.inputTimeColumns = inputTimeColumns; this.inputChangelogModes = inputChangelogModes; this.outputChangelogMode = outputChangelogMode; + this.inputUpsertKeys = inputUpsertKeys; } @Override @@ -173,13 +195,18 @@ public Optional getTableSemantics(int pos) { Optional.ofNullable(inputChangelogModes) .map(m -> m.get(tableArgCall.getInputIndex())) .orElse(null); + final int[] upsertKeys = + Optional.ofNullable(inputUpsertKeys) + .map(m -> m.get(tableArgCall.getInputIndex())) + .orElse(new int[0]); return Optional.of( OperatorBindingTableSemantics.create( argumentDataTypes.get(pos), staticArg, tableArgCall, timeColumn, - changelogMode)); + changelogMode, + upsertKeys)); } @Override @@ -283,20 +310,23 @@ private static class OperatorBindingTableSemantics implements TableSemantics { private final SortDirection[] orderByDirections; private final int timeColumn; private final @Nullable ChangelogMode changelogMode; + private final int[] upsertKeyColumns; public static OperatorBindingTableSemantics create( DataType tableDataType, StaticArgument staticArg, RexTableArgCall tableArgCall, int timeColumn, - @Nullable ChangelogMode changelogMode) { + @Nullable ChangelogMode changelogMode, + int[] upsertKeyColumns) { return new OperatorBindingTableSemantics( createDataType(tableDataType, staticArg), tableArgCall.getPartitionKeys(), tableArgCall.getOrderKeys(), RexTableArgCall.toSortDirections(tableArgCall.getSortOrder()), timeColumn, - changelogMode); + changelogMode, + upsertKeyColumns); } private OperatorBindingTableSemantics( @@ -305,13 +335,15 @@ private OperatorBindingTableSemantics( int[] orderByColumns, SortDirection[] orderByDirections, int timeColumn, - @Nullable ChangelogMode changelogMode) { + @Nullable ChangelogMode changelogMode, + int[] upsertKeyColumns) { this.dataType = dataType; this.partitionByColumns = partitionByColumns; this.orderByColumns = orderByColumns; this.orderByDirections = orderByDirections; this.timeColumn = timeColumn; this.changelogMode = changelogMode; + this.upsertKeyColumns = upsertKeyColumns; } private static DataType createDataType(DataType tableDataType, StaticArgument staticArg) { @@ -353,5 +385,10 @@ public int timeColumn() { public Optional changelogMode() { return Optional.ofNullable(changelogMode); } + + @Override + public int[] upsertKeyColumns() { + return upsertKeyColumns; + } } } diff --git a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/stream/StreamExecProcessTableFunction.java b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/stream/StreamExecProcessTableFunction.java index 3973329af7484..ad8fa35553d50 100644 --- a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/stream/StreamExecProcessTableFunction.java +++ b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/stream/StreamExecProcessTableFunction.java @@ -69,6 +69,7 @@ import org.apache.flink.table.types.logical.RowType; import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonCreator; +import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonInclude; import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonProperty; import org.apache.calcite.linq4j.Ord; @@ -108,6 +109,7 @@ public class StreamExecProcessTableFunction extends ExecNodeBase public static final String FIELD_NAME_FUNCTION_CALL = "functionCall"; public static final String FIELD_NAME_INPUT_CHANGELOG_MODES = "inputChangelogModes"; public static final String FIELD_NAME_OUTPUT_CHANGELOG_MODE = "outputChangelogMode"; + public static final String FIELD_NAME_INPUT_UPSERT_KEYS = "inputUpsertKeys"; @JsonProperty(FIELD_NAME_UID) private final @Nullable String uid; @@ -121,6 +123,10 @@ public class StreamExecProcessTableFunction extends ExecNodeBase @JsonProperty(FIELD_NAME_OUTPUT_CHANGELOG_MODE) private final ChangelogMode outputChangelogMode; + @JsonProperty(value = FIELD_NAME_INPUT_UPSERT_KEYS, defaultValue = "[]") + @JsonInclude(JsonInclude.Include.NON_DEFAULT) + private final List inputUpsertKeys; + public StreamExecProcessTableFunction( ReadableConfig tableConfig, List inputProperties, @@ -129,7 +135,8 @@ public StreamExecProcessTableFunction( @Nullable String uid, RexCall invocation, List inputChangelogModes, - ChangelogMode outputChangelogMode) { + ChangelogMode outputChangelogMode, + List inputUpsertKeys) { this( ExecNodeContext.newNodeId(), ExecNodeContext.newContext(StreamExecProcessTableFunction.class), @@ -141,7 +148,8 @@ public StreamExecProcessTableFunction( uid, invocation, inputChangelogModes, - outputChangelogMode); + outputChangelogMode, + inputUpsertKeys); } @JsonCreator @@ -155,7 +163,8 @@ public StreamExecProcessTableFunction( @JsonProperty(FIELD_NAME_UID) @Nullable String uid, @JsonProperty(FIELD_NAME_FUNCTION_CALL) RexNode invocation, @JsonProperty(FIELD_NAME_INPUT_CHANGELOG_MODES) List inputChangelogModes, - @JsonProperty(FIELD_NAME_OUTPUT_CHANGELOG_MODE) ChangelogMode outputChangelogMode) { + @JsonProperty(FIELD_NAME_OUTPUT_CHANGELOG_MODE) ChangelogMode outputChangelogMode, + @JsonProperty(FIELD_NAME_INPUT_UPSERT_KEYS) @Nullable List inputUpsertKeys) { super(id, context, persistedConfig, inputProperties, outputType, description); this.uid = uid; // Mirror the FlinkLogicalTableFunctionScan converter for the compiled-plan restore path: @@ -164,6 +173,12 @@ public StreamExecProcessTableFunction( this.invocation = BridgingSqlFunction.resolveCallTraits((RexCall) invocation); this.inputChangelogModes = inputChangelogModes; this.outputChangelogMode = outputChangelogMode; + this.inputUpsertKeys = + inputUpsertKeys != null + ? inputUpsertKeys + : inputProperties.stream() + .map(p -> new int[0]) + .collect(Collectors.toList()); } public @Nullable String getUid() { @@ -202,7 +217,12 @@ protected Transformation translateToPlanInternal( final RexCall udfCall = StreamPhysicalProcessTableFunction.toUdfCall(invocation); final GeneratedRunnerResult generated = ProcessTableRunnerGenerator.generate( - ctx, udfCall, inputTimeColumns, inputChangelogModes, outputChangelogMode); + ctx, + udfCall, + inputTimeColumns, + inputChangelogModes, + outputChangelogMode, + inputUpsertKeys); final GeneratedProcessTableRunner generatedRunner = generated.runner(); final LinkedHashMap stateInfos = generated.stateInfos(); @@ -310,9 +330,12 @@ private RuntimeTableSemantics createRuntimeTableSemantics( final int timeColumn = inputTimeColumns.get(tableArgCall.getInputIndex()); + final int inputIndex = tableArgCall.getInputIndex(); + final int[] upsertKeys = + inputIndex < inputUpsertKeys.size() ? inputUpsertKeys.get(inputIndex) : new int[0]; return new RuntimeTableSemantics( tableArg.getName(), - tableArgCall.getInputIndex(), + inputIndex, dataType, tableArgCall.getPartitionKeys(), tableArgCall.getOrderKeys(), @@ -320,7 +343,8 @@ private RuntimeTableSemantics createRuntimeTableSemantics( consumedChangelogMode, tableArg.is(StaticArgumentTrait.PASS_COLUMNS_THROUGH), tableArg.is(StaticArgumentTrait.SET_SEMANTIC_TABLE), - timeColumn); + timeColumn, + upsertKeys); } private Transformation createKeyedTransformation( diff --git a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/physical/stream/StreamPhysicalProcessTableFunction.java b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/physical/stream/StreamPhysicalProcessTableFunction.java index 5ccecf18e71be..d08b0854c46af 100644 --- a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/physical/stream/StreamPhysicalProcessTableFunction.java +++ b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/physical/stream/StreamPhysicalProcessTableFunction.java @@ -27,11 +27,13 @@ import org.apache.flink.table.planner.calcite.FlinkTypeFactory; import org.apache.flink.table.planner.calcite.RexTableArgCall; import org.apache.flink.table.planner.functions.bridging.BridgingSqlFunction; +import org.apache.flink.table.planner.plan.metadata.FlinkRelMetadataQuery; import org.apache.flink.table.planner.plan.nodes.exec.ExecNode; import org.apache.flink.table.planner.plan.nodes.exec.InputProperty; import org.apache.flink.table.planner.plan.nodes.exec.stream.StreamExecProcessTableFunction; import org.apache.flink.table.planner.plan.nodes.logical.FlinkLogicalTableFunctionScan; import org.apache.flink.table.planner.plan.utils.ChangelogPlanUtils; +import org.apache.flink.table.planner.plan.utils.UpsertKeyUtil; import org.apache.flink.table.planner.utils.JavaScalaConversionUtil; import org.apache.flink.table.planner.utils.ShortcutUtils; import org.apache.flink.table.types.inference.StaticArgument; @@ -165,6 +167,15 @@ public ExecNode translateToExecNode() { verifyTimeAttributes(getInputs(), call, inputChangelogModes, outputChangelogMode); final List> providedInputArgs = getProvidedInputArgs(call); verifyPassThroughColumnsForUpdates(providedInputArgs, outputChangelogMode); + final FlinkRelMetadataQuery fmq = + FlinkRelMetadataQuery.reuseOrCreate(getCluster().getMetadataQuery()); + final List inputUpsertKeys = + getInputs().stream() + .map( + input -> + UpsertKeyUtil.smallestKey(fmq.getUpsertKeys(input)) + .orElse(new int[0])) + .collect(Collectors.toList()); return new StreamExecProcessTableFunction( unwrapTableConfig(this), getInputs().stream().map(i -> InputProperty.DEFAULT).collect(Collectors.toList()), @@ -173,7 +184,8 @@ public ExecNode translateToExecNode() { uid, call, inputChangelogModes, - outputChangelogMode); + outputChangelogMode, + inputUpsertKeys); } @Override diff --git a/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/codegen/ProcessTableRunnerGenerator.scala b/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/codegen/ProcessTableRunnerGenerator.scala index 52df803d5c8f8..402157aa0fdb6 100644 --- a/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/codegen/ProcessTableRunnerGenerator.scala +++ b/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/codegen/ProcessTableRunnerGenerator.scala @@ -65,7 +65,8 @@ object ProcessTableRunnerGenerator { udfCall: RexCall, inputTimeColumns: java.util.List[Integer], inputChangelogModes: java.util.List[ChangelogMode], - outputChangelogMode: ChangelogMode): GeneratedRunnerResult = { + outputChangelogMode: ChangelogMode, + inputUpsertKeys: java.util.List[Array[Int]]): GeneratedRunnerResult = { val function: BridgingSqlFunction = udfCall.getOperator.asInstanceOf[BridgingSqlFunction] val definition: FunctionDefinition = function.getDefinition val dataTypeFactory = function.getDataTypeFactory @@ -77,7 +78,12 @@ object ProcessTableRunnerGenerator { // Thus, functions can reconfigure themselves for the exact use case. // Including updating their state layout. val callContext = - function.toCallContext(udfCall, inputTimeColumns, inputChangelogModes, outputChangelogMode) + function.toCallContext( + udfCall, + inputTimeColumns, + inputChangelogModes, + outputChangelogMode, + inputUpsertKeys) // Create the final UDF for runtime val udf = UserDefinedFunctionHelper.createSpecializedFunction( diff --git a/flink-table/flink-table-planner/src/test/resources/restore-tests/stream-exec-process-table-function_1/to-changelog-retract-restore/plan/to-changelog-retract-restore.json b/flink-table/flink-table-planner/src/test/resources/restore-tests/stream-exec-process-table-function_1/to-changelog-retract-restore/plan/to-changelog-retract-restore.json index ea961c665ca08..07684a9f095ec 100644 --- a/flink-table/flink-table-planner/src/test/resources/restore-tests/stream-exec-process-table-function_1/to-changelog-retract-restore/plan/to-changelog-retract-restore.json +++ b/flink-table/flink-table-planner/src/test/resources/restore-tests/stream-exec-process-table-function_1/to-changelog-retract-restore/plan/to-changelog-retract-restore.json @@ -78,7 +78,8 @@ "type" : "ROW<`op` VARCHAR(2147483647), `name` VARCHAR(2147483647) NOT NULL, `score` BIGINT> NOT NULL" }, "inputChangelogModes" : [ [ "INSERT", "UPDATE_BEFORE", "UPDATE_AFTER", "DELETE" ] ], - "outputChangelogMode" : [ "INSERT" ] + "outputChangelogMode" : [ "INSERT" ], + "inputUpsertKeys" : [ [ 0 ] ] }, { "id" : 3, "type" : "stream-exec-sink_2", diff --git a/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/process/RuntimeTableSemantics.java b/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/process/RuntimeTableSemantics.java index cabab4c613143..7d0c5cf5863ae 100644 --- a/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/process/RuntimeTableSemantics.java +++ b/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/process/RuntimeTableSemantics.java @@ -44,6 +44,7 @@ public class RuntimeTableSemantics implements TableSemantics, Serializable { private final boolean passColumnsThrough; private final boolean hasSetSemantics; private final int timeColumn; + private final int[] upsertKeyColumns; private transient ChangelogMode changelogMode; @@ -57,7 +58,8 @@ public RuntimeTableSemantics( RuntimeChangelogMode consumedChangelogMode, boolean passColumnsThrough, boolean hasSetSemantics, - int timeColumn) { + int timeColumn, + int[] upsertKeyColumns) { this.argName = argName; this.inputIndex = inputIndex; this.dataType = dataType; @@ -68,6 +70,7 @@ public RuntimeTableSemantics( this.passColumnsThrough = passColumnsThrough; this.hasSetSemantics = hasSetSemantics; this.timeColumn = timeColumn; + this.upsertKeyColumns = upsertKeyColumns; } public String getArgName() { @@ -122,4 +125,9 @@ public int timeColumn() { public Optional changelogMode() { return Optional.of(getChangelogMode()); } + + @Override + public int[] upsertKeyColumns() { + return upsertKeyColumns; + } } diff --git a/flink-table/flink-table-runtime/src/test/java/org/apache/flink/table/runtime/operators/process/ProcessSetTableOperatorInterruptibleTimersTest.java b/flink-table/flink-table-runtime/src/test/java/org/apache/flink/table/runtime/operators/process/ProcessSetTableOperatorInterruptibleTimersTest.java index be390ab5f5557..38a0df4811168 100644 --- a/flink-table/flink-table-runtime/src/test/java/org/apache/flink/table/runtime/operators/process/ProcessSetTableOperatorInterruptibleTimersTest.java +++ b/flink-table/flink-table-runtime/src/test/java/org/apache/flink/table/runtime/operators/process/ProcessSetTableOperatorInterruptibleTimersTest.java @@ -246,7 +246,8 @@ private static RuntimeTableSemantics tableSemantics() { RuntimeChangelogMode.serialize(ChangelogMode.insertOnly()), /* passColumnsThrough */ false, /* hasSetSemantics */ true, - /* timeColumn */ 1); + /* timeColumn */ 1, + /* upsertKeyColumns */ new int[0]); } // -------------------------------------------------------------------------------------------- diff --git a/flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/TestHarnessTableSemantics.java b/flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/TestHarnessTableSemantics.java index fadf21d7dd942..c55e715c065c4 100644 --- a/flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/TestHarnessTableSemantics.java +++ b/flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/TestHarnessTableSemantics.java @@ -30,10 +30,16 @@ class TestHarnessTableSemantics implements TableSemantics { private final DataType dataType; private final int[] partitionByColumns; + private final int[] upsertKeyColumns; TestHarnessTableSemantics(DataType dataType, int[] partitionByColumns) { + this(dataType, partitionByColumns, new int[0]); + } + + TestHarnessTableSemantics(DataType dataType, int[] partitionByColumns, int[] upsertKeyColumns) { this.dataType = dataType; this.partitionByColumns = partitionByColumns; + this.upsertKeyColumns = upsertKeyColumns; } @Override @@ -65,4 +71,9 @@ public int timeColumn() { public Optional changelogMode() { return Optional.empty(); } + + @Override + public int[] upsertKeyColumns() { + return upsertKeyColumns; + } } From ed6fd23944be012a61a1d22913a3f6b81a8ef38e Mon Sep 17 00:00:00 2001 From: Ramin Gharib Date: Tue, 26 May 2026 12:59:44 +0200 Subject: [PATCH 4/8] [FLINK-39636][table] Emit partial DELETE rows in TO_CHANGELOG via upsert key Consumes the new TableSemantics.upsertKeyColumns() in ToChangelogFunction to emit partial DELETE rows in row semantics when the input declares an upsert key. Identifying columns are preserved on the DELETE row, non-key columns are emitted as null. Without an upsert key the function still passes the input through unchanged. [hotfix][table] Use generic Map for ToChangelogTypeStrategy op_mapping Drops raw Map types and the 'rawtypes' suppressions in validateOpMappingKeys and mapsDelete by passing the resolved op_mapping argument as Map. Also extracts the literal 'DELETE' string into a constant derived from RowKind.DELETE.name(). --- .../docs/sql/reference/queries/changelog.md | 23 ++++-- flink-python/pyflink/table/table.py | 6 +- .../org/apache/flink/table/api/Table.java | 6 +- .../strategies/ToChangelogTypeStrategy.java | 37 ++++----- .../ToChangelogInputTypeStrategyTest.java | 12 +-- .../exec/stream/ToChangelogSemanticTests.java | 1 + .../exec/stream/ToChangelogTestPrograms.java | 33 ++++++-- .../functions/ptf/ToChangelogFunction.java | 79 ++++++++++++++++--- 8 files changed, 138 insertions(+), 59 deletions(-) diff --git a/docs/content/docs/sql/reference/queries/changelog.md b/docs/content/docs/sql/reference/queries/changelog.md index 7a1598ee850fb..9b3a6fe4f5645 100644 --- a/docs/content/docs/sql/reference/queries/changelog.md +++ b/docs/content/docs/sql/reference/queries/changelog.md @@ -308,7 +308,7 @@ SELECT * FROM TO_CHANGELOG( | `input` | Yes | The input table. With `PARTITION BY`, rows with the same key are co-located and run in the same operator instance. Without `PARTITION BY`, each row is processed independently. Accepts insert-only, retract, and upsert tables. For upsert tables, the provided `PARTITION BY` key should match or be a subset of the upsert key of the subquery. | | `op` | No | A `DESCRIPTOR` with a single column name for the operation code column. Defaults to `op`. | | `op_mapping` | No | A `MAP` mapping change operation names to custom output codes. Keys can contain comma-separated names to map multiple operations to the same code (e.g., `'INSERT, UPDATE_AFTER'`). When provided, only mapped operations are forwarded - unmapped events are dropped. Each change operation may appear at most once across all entries. | -| `produces_full_deletes` | No | A `BOOLEAN` literal that controls how DELETE rows are emitted. When `true`, the function requires fully-populated DELETE rows from the input. The planner inserts a `ChangelogNormalize` operator for upsert sources that emit key-only deletes, so downstream sees the full pre-image on DELETE. When `false` (default), no full-delete requirement is enforced. Partial DELETE rows from the input pass through unchanged. With `PARTITION BY` (set semantics), the function additionally nulls non-partition-key columns on DELETE rows even when the input row is fully populated, so the output always carries only the key on DELETE. | +| `produces_full_deletes` | No | A `BOOLEAN` literal that controls how DELETE rows are emitted. When `true`, the function requires fully-populated DELETE rows from the input. The planner inserts a `ChangelogNormalize` operator for upsert sources that emit key-only deletes, so downstream sees the full pre-image on DELETE. When `false` (default), no full-delete requirement is enforced. In row semantics, the function preserves the planner-derived upsert key columns of the input on DELETE rows and nulls the rest (or passes the input through unchanged when no upsert key is derivable). With `PARTITION BY` (set semantics), the function additionally nulls non-partition-key columns on DELETE rows even when the input row is fully populated, so the output always carries only the key on DELETE. | #### Default op_mapping @@ -417,15 +417,23 @@ SELECT * FROM TO_CHANGELOG( **With `produces_full_deletes => false` (default).** The planner does not require fully-populated DELETE rows on the input. For upsert sources that emit key-only deletes (e.g. Kafka compacted topics), this avoids the stateful `ChangelogNormalize` operator that would otherwise materialize the full pre-image of each deleted row. -In **row semantics** (no `PARTITION BY`) the function passes the input row through unchanged. If the source emits partial DELETE rows they remain partial downstream; if it emits full DELETE rows they remain full. +In **row semantics** (no `PARTITION BY`), the function preserves the upsert key columns of the input on DELETE rows and nulls the rest. The upsert key is the one the planner already derives from the input (typically a declared `PRIMARY KEY`), so no `PARTITION BY` is required to get partial DELETEs. ```sql --- Source emits -D[id:5] (key-only). +-- Source 't' declares PRIMARY KEY (id). Source emits -D[id:5, name:'Alice'] (full pre-image). +-- Output: +I[op:'DELETE', id:5, name:null] +SELECT * FROM TO_CHANGELOG(input => TABLE t) +``` + +When the input has no derivable upsert key (e.g. a pure append-only source, or an upstream operator that erased the key), there is no identifying column to preserve. The function then passes the input through unchanged. + +```sql +-- Source emits -D[id:5] (key-only, no declared key). -- Output: +I[op:'DELETE', id:5, name:null] SELECT * FROM TO_CHANGELOG(input => TABLE upsert_source) ``` -In **set semantics** (`PARTITION BY`) the function additionally nulls every non-partition-key column on DELETE rows. This forces the output to carry only the partition key on DELETE even when the input row was fully populated, which matches the shape expected by upsert sinks and Kafka compacted topics. +In **set semantics** (`PARTITION BY`), the function nulls every non-partition-key column on DELETE rows. This forces the output to carry only the partition key on DELETE even when the input row was fully populated, which matches the shape expected by upsert sinks and Kafka compacted topics. ```sql -- Source emits -D[id:5, name:'Alice'] (full pre-image, e.g. from a retract source). @@ -433,8 +441,6 @@ In **set semantics** (`PARTITION BY`) the function additionally nulls every non- SELECT * FROM TO_CHANGELOG(input => TABLE retract_source PARTITION BY id) ``` -There is no way to derive a partial DELETE in row semantics when the input emits a full pre-image, since the function has no key column to preserve. Use `PARTITION BY` for that case. - #### Partitioning by a key ```sql @@ -474,8 +480,9 @@ Table result = myTable.toChangelog( // Require fully-populated DELETE rows from the input (inserts a ChangelogNormalize for // upsert sources). When false (default), no full-delete requirement is enforced; in row -// semantics the input passes through unchanged, in set semantics non-partition-key columns -// are nulled on DELETE. +// semantics the function preserves the input's upsert key columns on DELETE and nulls the +// rest (passes the input through unchanged when no upsert key is derivable). In set +// semantics non-partition-key columns are nulled on DELETE. Table result = myTable.toChangelog( lit(true).asArgument("produces_full_deletes") ); diff --git a/flink-python/pyflink/table/table.py b/flink-python/pyflink/table/table.py index bb06642db9e12..45f3d8bf0e729 100644 --- a/flink-python/pyflink/table/table.py +++ b/flink-python/pyflink/table/table.py @@ -1200,8 +1200,10 @@ def to_changelog(self, *arguments: Expression) -> 'Table': emitted. When ``True``, the planner inserts a ``ChangelogNormalize`` operator for upsert sources that emit key-only deletes so the function emits fully populated DELETE rows downstream. When ``False`` (default), no full-delete - requirement is enforced. In row semantics the input is passed through unchanged, - and in set semantics (``PARTITION BY``) non-partition-key columns are nulled on + requirement is enforced. In row semantics the function preserves the + planner-derived upsert key columns of the input on DELETE rows and nulls the + rest, or passes the input through unchanged when no upsert key is derivable. + In set semantics (``PARTITION BY``) non-partition-key columns are nulled on DELETE rows. Example: diff --git a/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/Table.java b/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/Table.java index de6ce9d3e5ecf..d5f0ab6cb8f94 100644 --- a/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/Table.java +++ b/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/Table.java @@ -1461,8 +1461,10 @@ default TableResult executeInsert( * ); * * // Require fully-populated DELETE rows from the input (inserts a ChangelogNormalize for - * // upsert sources). When false (default), no full-delete requirement is enforced and partial - * // DELETE rows from the input pass through unchanged. + * // upsert sources). When false (default), no full-delete requirement is enforced; in row + * // semantics the function preserves the planner-derived upsert key columns of the input on + * // DELETE rows and nulls the rest, or passes the input through unchanged when no upsert key + * // is derivable. * Table result = table.toChangelog( * lit(true).asArgument("produces_full_deletes") * ); diff --git a/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/ToChangelogTypeStrategy.java b/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/ToChangelogTypeStrategy.java index d406875aa2fec..a9f9d8a38931e 100644 --- a/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/ToChangelogTypeStrategy.java +++ b/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/ToChangelogTypeStrategy.java @@ -28,6 +28,7 @@ import org.apache.flink.table.types.inference.InputTypeStrategy; import org.apache.flink.table.types.inference.TypeStrategy; import org.apache.flink.types.ColumnList; +import org.apache.flink.types.RowKind; import java.util.ArrayList; import java.util.Arrays; @@ -52,6 +53,8 @@ public final class ToChangelogTypeStrategy { private static final Set VALID_ROW_KIND_NAMES = Set.of("INSERT", "UPDATE_BEFORE", "UPDATE_AFTER", "DELETE"); + private static final String DELETE = RowKind.DELETE.name(); + // -------------------------------------------------------------------------------------------- // Input validation // -------------------------------------------------------------------------------------------- @@ -152,7 +155,8 @@ private static Optional> validateOpMapping( final Optional opMapping = callContext.getArgumentValue(ARG_OP_MAPPING, Map.class); if (opMapping.isPresent()) { - return validateOpMappingKeys(callContext, opMapping.get(), throwOnFailure); + return validateOpMappingKeys( + callContext, (Map) opMapping.get(), throwOnFailure); } return Optional.empty(); } @@ -163,16 +167,13 @@ private static Optional> validateOpMapping( * trimmed. Names are case-sensitive and must match exactly (e.g., {@code INSERT}, not {@code * insert}). Each name must be valid and appear at most once across all entries. */ - @SuppressWarnings("rawtypes") private static Optional> validateOpMappingKeys( - final CallContext callContext, final Map opMapping, final boolean throwOnFailure) { + final CallContext callContext, + final Map opMapping, + final boolean throwOnFailure) { final Set allRowKindsSeen = new HashSet<>(); - for (final Object key : opMapping.keySet()) { - if (!(key instanceof String)) { - return callContext.fail( - throwOnFailure, "Invalid target mapping for argument 'op_mapping'."); - } - final String[] rowKindNames = ((String) key).split(","); + for (final String key : opMapping.keySet()) { + final String[] rowKindNames = key.split(","); for (final String rawName : rowKindNames) { final String rowKindName = rawName.trim(); if (!VALID_ROW_KIND_NAMES.contains(rowKindName)) { @@ -217,7 +218,7 @@ private static Optional> validateProducesFullDeletes( // TableSemantics#changelogMode() returns empty here at type-inference time. The mapping // check below only needs the literal op_mapping argument, so it lives here. final Optional opMapping = callContext.getArgumentValue(ARG_OP_MAPPING, Map.class); - if (opMapping.isPresent() && !mapsDelete(opMapping.get())) { + if (opMapping.isPresent() && !mapsDelete((Map) opMapping.get())) { return callContext.fail( throwOnFailure, "Invalid 'produces_full_deletes' for TO_CHANGELOG: the active 'op_mapping' " @@ -229,17 +230,13 @@ private static Optional> validateProducesFullDeletes( } /** - * Returns {@code true} when at least one {@code op_mapping} key references {@code DELETE}. - * Keys may be comma-separated (e.g., {@code "INSERT, DELETE"}) per the user-facing contract. + * Returns {@code true} when at least one {@code op_mapping} key references {@code DELETE}. Keys + * may be comma-separated (e.g., {@code "INSERT, DELETE"}) per the user-facing contract. */ - @SuppressWarnings("rawtypes") - private static boolean mapsDelete(final Map opMapping) { - for (final Object key : opMapping.keySet()) { - if (!(key instanceof String)) { - continue; - } - for (final String rawName : ((String) key).split(",")) { - if ("DELETE".equals(rawName.trim())) { + private static boolean mapsDelete(final Map opMapping) { + for (final String key : opMapping.keySet()) { + for (final String rawName : key.split(",")) { + if (DELETE.equals(rawName.trim())) { return true; } } diff --git a/flink-table/flink-table-common/src/test/java/org/apache/flink/table/types/inference/strategies/ToChangelogInputTypeStrategyTest.java b/flink-table/flink-table-common/src/test/java/org/apache/flink/table/types/inference/strategies/ToChangelogInputTypeStrategyTest.java index e9b20a7a5ba02..fb1cb918997d6 100644 --- a/flink-table/flink-table-common/src/test/java/org/apache/flink/table/types/inference/strategies/ToChangelogInputTypeStrategyTest.java +++ b/flink-table/flink-table-common/src/test/java/org/apache/flink/table/types/inference/strategies/ToChangelogInputTypeStrategyTest.java @@ -56,8 +56,7 @@ protected Stream testData() { .calledWithLiteralAt(1, ColumnList.of("op")) .calledWithLiteralAt(2, null) .calledWithLiteralAt(3, true) - .expectArgumentTypes( - TABLE_TYPE, DESCRIPTOR_TYPE, MAP_TYPE, BOOLEAN_TYPE), + .expectArgumentTypes(TABLE_TYPE, DESCRIPTOR_TYPE, MAP_TYPE, BOOLEAN_TYPE), // Valid: produces_full_deletes=true with op_mapping that includes DELETE TestSpec.forStrategy( @@ -69,8 +68,7 @@ protected Stream testData() { .calledWithLiteralAt(1, ColumnList.of("op")) .calledWithLiteralAt(2, Map.of("INSERT", "I", "DELETE", "D")) .calledWithLiteralAt(3, true) - .expectArgumentTypes( - TABLE_TYPE, DESCRIPTOR_TYPE, MAP_TYPE, BOOLEAN_TYPE), + .expectArgumentTypes(TABLE_TYPE, DESCRIPTOR_TYPE, MAP_TYPE, BOOLEAN_TYPE), // Valid: produces_full_deletes=true with comma-separated DELETE key TestSpec.forStrategy( @@ -82,8 +80,7 @@ protected Stream testData() { .calledWithLiteralAt(1, ColumnList.of("op")) .calledWithLiteralAt(2, Map.of("INSERT, DELETE", "X")) .calledWithLiteralAt(3, true) - .expectArgumentTypes( - TABLE_TYPE, DESCRIPTOR_TYPE, MAP_TYPE, BOOLEAN_TYPE), + .expectArgumentTypes(TABLE_TYPE, DESCRIPTOR_TYPE, MAP_TYPE, BOOLEAN_TYPE), // Valid: produces_full_deletes=false with op_mapping that omits DELETE TestSpec.forStrategy( @@ -95,8 +92,7 @@ protected Stream testData() { .calledWithLiteralAt(1, ColumnList.of("op")) .calledWithLiteralAt(2, Map.of("INSERT, UPDATE_AFTER", "X")) .calledWithLiteralAt(3, false) - .expectArgumentTypes( - TABLE_TYPE, DESCRIPTOR_TYPE, MAP_TYPE, BOOLEAN_TYPE), + .expectArgumentTypes(TABLE_TYPE, DESCRIPTOR_TYPE, MAP_TYPE, BOOLEAN_TYPE), // Error: produces_full_deletes=true with op_mapping that strips DELETE TestSpec.forStrategy( diff --git a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ToChangelogSemanticTests.java b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ToChangelogSemanticTests.java index b3376b3a34677..4233dad198124 100644 --- a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ToChangelogSemanticTests.java +++ b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ToChangelogSemanticTests.java @@ -59,6 +59,7 @@ public List programs() { ToChangelogTestPrograms.PRODUCES_FULL_DELETES_ON_APPEND_ONLY_INPUT, ToChangelogTestPrograms.ROW_SEM_PARTIAL_DELETES, ToChangelogTestPrograms.ROW_SEM_FORCE_FULL_DELETES, + ToChangelogTestPrograms.ROW_SEM_PARTIAL_DELETES_VIA_UPSERT_KEY, ToChangelogTestPrograms.SET_SEM_PARTIAL_DELETES, ToChangelogTestPrograms.SET_SEM_FULL_DELETES, ToChangelogTestPrograms.SET_SEM_FORCE_FULL_DELETES, diff --git a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ToChangelogTestPrograms.java b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ToChangelogTestPrograms.java index 5126014b9a36a..996fb4fc6113e 100644 --- a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ToChangelogTestPrograms.java +++ b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ToChangelogTestPrograms.java @@ -86,7 +86,7 @@ public class ToChangelogTestPrograms { "+I[INSERT, Bob, 20]", "+I[UPDATE_BEFORE, Alice, 10]", "+I[UPDATE_AFTER, Alice, 30]", - "+I[DELETE, Bob, 20]") + "+I[DELETE, Bob, null]") .build()) .runSql("INSERT INTO sink SELECT * FROM TO_CHANGELOG(input => TABLE t)") .build(); @@ -118,7 +118,7 @@ public class ToChangelogTestPrograms { .consumedAfterRestore( "+I[UPDATE_BEFORE, Alice, 10]", "+I[UPDATE_AFTER, Alice, 30]", - "+I[DELETE, Bob, 20]") + "+I[DELETE, Bob, null]") .build()) .runSql("INSERT INTO sink SELECT * FROM TO_CHANGELOG(input => TABLE t)") .build(); @@ -157,7 +157,7 @@ public class ToChangelogTestPrograms { public static final TableTestProgram UPSERT = TableTestProgram.of( "to-changelog-upsert-input", - "upsert input in row semantics gets ChangelogNormalize for UPDATE_BEFORE and emits full deletes") + "upsert input in row semantics gets ChangelogNormalize for UPDATE_BEFORE and emits partial deletes") .setupTableSource( SourceTestStep.newBuilder("t") .addSchema( @@ -179,7 +179,7 @@ public class ToChangelogTestPrograms { "+I[INSERT, Bob, 20]", "+I[UPDATE_BEFORE, Alice, 10]", "+I[UPDATE_AFTER, Alice, 30]", - "+I[DELETE, Bob, 20]") + "+I[DELETE, Bob, null]") .build()) .runSql("INSERT INTO sink SELECT * FROM TO_CHANGELOG(input => TABLE t)") .build(); @@ -535,7 +535,7 @@ public class ToChangelogTestPrograms { "+I[false, Alice, 10]", "+I[false, Bob, 20]", "+I[false, Alice, 30]", - "+I[true, Bob, 20]") + "+I[true, Bob, null]") .build()) .runSql( "INSERT INTO sink SELECT * FROM TO_CHANGELOG(" @@ -613,7 +613,6 @@ public class ToChangelogTestPrograms { "the input table only produces [INSERT] and never emits DELETE rows") .build(); - // -------------------------------------------------------------------------------------------- // Row semantics x delete handling matrix // -------------------------------------------------------------------------------------------- @@ -676,6 +675,28 @@ public class ToChangelogTestPrograms { + "produces_full_deletes => true)") .build(); + public static final TableTestProgram ROW_SEM_PARTIAL_DELETES_VIA_UPSERT_KEY = + TableTestProgram.of( + "to-changelog-row-sem-partial-deletes-via-upsert-key", + "row semantics with single-column upsert key: DELETE preserves the key column and nulls the rest") + .setupTableSource( + SourceTestStep.newBuilder("t") + .addSchema( + "name STRING PRIMARY KEY NOT ENFORCED", "score BIGINT") + .addMode(ChangelogMode.upsert()) + .producedValues( + Row.ofKind(RowKind.INSERT, "Alice", 10L), + Row.ofKind(RowKind.DELETE, "Alice", 10L)) + .build()) + .setupTableSink( + SinkTestStep.newBuilder("sink") + .addSchema("op STRING", "name STRING", "score BIGINT") + .consumedValues( + "+I[INSERT, Alice, 10]", "+I[DELETE, Alice, null]") + .build()) + .runSql("INSERT INTO sink SELECT * FROM TO_CHANGELOG(input => TABLE t)") + .build(); + // -------------------------------------------------------------------------------------------- // Set semantics x delete handling matrix // -------------------------------------------------------------------------------------------- diff --git a/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/ptf/ToChangelogFunction.java b/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/ptf/ToChangelogFunction.java index d8874a2be1695..e7e048c8a6bd2 100644 --- a/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/ptf/ToChangelogFunction.java +++ b/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/ptf/ToChangelogFunction.java @@ -33,21 +33,26 @@ import org.apache.flink.table.functions.TableSemantics; import org.apache.flink.table.types.inference.CallContext; import org.apache.flink.table.types.inference.strategies.ChangelogTypeStrategyUtils; +import org.apache.flink.table.types.logical.LogicalType; +import org.apache.flink.table.types.logical.RowType; import org.apache.flink.types.ColumnList; import org.apache.flink.types.RowKind; import javax.annotation.Nullable; import java.util.EnumMap; +import java.util.HashSet; import java.util.List; import java.util.Map; +import java.util.Set; import java.util.stream.Collectors; /** * Runtime implementation of {@link BuiltInFunctionDefinitions#TO_CHANGELOG}. * - *

Converts each input row into an INSERT-only output row with an operation code column. The - * output schema is {@code [op_column, ...all_input_columns...]}. + *

Converts each input row into an INSERT-only output row with an operation code column. Output + * schema is {@code [op_column, ...projected_input_columns...]}. Partition columns are prepended by + * the framework outside this function and are not part of the projection. * *

Uses {@link JoinedRowData} to combine the op column with the full input row. */ @@ -65,13 +70,16 @@ public class ToChangelogFunction extends BuiltInProcessTableFunction { private final Map rawOpMap; private final int[] outputIndices; + private final RowType inputRowType; private final boolean producesFullDelete; + private final boolean[] preserveOnDelete; private transient Map opMap; private transient GenericRowData opRow; private transient JoinedRowData output; private transient ProjectedRowData projectedOutput; - private transient GenericRowData nullPayloadRow; + private transient GenericRowData partialDeletePayload; + private transient RowData.FieldGetter[] preservedFieldGetters; @SuppressWarnings("unchecked") public ToChangelogFunction(final SpecializedContext context) { @@ -91,7 +99,10 @@ public ToChangelogFunction(final SpecializedContext context) { validateProducesFullDeletes(producesFullDeletesArg, tableSemantics); this.outputIndices = ChangelogTypeStrategyUtils.computeOutputIndices(tableSemantics); + this.inputRowType = (RowType) tableSemantics.dataType().getLogicalType(); + final int[] upsertKeys = tableSemantics.upsertKeyColumns(); this.producesFullDelete = resolveProducesFullDelete(producesFullDeletesArg, tableSemantics); + this.preserveOnDelete = computePreserveOnDelete(this.outputIndices, upsertKeys); } /** @@ -100,8 +111,9 @@ public ToChangelogFunction(final SpecializedContext context) { * *

The framework prepends partition-key columns to the output without consulting this * function, so in set semantics partition keys are preserved on DELETE rows for free. In row - * semantics there is no key column to preserve, so the function passes the input through - * unchanged regardless of {@code produces_full_deletes}. + * semantics we rely on the input table's upsert key to identify columns. Without either signal + * we cannot null anything safely and fall back to passing the input through unchanged, which is + * equivalent to a full delete. */ private static boolean resolveProducesFullDelete( final boolean producesFullDeletesArg, final TableSemantics tableSemantics) { @@ -109,7 +121,21 @@ private static boolean resolveProducesFullDelete( return true; } final boolean hasPartitionBy = tableSemantics.partitionByColumns().length > 0; - return !hasPartitionBy; + final boolean hasUpsertKey = tableSemantics.upsertKeyColumns().length > 0; + return !hasPartitionBy && !hasUpsertKey; + } + + private static boolean[] computePreserveOnDelete( + final int[] outputIndices, final int[] upsertKeys) { + final Set keepInputIndices = new HashSet<>(); + for (final int key : upsertKeys) { + keepInputIndices.add(key); + } + final boolean[] preserve = new boolean[outputIndices.length]; + for (int i = 0; i < outputIndices.length; i++) { + preserve[i] = keepInputIndices.contains(outputIndices[i]); + } + return preserve; } @Override @@ -120,7 +146,16 @@ public void open(final FunctionContext context) throws Exception { opRow = new GenericRowData(1); output = new JoinedRowData(); projectedOutput = ProjectedRowData.from(outputIndices); - nullPayloadRow = new GenericRowData(outputIndices.length); + partialDeletePayload = new GenericRowData(outputIndices.length); + preservedFieldGetters = new RowData.FieldGetter[outputIndices.length]; + final List inputFieldTypes = inputRowType.getChildren(); + for (int i = 0; i < outputIndices.length; i++) { + if (preserveOnDelete[i]) { + preservedFieldGetters[i] = + RowData.createFieldGetter( + inputFieldTypes.get(outputIndices[i]), outputIndices[i]); + } + } } /** @@ -176,8 +211,8 @@ private static void validateOpMap( * *

Lives here rather than in the input type strategy because {@link * TableSemantics#changelogMode()} returns empty during type inference and is only populated at - * specialization time. The complementary check against the literal {@code op_mapping} - * argument runs earlier in {@code ToChangelogTypeStrategy}. + * specialization time. The complementary check against the literal {@code op_mapping} argument + * runs earlier in {@code ToChangelogTypeStrategy}. */ private static void validateProducesFullDeletes( final boolean producesFullDeletesArg, final TableSemantics tableSemantics) { @@ -210,10 +245,28 @@ public void eval( } opRow.setField(0, opCode); - final RowData payload = - (input.getRowKind() == RowKind.DELETE && !producesFullDelete) - ? nullPayloadRow - : projectedOutput.replaceRow(input); + final RowData payload; + if (input.getRowKind() == RowKind.DELETE && !producesFullDelete) { + payload = buildPartialDeletePayload(input); + } else { + payload = projectedOutput.replaceRow(input); + } collect(output.replace(opRow, payload)); } + + /** + * Builds the payload for a partial DELETE row: upsert-key columns are copied from the input, + * all other columns are emitted as {@code null}. Partition-key columns are not included here + * since the framework prepends them outside the function's projected output. + */ + private RowData buildPartialDeletePayload(final RowData input) { + for (int i = 0; i < outputIndices.length; i++) { + if (preserveOnDelete[i]) { + partialDeletePayload.setField(i, preservedFieldGetters[i].getFieldOrNull(input)); + } else { + partialDeletePayload.setField(i, null); + } + } + return partialDeletePayload; + } } From 06dbddc8458db45c5344c6f0514582325e5303e0 Mon Sep 17 00:00:00 2001 From: Ramin Gharib Date: Wed, 27 May 2026 10:08:27 +0200 Subject: [PATCH 5/8] [FLINK-39636][table] Make prduces_full_deletes=true by default --- .../docs/sql/reference/queries/changelog.md | 66 ++++++++---- flink-python/pyflink/table/table.py | 21 ++-- .../flink/table/api/PartitionedTable.java | 9 +- .../org/apache/flink/table/api/Table.java | 12 +-- .../functions/BuiltInFunctionDefinitions.java | 20 ++-- .../flink/table/functions/TableSemantics.java | 6 ++ .../strategies/ToChangelogTypeStrategy.java | 15 +-- .../exec/stream/ToChangelogTestPrograms.java | 19 ++-- .../functions/ptf/ToChangelogFunction.java | 100 +++++++++--------- 9 files changed, 160 insertions(+), 108 deletions(-) diff --git a/docs/content/docs/sql/reference/queries/changelog.md b/docs/content/docs/sql/reference/queries/changelog.md index 9b3a6fe4f5645..425062edd17f2 100644 --- a/docs/content/docs/sql/reference/queries/changelog.md +++ b/docs/content/docs/sql/reference/queries/changelog.md @@ -308,7 +308,7 @@ SELECT * FROM TO_CHANGELOG( | `input` | Yes | The input table. With `PARTITION BY`, rows with the same key are co-located and run in the same operator instance. Without `PARTITION BY`, each row is processed independently. Accepts insert-only, retract, and upsert tables. For upsert tables, the provided `PARTITION BY` key should match or be a subset of the upsert key of the subquery. | | `op` | No | A `DESCRIPTOR` with a single column name for the operation code column. Defaults to `op`. | | `op_mapping` | No | A `MAP` mapping change operation names to custom output codes. Keys can contain comma-separated names to map multiple operations to the same code (e.g., `'INSERT, UPDATE_AFTER'`). When provided, only mapped operations are forwarded - unmapped events are dropped. Each change operation may appear at most once across all entries. | -| `produces_full_deletes` | No | A `BOOLEAN` literal that controls how DELETE rows are emitted. When `true`, the function requires fully-populated DELETE rows from the input. The planner inserts a `ChangelogNormalize` operator for upsert sources that emit key-only deletes, so downstream sees the full pre-image on DELETE. When `false` (default), no full-delete requirement is enforced. In row semantics, the function preserves the planner-derived upsert key columns of the input on DELETE rows and nulls the rest (or passes the input through unchanged when no upsert key is derivable). With `PARTITION BY` (set semantics), the function additionally nulls non-partition-key columns on DELETE rows even when the input row is fully populated, so the output always carries only the key on DELETE. | +| `produces_full_deletes` | No | A `BOOLEAN` literal that controls how DELETE rows are emitted. When `true` (default), the function requires fully-populated DELETE rows from the input. The planner inserts a `ChangelogNormalize` operator for upsert sources that emit key-only deletes, so downstream sees the full pre-image on DELETE. When `false`, the function instead emits partial DELETE rows: in row semantics it preserves the planner-derived upsert key columns of the input and nulls the rest; in set semantics (`PARTITION BY`) it preserves the partition key and nulls the rest. Requires that the input declares an upsert key or that the call uses `PARTITION BY`; otherwise the function has no identifying columns to preserve and the call is rejected. | #### Default op_mapping @@ -401,44 +401,72 @@ SELECT * FROM TO_CHANGELOG( #### Delete handling -The `produces_full_deletes` argument controls how DELETE rows are emitted and what the planner requires from the input. The behavior depends on whether `PARTITION BY` is used (set semantics) or not (row semantics). +The `produces_full_deletes` argument controls how DELETE rows are emitted and what the planner requires from the input. The matrix below shows each combination with `PARTITION BY` (set semantics) and without (row semantics). -**With `produces_full_deletes => true`.** The planner requires the input to produce DELETE rows with all columns populated. For upsert sources, a `ChangelogNormalize` operator is inserted to materialize the full pre-image from state. The function then emits fully-populated DELETE rows downstream. +##### `produces_full_deletes => true` (default) + +The planner requires fully-populated DELETE rows on the input. For upsert sources that emit key-only deletes, a `ChangelogNormalize` operator is inserted upstream to materialize the full pre-image from state. For sources that already emit a full pre-image (e.g. retract), the flag is a no-op. The function then passes the input row through unchanged on DELETE. + +**Row semantics** (no `PARTITION BY`): ```sql -- Upsert source: -D[id:5] (key-only). -- ChangelogNormalize materializes the full pre-image from state. -- Output: +I[op:'DELETE', id:5, name:'Alice'] -SELECT * FROM TO_CHANGELOG( - input => TABLE upsert_source, - produces_full_deletes => true -) -``` +SELECT * FROM TO_CHANGELOG(input => TABLE upsert_source) -**With `produces_full_deletes => false` (default).** The planner does not require fully-populated DELETE rows on the input. For upsert sources that emit key-only deletes (e.g. Kafka compacted topics), this avoids the stateful `ChangelogNormalize` operator that would otherwise materialize the full pre-image of each deleted row. +-- Retract source: -D[id:5, name:'Alice'] (full pre-image). +-- No ChangelogNormalize inserted; the input row is passed through unchanged. +-- Output: +I[op:'DELETE', id:5, name:'Alice'] +SELECT * FROM TO_CHANGELOG(input => TABLE retract_source) +``` -In **row semantics** (no `PARTITION BY`), the function preserves the upsert key columns of the input on DELETE rows and nulls the rest. The upsert key is the one the planner already derives from the input (typically a declared `PRIMARY KEY`), so no `PARTITION BY` is required to get partial DELETEs. +**Set semantics** (`PARTITION BY`): ```sql --- Source 't' declares PRIMARY KEY (id). Source emits -D[id:5, name:'Alice'] (full pre-image). --- Output: +I[op:'DELETE', id:5, name:null] -SELECT * FROM TO_CHANGELOG(input => TABLE t) +-- Upsert source: -D[id:5] (key-only). +-- ChangelogNormalize materializes the full pre-image from state. +-- Output: +I[id:5, op:'DELETE', name:'Alice'] +SELECT * FROM TO_CHANGELOG(input => TABLE upsert_source PARTITION BY id) + +-- Retract source: -D[id:5, name:'Alice'] (full pre-image). +-- No ChangelogNormalize inserted; the input row is passed through unchanged. +-- Output: +I[id:5, op:'DELETE', name:'Alice'] +SELECT * FROM TO_CHANGELOG(input => TABLE retract_source PARTITION BY id) ``` -When the input has no derivable upsert key (e.g. a pure append-only source, or an upstream operator that erased the key), there is no identifying column to preserve. The function then passes the input through unchanged. +##### `produces_full_deletes => false` + +The planner skips `ChangelogNormalize` and the function emits partial DELETE rows. This avoids the stateful normalization operator for upsert sources (e.g. Kafka compacted topics) where the full pre-image is not needed downstream. Requires an upsert key (row semantics) or `PARTITION BY` (set semantics); otherwise the call is rejected with a validation error. + +**Row semantics** (no `PARTITION BY`): the function preserves the planner-derived upsert key columns on DELETE rows and nulls the rest. The upsert key is typically a declared `PRIMARY KEY`. ```sql --- Source emits -D[id:5] (key-only, no declared key). +-- Upsert source with PRIMARY KEY (id): -D[id:5] (key-only). -- Output: +I[op:'DELETE', id:5, name:null] -SELECT * FROM TO_CHANGELOG(input => TABLE upsert_source) +SELECT * FROM TO_CHANGELOG(input => TABLE upsert_source, produces_full_deletes => false) + +-- Retract source with PRIMARY KEY (id): -D[id:5, name:'Alice'] (full pre-image). +-- Output: +I[op:'DELETE', id:5, name:null] +SELECT * FROM TO_CHANGELOG(input => TABLE retract_source, produces_full_deletes => false) ``` -In **set semantics** (`PARTITION BY`), the function nulls every non-partition-key column on DELETE rows. This forces the output to carry only the partition key on DELETE even when the input row was fully populated, which matches the shape expected by upsert sinks and Kafka compacted topics. +**Set semantics** (`PARTITION BY`): the function preserves the partition key and nulls every non-partition-key column on DELETE rows. This matches the shape expected by upsert sinks and Kafka compacted topics. ```sql --- Source emits -D[id:5, name:'Alice'] (full pre-image, e.g. from a retract source). +-- Upsert source: -D[id:5] (key-only). -- Output: +I[id:5, op:'DELETE', name:null] -SELECT * FROM TO_CHANGELOG(input => TABLE retract_source PARTITION BY id) +SELECT * FROM TO_CHANGELOG( + input => TABLE upsert_source PARTITION BY id, + produces_full_deletes => false +) + +-- Retract source: -D[id:5, name:'Alice'] (full pre-image). +-- Output: +I[id:5, op:'DELETE', name:null] +SELECT * FROM TO_CHANGELOG( + input => TABLE retract_source PARTITION BY id, + produces_full_deletes => false +) ``` #### Partitioning by a key diff --git a/flink-python/pyflink/table/table.py b/flink-python/pyflink/table/table.py index 45f3d8bf0e729..e7898e8752d1f 100644 --- a/flink-python/pyflink/table/table.py +++ b/flink-python/pyflink/table/table.py @@ -1197,14 +1197,13 @@ def to_changelog(self, *arguments: Expression) -> 'Table': (INSERT, UPDATE_BEFORE, UPDATE_AFTER, DELETE). The optional ``produces_full_deletes`` boolean controls how DELETE rows are - emitted. When ``True``, the planner inserts a ``ChangelogNormalize`` operator - for upsert sources that emit key-only deletes so the function emits fully - populated DELETE rows downstream. When ``False`` (default), no full-delete - requirement is enforced. In row semantics the function preserves the - planner-derived upsert key columns of the input on DELETE rows and nulls the - rest, or passes the input through unchanged when no upsert key is derivable. - In set semantics (``PARTITION BY``) non-partition-key columns are nulled on - DELETE rows. + emitted. When ``True`` (default), the planner inserts a ``ChangelogNormalize`` + operator for upsert sources that emit key-only deletes so the function emits + fully populated DELETE rows downstream. When ``False``, the function emits + partial DELETE rows: row semantics preserves the planner-derived upsert key + columns and nulls the rest, set semantics (``PARTITION BY``) preserves the + partition key and nulls the rest. Requires an upsert key or ``PARTITION BY``; + otherwise the call is rejected. Example: :: @@ -1223,10 +1222,10 @@ def to_changelog(self, *arguments: Expression) -> 'Table': ... map_("INSERT, UPDATE_AFTER", "false", ... "DELETE", "true").as_argument("op_mapping") ... ) - >>> # Require fully populated DELETE rows from the input. Inserts a - >>> # ChangelogNormalize for upsert sources. + >>> # Opt out of full-delete semantics to emit partial DELETE rows. + >>> # Requires an upsert key or PARTITION BY; otherwise rejected. >>> result = table.to_changelog( - ... lit(True).as_argument("produces_full_deletes") + ... lit(False).as_argument("produces_full_deletes") ... ) :param arguments: Optional named arguments for ``op``, ``op_mapping``, and diff --git a/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/PartitionedTable.java b/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/PartitionedTable.java index 074bdfdbaeff3..227ffe14eecd3 100644 --- a/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/PartitionedTable.java +++ b/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/PartitionedTable.java @@ -205,12 +205,13 @@ public interface PartitionedTable { * map("INSERT, UPDATE_AFTER", "false", "DELETE", "true").asArgument("op_mapping") * ); * - * // Require fully-populated DELETE rows from the input (inserts a ChangelogNormalize for - * // upsert sources). When false (default), DELETE rows on upsert inputs may omit non-key - * // columns, which avoids the stateful normalization operator upstream. + * // Opt out of full-delete semantics. The default (true) requires fully-populated DELETE + * // rows and inserts a ChangelogNormalize for upsert sources. When false, the function + * // preserves the partition key on DELETE rows and nulls the rest, which avoids the + * // stateful normalization operator upstream. * Table result = table * .partitionBy($("id")) - * .toChangelog(lit(true).asArgument("produces_full_deletes")); + * .toChangelog(lit(false).asArgument("produces_full_deletes")); * } * * @param arguments optional named arguments for {@code op}, {@code op_mapping}, and {@code diff --git a/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/Table.java b/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/Table.java index d5f0ab6cb8f94..124a0638e2802 100644 --- a/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/Table.java +++ b/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/Table.java @@ -1460,13 +1460,13 @@ default TableResult executeInsert( * map("INSERT, UPDATE_AFTER", "false", "DELETE", "true").asArgument("op_mapping") * ); * - * // Require fully-populated DELETE rows from the input (inserts a ChangelogNormalize for - * // upsert sources). When false (default), no full-delete requirement is enforced; in row - * // semantics the function preserves the planner-derived upsert key columns of the input on - * // DELETE rows and nulls the rest, or passes the input through unchanged when no upsert key - * // is derivable. + * // Opt out of full-delete semantics. The default (true) requires fully-populated DELETE + * // rows and inserts a ChangelogNormalize for upsert sources. When false, the function + * // emits partial DELETE rows: row semantics preserves the planner-derived upsert key + * // columns and nulls the rest; set semantics preserves the partition key. Requires an + * // upsert key or PARTITION BY; otherwise the call is rejected. * Table result = table.toChangelog( - * lit(true).asArgument("produces_full_deletes") + * lit(false).asArgument("produces_full_deletes") * ); * } * diff --git a/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/BuiltInFunctionDefinitions.java b/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/BuiltInFunctionDefinitions.java index 7083b8c355b85..6cfd89110186f 100644 --- a/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/BuiltInFunctionDefinitions.java +++ b/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/BuiltInFunctionDefinitions.java @@ -840,14 +840,20 @@ ANY, and(logical(LogicalTypeRoot.BOOLEAN), LITERAL) "UPDATE_BEFORE")))) .withConditionalTrait( StaticArgumentTrait.REQUIRE_FULL_DELETE, - // Require full deletes only when the user explicitly - // asks for them via produces_full_deletes=TRUE *and* - // the active op_mapping includes DELETE. Otherwise the - // planner can skip ChangelogNormalize for upsert - // sources that emit key-only deletes. + // Require full deletes by default. The user can opt + // out via produces_full_deletes=FALSE. + // REQUIRE_FULL_DELETE + // still gates on the active op_mapping mapping DELETE; + // otherwise no DELETE rows reach the function and there + // is no point inserting ChangelogNormalize upstream. TraitCondition.and( - TraitCondition.argIsEqualTo( - "produces_full_deletes", Boolean.TRUE), + TraitCondition.or( + TraitCondition.not( + TraitCondition.argIsPresent( + "produces_full_deletes")), + TraitCondition.argIsEqualTo( + "produces_full_deletes", + Boolean.TRUE)), TraitCondition.or( TraitCondition.not( TraitCondition.argIsPresent( diff --git a/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/TableSemantics.java b/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/TableSemantics.java index 808a5e57a5231..5dc3287554387 100644 --- a/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/TableSemantics.java +++ b/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/TableSemantics.java @@ -140,6 +140,12 @@ public interface TableSemantics { *

Returns an empty array when no upsert key is derivable (e.g., a pure append-only source) * or when the planner has not yet computed metadata (during type inference). * + *

If the planner derives multiple candidate upsert keys for the same input (e.g., a table + * with several unique constraints), only one is returned. The planner today picks the candidate + * with the smallest number of columns, but ties between equal-cardinality keys are resolved by + * an implementation detail and may change across releases. Callers must not rely on which + * candidate is returned when more than one exists. + * * @return Indices of the upsert key columns of the passed table, or an empty array if none. */ int[] upsertKeyColumns(); diff --git a/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/ToChangelogTypeStrategy.java b/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/ToChangelogTypeStrategy.java index a9f9d8a38931e..f242f6f7931bb 100644 --- a/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/ToChangelogTypeStrategy.java +++ b/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/ToChangelogTypeStrategy.java @@ -201,22 +201,25 @@ private static Optional> validateOpMappingKeys( @SuppressWarnings("rawtypes") private static Optional> validateProducesFullDeletes( final CallContext callContext, final boolean throwOnFailure) { - final boolean hasArgProvided = !callContext.isArgumentNull(ARG_PRODUCES_FULL_DELETES); - if (hasArgProvided && !callContext.isArgumentLiteral(ARG_PRODUCES_FULL_DELETES)) { + final boolean isExplicit = !callContext.isArgumentNull(ARG_PRODUCES_FULL_DELETES); + if (!isExplicit) { + return Optional.empty(); + } + if (!callContext.isArgumentLiteral(ARG_PRODUCES_FULL_DELETES)) { return callContext.fail( throwOnFailure, "The 'produces_full_deletes' argument must be a constant BOOLEAN literal."); } final boolean producesFullDeletes = - callContext - .getArgumentValue(ARG_PRODUCES_FULL_DELETES, Boolean.class) - .orElse(false); + callContext.getArgumentValue(ARG_PRODUCES_FULL_DELETES, Boolean.class).orElse(true); if (!producesFullDeletes) { return Optional.empty(); } // The check against the input changelog mode lives in the function constructor since // TableSemantics#changelogMode() returns empty here at type-inference time. The mapping - // check below only needs the literal op_mapping argument, so it lives here. + // check below only needs the literal op_mapping argument, so it lives here. Only runs + // when the user explicitly set produces_full_deletes=true; the default true is not + // validated since it is a safe no-op for any input. final Optional opMapping = callContext.getArgumentValue(ARG_OP_MAPPING, Map.class); if (opMapping.isPresent() && !mapsDelete((Map) opMapping.get())) { return callContext.fail( diff --git a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ToChangelogTestPrograms.java b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ToChangelogTestPrograms.java index 996fb4fc6113e..07d74c90980f1 100644 --- a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ToChangelogTestPrograms.java +++ b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ToChangelogTestPrograms.java @@ -86,7 +86,7 @@ public class ToChangelogTestPrograms { "+I[INSERT, Bob, 20]", "+I[UPDATE_BEFORE, Alice, 10]", "+I[UPDATE_AFTER, Alice, 30]", - "+I[DELETE, Bob, null]") + "+I[DELETE, Bob, 20]") .build()) .runSql("INSERT INTO sink SELECT * FROM TO_CHANGELOG(input => TABLE t)") .build(); @@ -118,7 +118,7 @@ public class ToChangelogTestPrograms { .consumedAfterRestore( "+I[UPDATE_BEFORE, Alice, 10]", "+I[UPDATE_AFTER, Alice, 30]", - "+I[DELETE, Bob, null]") + "+I[DELETE, Bob, 20]") .build()) .runSql("INSERT INTO sink SELECT * FROM TO_CHANGELOG(input => TABLE t)") .build(); @@ -179,7 +179,7 @@ public class ToChangelogTestPrograms { "+I[INSERT, Bob, 20]", "+I[UPDATE_BEFORE, Alice, 10]", "+I[UPDATE_AFTER, Alice, 30]", - "+I[DELETE, Bob, null]") + "+I[DELETE, Bob, 20]") .build()) .runSql("INSERT INTO sink SELECT * FROM TO_CHANGELOG(input => TABLE t)") .build(); @@ -207,7 +207,7 @@ public class ToChangelogTestPrograms { "+I[Alice, C, 10]", "+I[Bob, C, 20]", "+I[Alice, C, 30]", - "+I[Bob, D, null]") + "+I[Bob, D, 20]") .build()) .runSql( "INSERT INTO sink SELECT * FROM TO_CHANGELOG(" @@ -535,7 +535,7 @@ public class ToChangelogTestPrograms { "+I[false, Alice, 10]", "+I[false, Bob, 20]", "+I[false, Alice, 30]", - "+I[true, Bob, null]") + "+I[true, Bob, 20]") .build()) .runSql( "INSERT INTO sink SELECT * FROM TO_CHANGELOG(" @@ -678,7 +678,9 @@ public class ToChangelogTestPrograms { public static final TableTestProgram ROW_SEM_PARTIAL_DELETES_VIA_UPSERT_KEY = TableTestProgram.of( "to-changelog-row-sem-partial-deletes-via-upsert-key", - "row semantics with single-column upsert key: DELETE preserves the key column and nulls the rest") + "row semantics with single-column upsert key + " + + "produces_full_deletes=false: DELETE preserves the key " + + "column and nulls the rest without requiring PARTITION BY") .setupTableSource( SourceTestStep.newBuilder("t") .addSchema( @@ -694,7 +696,10 @@ public class ToChangelogTestPrograms { .consumedValues( "+I[INSERT, Alice, 10]", "+I[DELETE, Alice, null]") .build()) - .runSql("INSERT INTO sink SELECT * FROM TO_CHANGELOG(input => TABLE t)") + .runSql( + "INSERT INTO sink SELECT * FROM TO_CHANGELOG(" + + "input => TABLE t, " + + "produces_full_deletes => false)") .build(); // -------------------------------------------------------------------------------------------- diff --git a/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/ptf/ToChangelogFunction.java b/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/ptf/ToChangelogFunction.java index e7e048c8a6bd2..60eaea73aad16 100644 --- a/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/ptf/ToChangelogFunction.java +++ b/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/ptf/ToChangelogFunction.java @@ -47,6 +47,10 @@ import java.util.Set; import java.util.stream.Collectors; +import static org.apache.flink.table.types.inference.strategies.ToChangelogTypeStrategy.ARG_OP_MAPPING; +import static org.apache.flink.table.types.inference.strategies.ToChangelogTypeStrategy.ARG_PRODUCES_FULL_DELETES; +import static org.apache.flink.table.types.inference.strategies.ToChangelogTypeStrategy.ARG_TABLE; + /** * Runtime implementation of {@link BuiltInFunctionDefinitions#TO_CHANGELOG}. * @@ -72,7 +76,7 @@ public class ToChangelogFunction extends BuiltInProcessTableFunction { private final int[] outputIndices; private final RowType inputRowType; private final boolean producesFullDelete; - private final boolean[] preserveOnDelete; + private final boolean[] upsertKeyColumn; private transient Map opMap; private transient GenericRowData opRow; @@ -86,56 +90,37 @@ public ToChangelogFunction(final SpecializedContext context) { super(BuiltInFunctionDefinitions.TO_CHANGELOG, context); final CallContext callContext = context.getCallContext(); // Table argument is guaranteed by the type strategy's validation phase. - final TableSemantics tableSemantics = callContext.getTableSemantics(0).get(); + final TableSemantics tableSemantics = callContext.getTableSemantics(ARG_TABLE).get(); final Map opMapping = - callContext.getArgumentValue(2, Map.class).orElse(null); + callContext.getArgumentValue(ARG_OP_MAPPING, Map.class).orElse(null); this.rawOpMap = buildOpMap(opMapping); if (opMapping != null) { validateOpMap(this.rawOpMap, tableSemantics); } final boolean producesFullDeletesArg = - callContext.getArgumentValue(3, Boolean.class).orElse(false); - validateProducesFullDeletes(producesFullDeletesArg, tableSemantics); + callContext.getArgumentValue(ARG_PRODUCES_FULL_DELETES, Boolean.class).orElse(true); + final boolean isExplicit = !callContext.isArgumentNull(ARG_PRODUCES_FULL_DELETES); + validateProducesFullDeletes(producesFullDeletesArg, isExplicit, tableSemantics); this.outputIndices = ChangelogTypeStrategyUtils.computeOutputIndices(tableSemantics); this.inputRowType = (RowType) tableSemantics.dataType().getLogicalType(); - final int[] upsertKeys = tableSemantics.upsertKeyColumns(); - this.producesFullDelete = resolveProducesFullDelete(producesFullDeletesArg, tableSemantics); - this.preserveOnDelete = computePreserveOnDelete(this.outputIndices, upsertKeys); - } - - /** - * Decides whether this function emits full DELETE rows (input passed through unchanged) or - * partial DELETE rows (only identifying columns preserved, rest nulled). - * - *

The framework prepends partition-key columns to the output without consulting this - * function, so in set semantics partition keys are preserved on DELETE rows for free. In row - * semantics we rely on the input table's upsert key to identify columns. Without either signal - * we cannot null anything safely and fall back to passing the input through unchanged, which is - * equivalent to a full delete. - */ - private static boolean resolveProducesFullDelete( - final boolean producesFullDeletesArg, final TableSemantics tableSemantics) { - if (producesFullDeletesArg) { - return true; - } - final boolean hasPartitionBy = tableSemantics.partitionByColumns().length > 0; - final boolean hasUpsertKey = tableSemantics.upsertKeyColumns().length > 0; - return !hasPartitionBy && !hasUpsertKey; + this.producesFullDelete = producesFullDeletesArg; + this.upsertKeyColumn = + computeUpsertKeyColumn(this.outputIndices, tableSemantics.upsertKeyColumns()); } - private static boolean[] computePreserveOnDelete( + private static boolean[] computeUpsertKeyColumn( final int[] outputIndices, final int[] upsertKeys) { final Set keepInputIndices = new HashSet<>(); for (final int key : upsertKeys) { keepInputIndices.add(key); } - final boolean[] preserve = new boolean[outputIndices.length]; + final boolean[] mask = new boolean[outputIndices.length]; for (int i = 0; i < outputIndices.length; i++) { - preserve[i] = keepInputIndices.contains(outputIndices[i]); + mask[i] = keepInputIndices.contains(outputIndices[i]); } - return preserve; + return mask; } @Override @@ -150,7 +135,7 @@ public void open(final FunctionContext context) throws Exception { preservedFieldGetters = new RowData.FieldGetter[outputIndices.length]; final List inputFieldTypes = inputRowType.getChildren(); for (int i = 0; i < outputIndices.length; i++) { - if (preserveOnDelete[i]) { + if (upsertKeyColumn[i]) { preservedFieldGetters[i] = RowData.createFieldGetter( inputFieldTypes.get(outputIndices[i]), outputIndices[i]); @@ -207,29 +192,48 @@ private static void validateOpMap( } /** - * Rejects {@code produces_full_deletes=true} when the input changelog never emits DELETE rows. + * Validates an explicit {@code produces_full_deletes} argument against the input. + * + *

For {@code produces_full_deletes=true}, the input changelog must emit DELETE rows; + * otherwise the parameter is dead. For {@code produces_full_deletes=false}, the input must + * declare an upsert key or the call must use {@code PARTITION BY}; otherwise the function has + * no identifying columns to preserve when nulling the rest. + * + *

No validation runs when the argument is absent, since the default (full deletes) is safe + * for any input. * *

Lives here rather than in the input type strategy because {@link - * TableSemantics#changelogMode()} returns empty during type inference and is only populated at - * specialization time. The complementary check against the literal {@code op_mapping} argument - * runs earlier in {@code ToChangelogTypeStrategy}. + * TableSemantics#changelogMode()} and {@link TableSemantics#upsertKeyColumns()} are only + * populated at specialization time. */ private static void validateProducesFullDeletes( - final boolean producesFullDeletesArg, final TableSemantics tableSemantics) { - if (!producesFullDeletesArg) { + final boolean producesFullDeletesArg, + final boolean isExplicit, + final TableSemantics tableSemantics) { + if (!isExplicit) { return; } - final ChangelogMode inputMode = tableSemantics.changelogMode().orElse(null); - if (inputMode == null) { + if (producesFullDeletesArg) { + final ChangelogMode inputMode = tableSemantics.changelogMode().orElse(null); + if (inputMode != null && !inputMode.contains(RowKind.DELETE)) { + throw new ValidationException( + String.format( + "Invalid 'produces_full_deletes' for TO_CHANGELOG: the input " + + "table only produces %s and never emits DELETE rows. " + + "Remove the 'produces_full_deletes' argument.", + inputMode.getContainedKinds())); + } return; } - if (!inputMode.contains(RowKind.DELETE)) { + final boolean hasPartitionBy = tableSemantics.partitionByColumns().length > 0; + final boolean hasUpsertKey = tableSemantics.upsertKeyColumns().length > 0; + if (!hasPartitionBy && !hasUpsertKey) { throw new ValidationException( - String.format( - "Invalid 'produces_full_deletes' for TO_CHANGELOG: the input table " - + "only produces %s and never emits DELETE rows. Remove the " - + "'produces_full_deletes' argument.", - inputMode.getContainedKinds())); + "Invalid 'produces_full_deletes=false' for TO_CHANGELOG: the input has no " + + "upsert key and the call has no PARTITION BY, so the function has " + + "no identifying columns to preserve on DELETE rows. Remove the " + + "argument (the default emits full DELETE rows) or add a " + + "PARTITION BY."); } } @@ -261,7 +265,7 @@ public void eval( */ private RowData buildPartialDeletePayload(final RowData input) { for (int i = 0; i < outputIndices.length; i++) { - if (preserveOnDelete[i]) { + if (upsertKeyColumn[i]) { partialDeletePayload.setField(i, preservedFieldGetters[i].getFieldOrNull(input)); } else { partialDeletePayload.setField(i, null); From 7157a22bcd8b42bfd14f011b5a54882f1e4e7351 Mon Sep 17 00:00:00 2001 From: Ramin Gharib Date: Wed, 27 May 2026 15:31:24 +0200 Subject: [PATCH 6/8] [FLINK-39636][table] Address PR review feedback Changes TableSemantics#upsertKeyColumns to return List exposing all candidate keys rather than a single planner-picked key, and moves the picking logic to the consuming PTF. A shared UpsertKeyUtil in flink-table-common provides a stable smallestKey(List) helper used by both planner-side UpsertKeyUtil (Calcite-based) and runtime ToChangelogFunction so the chosen key stays consistent across releases and job restarts. Test renames follow the _[PARTITION_BY_]PRODUCES__DELETES scheme. Drops the redundant SET_SEM_PARTIAL_DELETES program that no longer exercises a distinct path under the flipped default. Validation error for unknown op_mapping keys now mentions case sensitivity. ToChangelogInputTypeStrategyTest uses the ARG_* constants instead of positional integers. Trims Table#toChangelog / PartitionedTable#toChangelog Javadoc to a short pointer to the docs page and rewrites the user-facing 'Full vs partial deletes' section, adding an explainer of what an upsert key is. --- .../docs/sql/reference/queries/changelog.md | 26 +++++-- .../flink/table/api/PartitionedTable.java | 7 +- .../org/apache/flink/table/api/Table.java | 8 +-- .../rules/ResolveCallByArgumentsRule.java | 4 +- .../flink/table/functions/TableSemantics.java | 32 +++++---- .../strategies/ToChangelogTypeStrategy.java | 2 +- .../flink/table/utils/UpsertKeyUtils.java | 67 +++++++++++++++++ .../ToChangelogInputTypeStrategyTest.java | 63 ++++++++-------- .../inference/utils/TableSemanticsMock.java | 16 +++-- .../inference/CallBindingCallContext.java | 5 +- .../inference/OperatorBindingCallContext.java | 13 ++-- .../StreamExecProcessTableFunction.java | 11 +-- .../StreamPhysicalProcessTableFunction.java | 14 ++-- .../planner/plan/utils/UpsertKeyUtil.java | 23 ++---- .../exec/stream/ToChangelogSemanticTests.java | 15 ++-- .../exec/stream/ToChangelogTestPrograms.java | 72 ++++++------------- .../functions/ptf/ToChangelogFunction.java | 11 +-- .../process/RuntimeTableSemantics.java | 7 +- .../functions/TestHarnessTableSemantics.java | 11 +-- 19 files changed, 234 insertions(+), 173 deletions(-) create mode 100644 flink-table/flink-table-common/src/main/java/org/apache/flink/table/utils/UpsertKeyUtils.java diff --git a/docs/content/docs/sql/reference/queries/changelog.md b/docs/content/docs/sql/reference/queries/changelog.md index 425062edd17f2..c9bb439cc5308 100644 --- a/docs/content/docs/sql/reference/queries/changelog.md +++ b/docs/content/docs/sql/reference/queries/changelog.md @@ -308,7 +308,7 @@ SELECT * FROM TO_CHANGELOG( | `input` | Yes | The input table. With `PARTITION BY`, rows with the same key are co-located and run in the same operator instance. Without `PARTITION BY`, each row is processed independently. Accepts insert-only, retract, and upsert tables. For upsert tables, the provided `PARTITION BY` key should match or be a subset of the upsert key of the subquery. | | `op` | No | A `DESCRIPTOR` with a single column name for the operation code column. Defaults to `op`. | | `op_mapping` | No | A `MAP` mapping change operation names to custom output codes. Keys can contain comma-separated names to map multiple operations to the same code (e.g., `'INSERT, UPDATE_AFTER'`). When provided, only mapped operations are forwarded - unmapped events are dropped. Each change operation may appear at most once across all entries. | -| `produces_full_deletes` | No | A `BOOLEAN` literal that controls how DELETE rows are emitted. When `true` (default), the function requires fully-populated DELETE rows from the input. The planner inserts a `ChangelogNormalize` operator for upsert sources that emit key-only deletes, so downstream sees the full pre-image on DELETE. When `false`, the function instead emits partial DELETE rows: in row semantics it preserves the planner-derived upsert key columns of the input and nulls the rest; in set semantics (`PARTITION BY`) it preserves the partition key and nulls the rest. Requires that the input declares an upsert key or that the call uses `PARTITION BY`; otherwise the function has no identifying columns to preserve and the call is rejected. | +| `produces_full_deletes` | No | A `BOOLEAN` literal that controls how DELETE rows are emitted. When `true` (default), DELETE rows carry all columns, the full image. When `false`, only the identifying key columns are preserved and the rest are nulled. See [Full vs partial deletes](#full-vs-partial-deletes) for more details. | #### Default op_mapping @@ -399,9 +399,23 @@ SELECT * FROM TO_CHANGELOG( -- UPDATE_BEFORE is dropped (not in the mapping) ``` -#### Delete handling +#### Upsert key -The `produces_full_deletes` argument controls how DELETE rows are emitted and what the planner requires from the input. The matrix below shows each combination with `PARTITION BY` (set semantics) and without (row semantics). +An **upsert key** is a column or set of columns that uniquely identifies a row across its lifecycle in a changelog. It is what downstream operators and sinks use to decide which earlier row a new INSERT, UPDATE_AFTER, or DELETE refers to. + +The planner derives the upsert key from the input table: + +* A declared `PRIMARY KEY` on the source table when reading directly. +* The grouping columns of an upstream `GROUP BY `. +* The keys propagated by operators that preserve them (e.g. lookup joins, calc-projections that keep the key columns). + +When no upsert key can be derived (e.g. a plain append-only source with no key constraint and no grouping upstream), the input has no row identity and downstream operators must treat it as append-only or fall back to retract semantics. + +`TO_CHANGELOG` consumes the upsert key to decide which columns to preserve when emitting partial DELETE rows. See [Full vs partial deletes](#full-vs-partial-deletes) below. + +#### Full vs partial deletes + +The `produces_full_deletes` argument controls how DELETE rows are emitted and what the planner requires from the input. The matrix below shows each combination with `PARTITION BY` (set semantics) and without (row semantics). When `false`, the function relies on the input table's [upsert key](#upsert-key) to decide which columns to preserve. ##### `produces_full_deletes => true` (default) @@ -437,9 +451,9 @@ SELECT * FROM TO_CHANGELOG(input => TABLE retract_source PARTITION BY id) ##### `produces_full_deletes => false` -The planner skips `ChangelogNormalize` and the function emits partial DELETE rows. This avoids the stateful normalization operator for upsert sources (e.g. Kafka compacted topics) where the full pre-image is not needed downstream. Requires an upsert key (row semantics) or `PARTITION BY` (set semantics); otherwise the call is rejected with a validation error. +The planner skips `ChangelogNormalize` and the function emits partial DELETE rows. This avoids the stateful normalization operator for upsert sources (e.g. Kafka compacted topics) where the full pre-image is not needed downstream. Requires an [upsert key](#upsert-key) to be present for the input table (row semantics) or `PARTITION BY` (set semantics); otherwise the call is rejected with a validation error. -**Row semantics** (no `PARTITION BY`): the function preserves the planner-derived upsert key columns on DELETE rows and nulls the rest. The upsert key is typically a declared `PRIMARY KEY`. +**Row semantics** (no `PARTITION BY`): the function preserves the planner-derived upsert key columns on DELETE rows and nulls the rest. The upsert key is typically a declared `PRIMARY KEY` when directly reading from a source or the key provided in a `GROUP BY `. ```sql -- Upsert source with PRIMARY KEY (id): -D[id:5] (key-only). @@ -451,7 +465,7 @@ SELECT * FROM TO_CHANGELOG(input => TABLE upsert_source, produces_full_deletes = SELECT * FROM TO_CHANGELOG(input => TABLE retract_source, produces_full_deletes => false) ``` -**Set semantics** (`PARTITION BY`): the function preserves the partition key and nulls every non-partition-key column on DELETE rows. This matches the shape expected by upsert sinks and Kafka compacted topics. +**Set semantics** (`PARTITION BY`): the function preserves the partition key and nulls every non-partition-key column on DELETE rows. The key used as the partition-key column should be the unique key that will be used as the record identifier. This matches the shape expected by upsert sinks and Kafka compacted topics. ```sql -- Upsert source: -D[id:5] (key-only). diff --git a/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/PartitionedTable.java b/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/PartitionedTable.java index 227ffe14eecd3..ed541dd53b974 100644 --- a/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/PartitionedTable.java +++ b/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/PartitionedTable.java @@ -205,10 +205,9 @@ public interface PartitionedTable { * map("INSERT, UPDATE_AFTER", "false", "DELETE", "true").asArgument("op_mapping") * ); * - * // Opt out of full-delete semantics. The default (true) requires fully-populated DELETE - * // rows and inserts a ChangelogNormalize for upsert sources. When false, the function - * // preserves the partition key on DELETE rows and nulls the rest, which avoids the - * // stateful normalization operator upstream. + * // Opt out of full-delete semantics. When `true` (default), DELETE rows carry the full + * // pre-image. When `false`, only the identifying key columns are preserved and the rest + * // are nulled. See [Delete handling](#delete-handling) for more details. * Table result = table * .partitionBy($("id")) * .toChangelog(lit(false).asArgument("produces_full_deletes")); diff --git a/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/Table.java b/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/Table.java index 124a0638e2802..5581c096aa184 100644 --- a/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/Table.java +++ b/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/Table.java @@ -1460,11 +1460,9 @@ default TableResult executeInsert( * map("INSERT, UPDATE_AFTER", "false", "DELETE", "true").asArgument("op_mapping") * ); * - * // Opt out of full-delete semantics. The default (true) requires fully-populated DELETE - * // rows and inserts a ChangelogNormalize for upsert sources. When false, the function - * // emits partial DELETE rows: row semantics preserves the planner-derived upsert key - * // columns and nulls the rest; set semantics preserves the partition key. Requires an - * // upsert key or PARTITION BY; otherwise the call is rejected. + * // Opt out of full-delete semantics. When `true` (default), DELETE rows carry the full + * // pre-image. When `false`, only the identifying key columns are preserved and the rest + * // are nulled. See [Delete handling](#delete-handling) for more details. * Table result = table.toChangelog( * lit(false).asArgument("produces_full_deletes") * ); diff --git a/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/expressions/resolver/rules/ResolveCallByArgumentsRule.java b/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/expressions/resolver/rules/ResolveCallByArgumentsRule.java index 5680cb6521887..5e4a02cb9fb67 100644 --- a/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/expressions/resolver/rules/ResolveCallByArgumentsRule.java +++ b/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/expressions/resolver/rules/ResolveCallByArgumentsRule.java @@ -782,8 +782,8 @@ public Optional changelogMode() { } @Override - public int[] upsertKeyColumns() { - return new int[0]; + public List upsertKeyColumns() { + return Collections.emptyList(); } private PartitionQueryOperation findPartitionOperation(QueryOperation op) { diff --git a/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/TableSemantics.java b/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/TableSemantics.java index 5dc3287554387..3d17401c5efab 100644 --- a/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/TableSemantics.java +++ b/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/TableSemantics.java @@ -24,6 +24,7 @@ import org.apache.flink.table.connector.ChangelogMode; import org.apache.flink.table.types.DataType; +import java.util.List; import java.util.Optional; /** @@ -129,26 +130,27 @@ public interface TableSemantics { Optional changelogMode(); /** - * Upsert key columns derived from the passed table's metadata. + * Upsert key candidates derived from the passed table's metadata. * - *

Returns 0-based column indices that uniquely identify a row for upsert semantics. This is - * distinct from {@link #partitionByColumns()}: partition keys describe distribution and - * co-location, upsert keys describe row identity. Useful for functions that need to emit - * key-only deletes, match UPDATE_BEFORE / UPDATE_AFTER pairs, or route CDC events without - * forcing the caller to repeat the key via {@code PARTITION BY}. + *

Returns a list of 0-based column index arrays that uniquely identify a row for upsert + * semantics. This is distinct from {@link #partitionByColumns()}: partition keys describe + * distribution and co-location, upsert keys describe row identity. Useful for functions that + * need to emit key-only deletes, match UPDATE_BEFORE / UPDATE_AFTER pairs, or want to have a + * unique identifier to interact with state. * - *

Returns an empty array when no upsert key is derivable (e.g., a pure append-only source) - * or when the planner has not yet computed metadata (during type inference). + *

Returns an empty list when no upsert key is derivable, or when the planner has not yet + * computed metadata (during type inference). * - *

If the planner derives multiple candidate upsert keys for the same input (e.g., a table - * with several unique constraints), only one is returned. The planner today picks the candidate - * with the smallest number of columns, but ties between equal-cardinality keys are resolved by - * an implementation detail and may change across releases. Callers must not rely on which - * candidate is returned when more than one exists. + *

When the planner derives multiple candidate upsert keys for the same input (e.g., a table + * with several primary key constraints), all of them are returned. Picking which candidate to + * use is the function's responsibility, and the choice must be stable across releases to keep + * PTF state consistent after job restarts and upgrades. The order of the returned list is not + * part of the contract; PTF authors should not depend on it. A typical choice is the smallest + * candidate by cardinality, with ties broken by the column indices in ascending order. * - * @return Indices of the upsert key columns of the passed table, or an empty array if none. + * @return Candidate upsert keys of the passed table, or an empty list if none. */ - int[] upsertKeyColumns(); + List upsertKeyColumns(); /** The sort direction for ORDER BY columns in table arguments with set semantics. */ @PublicEvolving diff --git a/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/ToChangelogTypeStrategy.java b/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/ToChangelogTypeStrategy.java index f242f6f7931bb..4c27aae4723a9 100644 --- a/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/ToChangelogTypeStrategy.java +++ b/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/ToChangelogTypeStrategy.java @@ -181,7 +181,7 @@ private static Optional> validateOpMappingKeys( throwOnFailure, String.format( "Invalid target mapping for argument 'op_mapping'. " - + "Unknown change operation: '%s'. Valid values are: %s.", + + "Unknown change operation: '%s'. Operations are case-sensitive. Valid values are: %s.", rowKindName, VALID_ROW_KIND_NAMES)); } final boolean isDuplicate = !allRowKindsSeen.add(rowKindName); diff --git a/flink-table/flink-table-common/src/main/java/org/apache/flink/table/utils/UpsertKeyUtils.java b/flink-table/flink-table-common/src/main/java/org/apache/flink/table/utils/UpsertKeyUtils.java new file mode 100644 index 0000000000000..528a977d037e1 --- /dev/null +++ b/flink-table/flink-table-common/src/main/java/org/apache/flink/table/utils/UpsertKeyUtils.java @@ -0,0 +1,67 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.table.utils; + +import org.apache.flink.annotation.Internal; + +import java.util.Comparator; +import java.util.List; + +/** Helpers for working with upsert key candidates. */ +@Internal +public final class UpsertKeyUtils { + + /** + * Comparator that orders upsert-key candidates deterministically and stably across releases: + * candidates with fewer columns come first; ties between equal-cardinality candidates are + * broken by the column indices in ascending order, leading column first. + */ + private static final Comparator SMALLEST_FIRST = + Comparator.comparingInt(a -> a.length) + .thenComparing( + (a, b) -> { + for (int i = 0; i < a.length; i++) { + final int cmp = Integer.compare(a[i], b[i]); + if (cmp != 0) { + return cmp; + } + } + return 0; + }); + + /** + * Picks the smallest upsert key from the given candidates using {@link #SMALLEST_FIRST}. + * Returns an empty array when the candidate list is empty. The returned reference is one of the + * input arrays; callers must not mutate it. + */ + public static int[] smallestKey(final List candidates) { + if (candidates.isEmpty()) { + return new int[0]; + } + int[] smallest = candidates.get(0); + for (int i = 1; i < candidates.size(); i++) { + if (SMALLEST_FIRST.compare(candidates.get(i), smallest) < 0) { + smallest = candidates.get(i); + } + } + return smallest; + } + + private UpsertKeyUtils() {} +} diff --git a/flink-table/flink-table-common/src/test/java/org/apache/flink/table/types/inference/strategies/ToChangelogInputTypeStrategyTest.java b/flink-table/flink-table-common/src/test/java/org/apache/flink/table/types/inference/strategies/ToChangelogInputTypeStrategyTest.java index fb1cb918997d6..6b5773949dbd0 100644 --- a/flink-table/flink-table-common/src/test/java/org/apache/flink/table/types/inference/strategies/ToChangelogInputTypeStrategyTest.java +++ b/flink-table/flink-table-common/src/test/java/org/apache/flink/table/types/inference/strategies/ToChangelogInputTypeStrategyTest.java @@ -28,6 +28,10 @@ import java.util.stream.Stream; import static org.apache.flink.table.types.inference.strategies.SpecificInputTypeStrategies.TO_CHANGELOG_INPUT_TYPE_STRATEGY; +import static org.apache.flink.table.types.inference.strategies.ToChangelogTypeStrategy.ARG_OP; +import static org.apache.flink.table.types.inference.strategies.ToChangelogTypeStrategy.ARG_OP_MAPPING; +import static org.apache.flink.table.types.inference.strategies.ToChangelogTypeStrategy.ARG_PRODUCES_FULL_DELETES; +import static org.apache.flink.table.types.inference.strategies.ToChangelogTypeStrategy.ARG_TABLE; /** Tests for {@link ToChangelogTypeStrategy#INPUT_TYPE_STRATEGY}. */ class ToChangelogInputTypeStrategyTest extends InputTypeStrategiesTestBase { @@ -52,10 +56,10 @@ protected Stream testData() { TO_CHANGELOG_INPUT_TYPE_STRATEGY) .calledWithArgumentTypes( TABLE_TYPE, DESCRIPTOR_TYPE, MAP_TYPE, BOOLEAN_TYPE) - .calledWithTableSemanticsAt(0, new TableSemanticsMock(TABLE_TYPE)) - .calledWithLiteralAt(1, ColumnList.of("op")) - .calledWithLiteralAt(2, null) - .calledWithLiteralAt(3, true) + .calledWithTableSemanticsAt(ARG_TABLE, new TableSemanticsMock(TABLE_TYPE)) + .calledWithLiteralAt(ARG_OP, ColumnList.of("op")) + .calledWithLiteralAt(ARG_OP_MAPPING, null) + .calledWithLiteralAt(ARG_PRODUCES_FULL_DELETES, true) .expectArgumentTypes(TABLE_TYPE, DESCRIPTOR_TYPE, MAP_TYPE, BOOLEAN_TYPE), // Valid: produces_full_deletes=true with op_mapping that includes DELETE @@ -64,10 +68,10 @@ protected Stream testData() { TO_CHANGELOG_INPUT_TYPE_STRATEGY) .calledWithArgumentTypes( TABLE_TYPE, DESCRIPTOR_TYPE, MAP_TYPE, BOOLEAN_TYPE) - .calledWithTableSemanticsAt(0, new TableSemanticsMock(TABLE_TYPE)) - .calledWithLiteralAt(1, ColumnList.of("op")) - .calledWithLiteralAt(2, Map.of("INSERT", "I", "DELETE", "D")) - .calledWithLiteralAt(3, true) + .calledWithTableSemanticsAt(ARG_TABLE, new TableSemanticsMock(TABLE_TYPE)) + .calledWithLiteralAt(ARG_OP, ColumnList.of("op")) + .calledWithLiteralAt(ARG_OP_MAPPING, Map.of("INSERT", "I", "DELETE", "D")) + .calledWithLiteralAt(ARG_PRODUCES_FULL_DELETES, true) .expectArgumentTypes(TABLE_TYPE, DESCRIPTOR_TYPE, MAP_TYPE, BOOLEAN_TYPE), // Valid: produces_full_deletes=true with comma-separated DELETE key @@ -76,10 +80,10 @@ protected Stream testData() { TO_CHANGELOG_INPUT_TYPE_STRATEGY) .calledWithArgumentTypes( TABLE_TYPE, DESCRIPTOR_TYPE, MAP_TYPE, BOOLEAN_TYPE) - .calledWithTableSemanticsAt(0, new TableSemanticsMock(TABLE_TYPE)) - .calledWithLiteralAt(1, ColumnList.of("op")) - .calledWithLiteralAt(2, Map.of("INSERT, DELETE", "X")) - .calledWithLiteralAt(3, true) + .calledWithTableSemanticsAt(ARG_TABLE, new TableSemanticsMock(TABLE_TYPE)) + .calledWithLiteralAt(ARG_OP, ColumnList.of("op")) + .calledWithLiteralAt(ARG_OP_MAPPING, Map.of("INSERT, DELETE", "X")) + .calledWithLiteralAt(ARG_PRODUCES_FULL_DELETES, true) .expectArgumentTypes(TABLE_TYPE, DESCRIPTOR_TYPE, MAP_TYPE, BOOLEAN_TYPE), // Valid: produces_full_deletes=false with op_mapping that omits DELETE @@ -88,10 +92,10 @@ protected Stream testData() { TO_CHANGELOG_INPUT_TYPE_STRATEGY) .calledWithArgumentTypes( TABLE_TYPE, DESCRIPTOR_TYPE, MAP_TYPE, BOOLEAN_TYPE) - .calledWithTableSemanticsAt(0, new TableSemanticsMock(TABLE_TYPE)) - .calledWithLiteralAt(1, ColumnList.of("op")) - .calledWithLiteralAt(2, Map.of("INSERT, UPDATE_AFTER", "X")) - .calledWithLiteralAt(3, false) + .calledWithTableSemanticsAt(ARG_TABLE, new TableSemanticsMock(TABLE_TYPE)) + .calledWithLiteralAt(ARG_OP, ColumnList.of("op")) + .calledWithLiteralAt(ARG_OP_MAPPING, Map.of("INSERT, UPDATE_AFTER", "X")) + .calledWithLiteralAt(ARG_PRODUCES_FULL_DELETES, false) .expectArgumentTypes(TABLE_TYPE, DESCRIPTOR_TYPE, MAP_TYPE, BOOLEAN_TYPE), // Error: produces_full_deletes=true with op_mapping that strips DELETE @@ -100,22 +104,22 @@ protected Stream testData() { TO_CHANGELOG_INPUT_TYPE_STRATEGY) .calledWithArgumentTypes( TABLE_TYPE, DESCRIPTOR_TYPE, MAP_TYPE, BOOLEAN_TYPE) - .calledWithTableSemanticsAt(0, new TableSemanticsMock(TABLE_TYPE)) - .calledWithLiteralAt(1, ColumnList.of("op")) - .calledWithLiteralAt(2, Map.of("INSERT, UPDATE_AFTER", "X")) - .calledWithLiteralAt(3, true) + .calledWithTableSemanticsAt(ARG_TABLE, new TableSemanticsMock(TABLE_TYPE)) + .calledWithLiteralAt(ARG_OP, ColumnList.of("op")) + .calledWithLiteralAt(ARG_OP_MAPPING, Map.of("INSERT, UPDATE_AFTER", "X")) + .calledWithLiteralAt(ARG_PRODUCES_FULL_DELETES, true) .expectErrorMessage( "Invalid 'produces_full_deletes' for TO_CHANGELOG: the active " + "'op_mapping' does not map DELETE rows"), - // Error: multi-column descriptor + // Error: multi-column descriptor for `op` TestSpec.forStrategy( "Descriptor with multiple columns", TO_CHANGELOG_INPUT_TYPE_STRATEGY) .calledWithArgumentTypes( TABLE_TYPE, DESCRIPTOR_TYPE, MAP_TYPE, BOOLEAN_TYPE) - .calledWithTableSemanticsAt(0, new TableSemanticsMock(TABLE_TYPE)) - .calledWithLiteralAt(1, ColumnList.of("a", "b")) + .calledWithTableSemanticsAt(ARG_TABLE, new TableSemanticsMock(TABLE_TYPE)) + .calledWithLiteralAt(ARG_OP, ColumnList.of("a", "b")) .expectErrorMessage("must contain exactly one column name"), // Error: invalid RowKind in op_mapping key @@ -123,9 +127,9 @@ protected Stream testData() { "Invalid RowKind in mapping key", TO_CHANGELOG_INPUT_TYPE_STRATEGY) .calledWithArgumentTypes( TABLE_TYPE, DESCRIPTOR_TYPE, MAP_TYPE, BOOLEAN_TYPE) - .calledWithTableSemanticsAt(0, new TableSemanticsMock(TABLE_TYPE)) - .calledWithLiteralAt(1, ColumnList.of("op")) - .calledWithLiteralAt(2, Map.of("INVALID_KIND", "X")) + .calledWithTableSemanticsAt(ARG_TABLE, new TableSemanticsMock(TABLE_TYPE)) + .calledWithLiteralAt(ARG_OP, ColumnList.of("op")) + .calledWithLiteralAt(ARG_OP_MAPPING, Map.of("INVALID_KIND", "X")) .expectErrorMessage("Unknown change operation: 'INVALID_KIND'"), // Error: duplicate RowKind across entries @@ -134,9 +138,10 @@ protected Stream testData() { TO_CHANGELOG_INPUT_TYPE_STRATEGY) .calledWithArgumentTypes( TABLE_TYPE, DESCRIPTOR_TYPE, MAP_TYPE, BOOLEAN_TYPE) - .calledWithTableSemanticsAt(0, new TableSemanticsMock(TABLE_TYPE)) - .calledWithLiteralAt(1, ColumnList.of("op")) - .calledWithLiteralAt(2, Map.of("INSERT, DELETE", "A", "DELETE", "B")) + .calledWithTableSemanticsAt(ARG_TABLE, new TableSemanticsMock(TABLE_TYPE)) + .calledWithLiteralAt(ARG_OP, ColumnList.of("op")) + .calledWithLiteralAt( + ARG_OP_MAPPING, Map.of("INSERT, DELETE", "A", "DELETE", "B")) .expectErrorMessage("Duplicate change operation: 'DELETE'")); } } diff --git a/flink-table/flink-table-common/src/test/java/org/apache/flink/table/types/inference/utils/TableSemanticsMock.java b/flink-table/flink-table-common/src/test/java/org/apache/flink/table/types/inference/utils/TableSemanticsMock.java index bd0ead2a3180b..9b9830870fd4e 100644 --- a/flink-table/flink-table-common/src/test/java/org/apache/flink/table/types/inference/utils/TableSemanticsMock.java +++ b/flink-table/flink-table-common/src/test/java/org/apache/flink/table/types/inference/utils/TableSemanticsMock.java @@ -24,6 +24,8 @@ import javax.annotation.Nullable; +import java.util.Collections; +import java.util.List; import java.util.Optional; /** Mock implementation of {@link TableSemantics} for testing purposes. */ @@ -35,7 +37,7 @@ public class TableSemanticsMock implements TableSemantics { private final SortDirection[] orderByDirections; private final int timeColumn; private final ChangelogMode changelogMode; - private final int[] upsertKeyColumns; + private final List upsertKeyColumns; public TableSemanticsMock(DataType dataType) { this(dataType, new int[0], new int[0], -1, null); @@ -47,7 +49,13 @@ public TableSemanticsMock( int[] orderByColumns, int timeColumn, @Nullable ChangelogMode changelogMode) { - this(dataType, partitionByColumns, orderByColumns, timeColumn, changelogMode, new int[0]); + this( + dataType, + partitionByColumns, + orderByColumns, + timeColumn, + changelogMode, + Collections.emptyList()); } public TableSemanticsMock( @@ -56,7 +64,7 @@ public TableSemanticsMock( int[] orderByColumns, int timeColumn, @Nullable ChangelogMode changelogMode, - int[] upsertKeyColumns) { + List upsertKeyColumns) { this.dataType = dataType; this.partitionByColumns = partitionByColumns; this.orderByColumns = orderByColumns; @@ -100,7 +108,7 @@ public Optional changelogMode() { } @Override - public int[] upsertKeyColumns() { + public List upsertKeyColumns() { return upsertKeyColumns; } } diff --git a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/inference/CallBindingCallContext.java b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/inference/CallBindingCallContext.java index e5615273f3109..b86fe9b4175ac 100644 --- a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/inference/CallBindingCallContext.java +++ b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/inference/CallBindingCallContext.java @@ -50,6 +50,7 @@ import javax.annotation.Nullable; import java.util.AbstractList; +import java.util.Collections; import java.util.LinkedHashMap; import java.util.List; import java.util.Map; @@ -314,8 +315,8 @@ public Optional changelogMode() { } @Override - public int[] upsertKeyColumns() { - return new int[0]; + public List upsertKeyColumns() { + return Collections.emptyList(); } } diff --git a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/inference/OperatorBindingCallContext.java b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/inference/OperatorBindingCallContext.java index f69de54c54d82..2127fd47814b9 100644 --- a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/inference/OperatorBindingCallContext.java +++ b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/inference/OperatorBindingCallContext.java @@ -195,10 +195,7 @@ public Optional getTableSemantics(int pos) { Optional.ofNullable(inputChangelogModes) .map(m -> m.get(tableArgCall.getInputIndex())) .orElse(null); - final int[] upsertKeys = - Optional.ofNullable(inputUpsertKeys) - .map(m -> m.get(tableArgCall.getInputIndex())) - .orElse(new int[0]); + final List upsertKeys = inputUpsertKeys != null ? inputUpsertKeys : List.of(); return Optional.of( OperatorBindingTableSemantics.create( argumentDataTypes.get(pos), @@ -310,7 +307,7 @@ private static class OperatorBindingTableSemantics implements TableSemantics { private final SortDirection[] orderByDirections; private final int timeColumn; private final @Nullable ChangelogMode changelogMode; - private final int[] upsertKeyColumns; + private final List upsertKeyColumns; public static OperatorBindingTableSemantics create( DataType tableDataType, @@ -318,7 +315,7 @@ public static OperatorBindingTableSemantics create( RexTableArgCall tableArgCall, int timeColumn, @Nullable ChangelogMode changelogMode, - int[] upsertKeyColumns) { + List upsertKeyColumns) { return new OperatorBindingTableSemantics( createDataType(tableDataType, staticArg), tableArgCall.getPartitionKeys(), @@ -336,7 +333,7 @@ private OperatorBindingTableSemantics( SortDirection[] orderByDirections, int timeColumn, @Nullable ChangelogMode changelogMode, - int[] upsertKeyColumns) { + List upsertKeyColumns) { this.dataType = dataType; this.partitionByColumns = partitionByColumns; this.orderByColumns = orderByColumns; @@ -387,7 +384,7 @@ public Optional changelogMode() { } @Override - public int[] upsertKeyColumns() { + public List upsertKeyColumns() { return upsertKeyColumns; } } diff --git a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/stream/StreamExecProcessTableFunction.java b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/stream/StreamExecProcessTableFunction.java index ad8fa35553d50..5df54baf397b2 100644 --- a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/stream/StreamExecProcessTableFunction.java +++ b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/stream/StreamExecProcessTableFunction.java @@ -173,12 +173,7 @@ public StreamExecProcessTableFunction( this.invocation = BridgingSqlFunction.resolveCallTraits((RexCall) invocation); this.inputChangelogModes = inputChangelogModes; this.outputChangelogMode = outputChangelogMode; - this.inputUpsertKeys = - inputUpsertKeys != null - ? inputUpsertKeys - : inputProperties.stream() - .map(p -> new int[0]) - .collect(Collectors.toList()); + this.inputUpsertKeys = inputUpsertKeys != null ? inputUpsertKeys : List.of(); } public @Nullable String getUid() { @@ -331,8 +326,8 @@ private RuntimeTableSemantics createRuntimeTableSemantics( final int timeColumn = inputTimeColumns.get(tableArgCall.getInputIndex()); final int inputIndex = tableArgCall.getInputIndex(); - final int[] upsertKeys = - inputIndex < inputUpsertKeys.size() ? inputUpsertKeys.get(inputIndex) : new int[0]; + final List upsertKeys = + inputIndex < inputUpsertKeys.size() ? inputUpsertKeys : List.of(); return new RuntimeTableSemantics( tableArg.getName(), inputIndex, diff --git a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/physical/stream/StreamPhysicalProcessTableFunction.java b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/physical/stream/StreamPhysicalProcessTableFunction.java index d08b0854c46af..9de4ab1871d6c 100644 --- a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/physical/stream/StreamPhysicalProcessTableFunction.java +++ b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/physical/stream/StreamPhysicalProcessTableFunction.java @@ -33,7 +33,6 @@ import org.apache.flink.table.planner.plan.nodes.exec.stream.StreamExecProcessTableFunction; import org.apache.flink.table.planner.plan.nodes.logical.FlinkLogicalTableFunctionScan; import org.apache.flink.table.planner.plan.utils.ChangelogPlanUtils; -import org.apache.flink.table.planner.plan.utils.UpsertKeyUtil; import org.apache.flink.table.planner.utils.JavaScalaConversionUtil; import org.apache.flink.table.planner.utils.ShortcutUtils; import org.apache.flink.table.types.inference.StaticArgument; @@ -64,6 +63,7 @@ import java.util.ArrayList; import java.util.Arrays; +import java.util.Collections; import java.util.HashSet; import java.util.List; import java.util.Objects; @@ -172,9 +172,15 @@ public ExecNode translateToExecNode() { final List inputUpsertKeys = getInputs().stream() .map( - input -> - UpsertKeyUtil.smallestKey(fmq.getUpsertKeys(input)) - .orElse(new int[0])) + input -> { + final Set keys = fmq.getUpsertKeys(input); + return keys == null + ? Collections.emptyList() + : keys.stream() + .map(ImmutableBitSet::toArray) + .collect(Collectors.toList()); + }) + .flatMap(List::stream) .collect(Collectors.toList()); return new StreamExecProcessTableFunction( unwrapTableConfig(this), diff --git a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/utils/UpsertKeyUtil.java b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/utils/UpsertKeyUtil.java index cc301118b29f1..af99d92e9ed3b 100644 --- a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/utils/UpsertKeyUtil.java +++ b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/utils/UpsertKeyUtil.java @@ -18,13 +18,17 @@ package org.apache.flink.table.planner.plan.utils; +import org.apache.flink.table.utils.UpsertKeyUtils; + import org.apache.calcite.util.ImmutableBitSet; import javax.annotation.Nonnull; import javax.annotation.Nullable; +import java.util.List; import java.util.Optional; import java.util.Set; +import java.util.stream.Collectors; /** * Utility for upsertKey which represented as a Set of {@link @@ -55,21 +59,8 @@ public static Optional smallestKey(@Nullable Set upsertK if (null == upsertKeys || upsertKeys.isEmpty()) { return Optional.empty(); } - return upsertKeys.stream() - .map(ImmutableBitSet::toArray) - .reduce( - (k1, k2) -> { - if (k1.length < k2.length) { - return k1; - } - if (k1.length == k2.length) { - for (int index = 0; index < k1.length; index++) { - if (k1[index] < k2[index]) { - return k1; - } - } - } - return k2; - }); + final List asArrays = + upsertKeys.stream().map(ImmutableBitSet::toArray).collect(Collectors.toList()); + return Optional.of(UpsertKeyUtils.smallestKey(asArrays)); } } diff --git a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ToChangelogSemanticTests.java b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ToChangelogSemanticTests.java index 4233dad198124..41c539790679a 100644 --- a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ToChangelogSemanticTests.java +++ b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ToChangelogSemanticTests.java @@ -56,13 +56,12 @@ public List programs() { ToChangelogTestPrograms.INVALID_OP_MAPPING, ToChangelogTestPrograms.OP_MAPPING_REFERENCES_UNSUPPORTED_KIND, ToChangelogTestPrograms.DUPLICATE_ROW_KIND, - ToChangelogTestPrograms.PRODUCES_FULL_DELETES_ON_APPEND_ONLY_INPUT, - ToChangelogTestPrograms.ROW_SEM_PARTIAL_DELETES, - ToChangelogTestPrograms.ROW_SEM_FORCE_FULL_DELETES, - ToChangelogTestPrograms.ROW_SEM_PARTIAL_DELETES_VIA_UPSERT_KEY, - ToChangelogTestPrograms.SET_SEM_PARTIAL_DELETES, - ToChangelogTestPrograms.SET_SEM_FULL_DELETES, - ToChangelogTestPrograms.SET_SEM_FORCE_FULL_DELETES, - ToChangelogTestPrograms.SET_SEM_FORCE_PARTIAL_DELETES); + ToChangelogTestPrograms.INVALID_PRODUCES_FULL_DELETES_FOR_APPEND_ONLY, + ToChangelogTestPrograms.RETRACT_PRODUCES_PARTIAL_DELETES, + ToChangelogTestPrograms.UPSERT_PRODUCES_FULL_DELETES, + ToChangelogTestPrograms.UPSERT_PRODUCES_PARTIAL_DELETES, + ToChangelogTestPrograms.RETRACT_PARTITION_BY_PRODUCES_PARTIAL_DELETES, + ToChangelogTestPrograms.RETRACT_PARTITION_BY_PRODUCES_FULL_DELETES, + ToChangelogTestPrograms.UPSERT_PARTITION_BY_PRODUCES_FULL_DELETES); } } diff --git a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ToChangelogTestPrograms.java b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ToChangelogTestPrograms.java index 07d74c90980f1..c602aa49debc4 100644 --- a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ToChangelogTestPrograms.java +++ b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ToChangelogTestPrograms.java @@ -600,9 +600,9 @@ public class ToChangelogTestPrograms { "Duplicate change operation: 'DELETE'") .build(); - public static final TableTestProgram PRODUCES_FULL_DELETES_ON_APPEND_ONLY_INPUT = + public static final TableTestProgram INVALID_PRODUCES_FULL_DELETES_FOR_APPEND_ONLY = TableTestProgram.of( - "to-changelog-produces-full-deletes-on-append-only-input", + "to-changelog-invalid-produces-full-deletes-for-append-only", "fails when produces_full_deletes=true on an input that never emits DELETE rows") .setupTableSource(SIMPLE_SOURCE) .runFailingSql( @@ -614,13 +614,13 @@ public class ToChangelogTestPrograms { .build(); // -------------------------------------------------------------------------------------------- - // Row semantics x delete handling matrix + // Full vs partial deletes matrix (input kind x PARTITION BY x produces_full_deletes) // -------------------------------------------------------------------------------------------- - public static final TableTestProgram ROW_SEM_PARTIAL_DELETES = + public static final TableTestProgram RETRACT_PRODUCES_PARTIAL_DELETES = TableTestProgram.of( - "to-changelog-row-sem-partial-deletes", - "row semantics: produces_full_deletes=false skips ChangelogNormalize and a partial DELETE row from the input passes through unchanged") + "to-changelog-retract-produces-partial-deletes", + "retract input in row semantics with produces_full_deletes=false: skips ChangelogNormalize and the partial DELETE row from the input passes through unchanged") .setupTableSource( SourceTestStep.newBuilder("t") .addSchema( @@ -649,10 +649,10 @@ public class ToChangelogTestPrograms { + "produces_full_deletes => false)") .build(); - public static final TableTestProgram ROW_SEM_FORCE_FULL_DELETES = + public static final TableTestProgram UPSERT_PRODUCES_FULL_DELETES = TableTestProgram.of( - "to-changelog-row-sem-force-full-deletes", - "row semantics: produces_full_deletes=true forces ChangelogNormalize to materialize the full DELETE row from an upsert source emitting key-only deletes") + "to-changelog-upsert-produces-full-deletes", + "upsert input in row semantics with produces_full_deletes=true: ChangelogNormalize materializes the full DELETE row from a key-only delete") .setupTableSource( SourceTestStep.newBuilder("t") .addSchema( @@ -675,11 +675,11 @@ public class ToChangelogTestPrograms { + "produces_full_deletes => true)") .build(); - public static final TableTestProgram ROW_SEM_PARTIAL_DELETES_VIA_UPSERT_KEY = + public static final TableTestProgram UPSERT_PRODUCES_PARTIAL_DELETES = TableTestProgram.of( - "to-changelog-row-sem-partial-deletes-via-upsert-key", - "row semantics with single-column upsert key + " - + "produces_full_deletes=false: DELETE preserves the key " + "to-changelog-upsert-produces-partial-deletes", + "upsert input in row semantics with single-column upsert key + " + + "produces_full_deletes=false: DELETE preserves the upsert key " + "column and nulls the rest without requiring PARTITION BY") .setupTableSource( SourceTestStep.newBuilder("t") @@ -702,14 +702,10 @@ public class ToChangelogTestPrograms { + "produces_full_deletes => false)") .build(); - // -------------------------------------------------------------------------------------------- - // Set semantics x delete handling matrix - // -------------------------------------------------------------------------------------------- - - public static final TableTestProgram SET_SEM_FORCE_PARTIAL_DELETES = + public static final TableTestProgram RETRACT_PARTITION_BY_PRODUCES_PARTIAL_DELETES = TableTestProgram.of( - "to-changelog-set-sem-force-partial-deletes", - "set semantics: produces_full_deletes=false nulls non-partition-key columns on DELETE even when the input row is fully populated") + "to-changelog-retract-partition-by-produces-partial-deletes", + "retract input in set semantics with produces_full_deletes=false: nulls non-partition-key columns on DELETE even when the input row is fully populated") .setupTableSource( SourceTestStep.newBuilder("t") .addSchema( @@ -736,34 +732,10 @@ public class ToChangelogTestPrograms { + "produces_full_deletes => false)") .build(); - public static final TableTestProgram SET_SEM_PARTIAL_DELETES = - TableTestProgram.of( - "to-changelog-set-sem-partial-deletes", - "set semantics: produces_full_deletes=false (default) lets a partial DELETE row from the input pass through with non-partition-key columns null") - .setupTableSource( - SourceTestStep.newBuilder("t") - .addSchema( - "name STRING PRIMARY KEY NOT ENFORCED", "score BIGINT") - .addMode(ChangelogMode.all()) - .producedValues( - Row.ofKind(RowKind.INSERT, "Alice", 10L), - Row.ofKind(RowKind.DELETE, "Alice", null)) - .build()) - .setupTableSink( - SinkTestStep.newBuilder("sink") - .addSchema("name STRING", "op STRING", "score BIGINT") - .consumedValues( - "+I[Alice, INSERT, 10]", "+I[Alice, DELETE, null]") - .build()) - .runSql( - "INSERT INTO sink SELECT * FROM TO_CHANGELOG(" - + "input => TABLE t PARTITION BY name)") - .build(); - - public static final TableTestProgram SET_SEM_FULL_DELETES = + public static final TableTestProgram RETRACT_PARTITION_BY_PRODUCES_FULL_DELETES = TableTestProgram.of( - "to-changelog-set-sem-full-deletes", - "set semantics: produces_full_deletes=true on an input that already emits full deletes is a no-op for the planner and the full DELETE row reaches the output") + "to-changelog-retract-partition-by-produces-full-deletes", + "retract input in set semantics with produces_full_deletes=true (default): the input row passes through unchanged, full DELETE pre-image reaches the output") .setupTableSource( SourceTestStep.newBuilder("t") .addSchema( @@ -785,10 +757,10 @@ public class ToChangelogTestPrograms { + "produces_full_deletes => true)") .build(); - public static final TableTestProgram SET_SEM_FORCE_FULL_DELETES = + public static final TableTestProgram UPSERT_PARTITION_BY_PRODUCES_FULL_DELETES = TableTestProgram.of( - "to-changelog-set-sem-force-full-deletes", - "set semantics: produces_full_deletes=true forces ChangelogNormalize to materialize the full DELETE row from an upsert source emitting key-only deletes") + "to-changelog-upsert-partition-by-produces-full-deletes", + "upsert input in set semantics with produces_full_deletes=true: ChangelogNormalize materializes the full DELETE row from a key-only delete") .setupTableSource( SourceTestStep.newBuilder("t") .addSchema( diff --git a/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/ptf/ToChangelogFunction.java b/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/ptf/ToChangelogFunction.java index 60eaea73aad16..66162e205b9ab 100644 --- a/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/ptf/ToChangelogFunction.java +++ b/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/ptf/ToChangelogFunction.java @@ -35,6 +35,7 @@ import org.apache.flink.table.types.inference.strategies.ChangelogTypeStrategyUtils; import org.apache.flink.table.types.logical.LogicalType; import org.apache.flink.table.types.logical.RowType; +import org.apache.flink.table.utils.UpsertKeyUtils; import org.apache.flink.types.ColumnList; import org.apache.flink.types.RowKind; @@ -107,13 +108,15 @@ public ToChangelogFunction(final SpecializedContext context) { this.inputRowType = (RowType) tableSemantics.dataType().getLogicalType(); this.producesFullDelete = producesFullDeletesArg; this.upsertKeyColumn = - computeUpsertKeyColumn(this.outputIndices, tableSemantics.upsertKeyColumns()); + computeUpsertKeyColumn( + this.outputIndices, + UpsertKeyUtils.smallestKey(tableSemantics.upsertKeyColumns())); } private static boolean[] computeUpsertKeyColumn( - final int[] outputIndices, final int[] upsertKeys) { + final int[] outputIndices, final int[] upsertKey) { final Set keepInputIndices = new HashSet<>(); - for (final int key : upsertKeys) { + for (final int key : upsertKey) { keepInputIndices.add(key); } final boolean[] mask = new boolean[outputIndices.length]; @@ -226,7 +229,7 @@ private static void validateProducesFullDeletes( return; } final boolean hasPartitionBy = tableSemantics.partitionByColumns().length > 0; - final boolean hasUpsertKey = tableSemantics.upsertKeyColumns().length > 0; + final boolean hasUpsertKey = !tableSemantics.upsertKeyColumns().isEmpty(); if (!hasPartitionBy && !hasUpsertKey) { throw new ValidationException( "Invalid 'produces_full_deletes=false' for TO_CHANGELOG: the input has no " diff --git a/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/process/RuntimeTableSemantics.java b/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/process/RuntimeTableSemantics.java index 7d0c5cf5863ae..7f0fbf0c3d382 100644 --- a/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/process/RuntimeTableSemantics.java +++ b/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/process/RuntimeTableSemantics.java @@ -24,6 +24,7 @@ import org.apache.flink.table.types.DataType; import java.io.Serializable; +import java.util.List; import java.util.Optional; /** @@ -44,7 +45,7 @@ public class RuntimeTableSemantics implements TableSemantics, Serializable { private final boolean passColumnsThrough; private final boolean hasSetSemantics; private final int timeColumn; - private final int[] upsertKeyColumns; + private final List upsertKeyColumns; private transient ChangelogMode changelogMode; @@ -59,7 +60,7 @@ public RuntimeTableSemantics( boolean passColumnsThrough, boolean hasSetSemantics, int timeColumn, - int[] upsertKeyColumns) { + List upsertKeyColumns) { this.argName = argName; this.inputIndex = inputIndex; this.dataType = dataType; @@ -127,7 +128,7 @@ public Optional changelogMode() { } @Override - public int[] upsertKeyColumns() { + public List upsertKeyColumns() { return upsertKeyColumns; } } diff --git a/flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/TestHarnessTableSemantics.java b/flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/TestHarnessTableSemantics.java index c55e715c065c4..91edd9a059d69 100644 --- a/flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/TestHarnessTableSemantics.java +++ b/flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/TestHarnessTableSemantics.java @@ -23,6 +23,8 @@ import org.apache.flink.table.functions.TableSemantics; import org.apache.flink.table.types.DataType; +import java.util.Collections; +import java.util.List; import java.util.Optional; /** {@link TableSemantics} implementation for {@link ProcessTableFunctionTestHarness}. */ @@ -30,13 +32,14 @@ class TestHarnessTableSemantics implements TableSemantics { private final DataType dataType; private final int[] partitionByColumns; - private final int[] upsertKeyColumns; + private final List upsertKeyColumns; TestHarnessTableSemantics(DataType dataType, int[] partitionByColumns) { - this(dataType, partitionByColumns, new int[0]); + this(dataType, partitionByColumns, Collections.emptyList()); } - TestHarnessTableSemantics(DataType dataType, int[] partitionByColumns, int[] upsertKeyColumns) { + TestHarnessTableSemantics( + DataType dataType, int[] partitionByColumns, List upsertKeyColumns) { this.dataType = dataType; this.partitionByColumns = partitionByColumns; this.upsertKeyColumns = upsertKeyColumns; @@ -73,7 +76,7 @@ public Optional changelogMode() { } @Override - public int[] upsertKeyColumns() { + public List upsertKeyColumns() { return upsertKeyColumns; } } From dbaf5f90a14cc8f7e77dbe3f86615ac536d17d52 Mon Sep 17 00:00:00 2001 From: Ramin Gharib Date: Wed, 27 May 2026 17:48:04 +0200 Subject: [PATCH 7/8] [FLINK-39636][table] Address PR review feedback 2 --- docs/content/docs/sql/reference/queries/changelog.md | 10 ++++------ .../org/apache/flink/table/api/PartitionedTable.java | 4 +++- .../main/java/org/apache/flink/table/api/Table.java | 4 +++- .../nodes/exec/stream/ToChangelogTestPrograms.java | 3 +-- 4 files changed, 11 insertions(+), 10 deletions(-) diff --git a/docs/content/docs/sql/reference/queries/changelog.md b/docs/content/docs/sql/reference/queries/changelog.md index c9bb439cc5308..c91107b2df245 100644 --- a/docs/content/docs/sql/reference/queries/changelog.md +++ b/docs/content/docs/sql/reference/queries/changelog.md @@ -520,13 +520,11 @@ Table result = myTable.toChangelog( map("INSERT, UPDATE_AFTER", "false", "DELETE", "true").asArgument("op_mapping") ); -// Require fully-populated DELETE rows from the input (inserts a ChangelogNormalize for -// upsert sources). When false (default), no full-delete requirement is enforced; in row -// semantics the function preserves the input's upsert key columns on DELETE and nulls the -// rest (passes the input through unchanged when no upsert key is derivable). In set -// semantics non-partition-key columns are nulled on DELETE. +// Opt out of full-delete semantics. When `true` (default), DELETE rows carry the full +// pre-image. When `false`, only the identifying key columns are preserved and the rest +// are nulled. See [Full vs partial deletes](#full-vs-partial-deletes) for more details. Table result = myTable.toChangelog( - lit(true).asArgument("produces_full_deletes") + lit(false).asArgument("produces_full_deletes") ); // Set semantics: co-locate rows with the same key in the same parallel operator instance. diff --git a/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/PartitionedTable.java b/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/PartitionedTable.java index ed541dd53b974..401317d125d4c 100644 --- a/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/PartitionedTable.java +++ b/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/PartitionedTable.java @@ -207,7 +207,9 @@ public interface PartitionedTable { * * // Opt out of full-delete semantics. When `true` (default), DELETE rows carry the full * // pre-image. When `false`, only the identifying key columns are preserved and the rest - * // are nulled. See [Delete handling](#delete-handling) for more details. + * // are nulled. See [Full vs partial deletes]( + * // https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/changelog/#full-vs-partial-deletes) + * // for more details. * Table result = table * .partitionBy($("id")) * .toChangelog(lit(false).asArgument("produces_full_deletes")); diff --git a/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/Table.java b/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/Table.java index 5581c096aa184..ab4a617a2df6a 100644 --- a/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/Table.java +++ b/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/Table.java @@ -1462,7 +1462,9 @@ default TableResult executeInsert( * * // Opt out of full-delete semantics. When `true` (default), DELETE rows carry the full * // pre-image. When `false`, only the identifying key columns are preserved and the rest - * // are nulled. See [Delete handling](#delete-handling) for more details. + * // are nulled. See [Full vs partial deletes]( + * // https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/changelog/#full-vs-partial-deletes) + * // for more details. * Table result = table.toChangelog( * lit(false).asArgument("produces_full_deletes") * ); diff --git a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ToChangelogTestPrograms.java b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ToChangelogTestPrograms.java index c602aa49debc4..a4ba781a8ef5d 100644 --- a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ToChangelogTestPrograms.java +++ b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ToChangelogTestPrograms.java @@ -187,8 +187,7 @@ public class ToChangelogTestPrograms { public static final TableTestProgram UPSERT_PARTITION_BY = TableTestProgram.of( "to-changelog-upsert-partition-by", - "PARTITION BY upsert key + mapping without UB skips ChangelogNormalize; " - + "default produces_full_deletes=false nulls non-key columns on DELETE") + "PARTITION BY upsert key + mapping without UB skips ChangelogNormalize") .setupTableSource( SourceTestStep.newBuilder("t") .addSchema( From 3960d06b67dc30fc9162b20274f0fb94ef2deb5b Mon Sep 17 00:00:00 2001 From: Ramin Gharib Date: Wed, 27 May 2026 18:02:00 +0200 Subject: [PATCH 8/8] [FLINK-39636][table] Use NON_EMPTY for inputUpsertKeys JSON property Switches @JsonInclude on StreamExecProcessTableFunction.inputUpsertKeys from NON_DEFAULT (with a docs-only defaultValue = "[]") to NON_EMPTY so both null and empty lists are omitted from the serialized plan. Null-coalesces to Collections.emptyList() in the @JsonCreator so older compiled plans that omit the field deserialize to an empty list rather than null. --- .../nodes/exec/stream/StreamExecProcessTableFunction.java | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/stream/StreamExecProcessTableFunction.java b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/stream/StreamExecProcessTableFunction.java index 5df54baf397b2..5c5e9375c35ac 100644 --- a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/stream/StreamExecProcessTableFunction.java +++ b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/stream/StreamExecProcessTableFunction.java @@ -123,8 +123,8 @@ public class StreamExecProcessTableFunction extends ExecNodeBase @JsonProperty(FIELD_NAME_OUTPUT_CHANGELOG_MODE) private final ChangelogMode outputChangelogMode; - @JsonProperty(value = FIELD_NAME_INPUT_UPSERT_KEYS, defaultValue = "[]") - @JsonInclude(JsonInclude.Include.NON_DEFAULT) + @JsonProperty(FIELD_NAME_INPUT_UPSERT_KEYS) + @JsonInclude(JsonInclude.Include.NON_EMPTY) private final List inputUpsertKeys; public StreamExecProcessTableFunction(