-
Notifications
You must be signed in to change notification settings - Fork 13.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FLINK-34172] Add support for altering a distribution via ALTER TABLE #24886
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some initial feedback.
@@ -40,7 +41,8 @@ public static void unparseTableSchema( | |||
int rightPrec, | |||
SqlNodeList columnList, | |||
List<SqlTableConstraint> constraints, | |||
@Nullable SqlWatermark watermark) { | |||
@Nullable SqlWatermark watermark, | |||
@Nullable SqlDistribution distribution) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This doesn't seem right. A distribution is not part of the table schema. A table schema is enclosed in ( )
like in CREATE TABLE <name> (<table schema>)
. The distribution comes after the schema.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed.
...table/flink-sql-parser/src/test/java/org/apache/flink/sql/parser/FlinkSqlParserImplTest.java
Outdated
Show resolved
Hide resolved
...table/flink-sql-parser/src/test/java/org/apache/flink/sql/parser/FlinkSqlParserImplTest.java
Outdated
Show resolved
Hide resolved
...table/flink-sql-parser/src/test/java/org/apache/flink/sql/parser/FlinkSqlParserImplTest.java
Outdated
Show resolved
Hide resolved
...table/flink-sql-parser/src/test/java/org/apache/flink/sql/parser/FlinkSqlParserImplTest.java
Outdated
Show resolved
Hide resolved
...le-planner/src/main/java/org/apache/flink/table/planner/operations/AlterSchemaConverter.java
Outdated
Show resolved
Hide resolved
.partitionKeys(oldTable.getPartitionKeys()) | ||
.options(oldTable.getOptions()); | ||
|
||
if (alterTable instanceof SqlAlterTableSchema) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why can't we simply call oldTable.getDistribution
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, I ended up adding oldTable
towards the end of my work. I can try to use it here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
After looking at this again, I figured out that we need to create the new table. Since that table needs the added/modified distribution, we have to take it from the SqlAlterTableSchema
rather than the oldTable
.
...le-planner/src/main/java/org/apache/flink/table/planner/operations/AlterSchemaConverter.java
Outdated
Show resolved
Hide resolved
@@ -110,6 +113,7 @@ public Operation convertAlterSchema( | |||
SchemaConverter converter = createSchemaConverter(alterTableSchema, oldTable); | |||
converter.updateColumn(alterTableSchema.getColumnPositions().getList()); | |||
alterTableSchema.getWatermark().ifPresent(converter::updateWatermark); | |||
alterTableSchema.getDistribution().ifPresent(converter::updateDistribution); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I thought the main change in this class is only to update the distribution in case of a column name change similar to buildUpdatedPrimaryKey
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we have to pass the distribution around in order to create the new table objects. I may be missing some alternative here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the update. I only had some minor comments. The PR should be good to go in the next iteration.
flink-table/flink-sql-parser/src/main/java/org/apache/flink/sql/parser/ddl/SqlDistribution.java
Outdated
Show resolved
Hide resolved
.isNotPresent(); | ||
|
||
prepareNonManagedTableWithDistribution("tb3"); | ||
// rename column used as distribution key |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
comment seems to be wrong, also isn't this tested above already, at least is used as a distribution key
occurs two times in this clas
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Removing this case of dropping the distribution. The other two cases are for a drop and modify, so I think both of those make sense.
.../src/test/java/org/apache/flink/table/planner/operations/SqlDdlToOperationConverterTest.java
Outdated
Show resolved
Hide resolved
.../src/test/java/org/apache/flink/table/planner/operations/SqlDdlToOperationConverterTest.java
Outdated
Show resolved
Hide resolved
.../src/test/java/org/apache/flink/table/planner/operations/SqlDdlToOperationConverterTest.java
Outdated
Show resolved
Hide resolved
…ropped. Moving logic out of AlterSchemaConverter.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the update @jnh5y. LGTM.
What is the purpose of the change
This PR implements the SQL parser changes for ALTER TABLE to support ADD, MODIFY, and DROP DISTRIBUTION statements.
Brief change log
The SQL Parser has been updated.
The
AlterSchemaConverter
has been updated to pass the changes in a DISTRIBUTION through to theOperation
.Verifying this change
This change added tests and can be verified as follows:
(example:)
Does this pull request potentially affect one of the following parts:
@Public(Evolving)
: (no)Documentation