Describe the problem you faced
AWSGlueCatalogSyncClient.updateTableComments no longer applies any column or partition column comments since the AWS SDK v2 upgrade.
Its helper setComments builds a new Column and discards the result instead of replacing the element in the list:
private void setComments(List<Column> columns, Map<String, Option<String>> commentsMap) {
columns.forEach(column -> {
String comment = commentsMap.getOrDefault(column.name(), Option.empty()).orElse(null);
Column.builder().comment(comment).build(); // result dropped, column is unchanged
});
}
AWS SDK v2 model classes are immutable, so mutating in place is not possible. Before the SDK v2 upgrade (e9dd73f, #9347) this code called column.setComment(...) on the mutable v1 model, which worked. As a result updateTableComments never detects a change and always returns false, and no comments are synced to Glue on the update path.
The existing Glue comment tests do not catch this because they only assert comments that were pre-baked into the mocked Column objects.
Expected behavior
With hoodie.datasource.hive_sync.sync_comment=true, column and partition column comments should be applied to the Glue table.
Possible fix
Rebuild the column lists with the comment applied, e.g.:
for (int i = 0; i < columns.size(); i++) {
Column column = columns.get(i);
String comment = commentsMap.getOrDefault(column.name(), Option.empty()).orElse(null);
columns.set(i, column.toBuilder().comment(comment).build());
}
(using mutable copies of storageDescriptor.columns() / table.partitionKeys(), which are returned as unmodifiable lists by the SDK), and update the tests to assert comments that were not already present on the input columns.
Environment Description
Additional context
Found during review of #19289, which fixes the equivalent HMS paths. Related to HUDI-8843 / #17359.
Describe the problem you faced
AWSGlueCatalogSyncClient.updateTableCommentsno longer applies any column or partition column comments since the AWS SDK v2 upgrade.Its helper
setCommentsbuilds a newColumnand discards the result instead of replacing the element in the list:AWS SDK v2 model classes are immutable, so mutating in place is not possible. Before the SDK v2 upgrade (e9dd73f, #9347) this code called
column.setComment(...)on the mutable v1 model, which worked. As a resultupdateTableCommentsnever detects a change and always returnsfalse, and no comments are synced to Glue on the update path.The existing Glue comment tests do not catch this because they only assert comments that were pre-baked into the mocked
Columnobjects.Expected behavior
With
hoodie.datasource.hive_sync.sync_comment=true, column and partition column comments should be applied to the Glue table.Possible fix
Rebuild the column lists with the comment applied, e.g.:
(using mutable copies of
storageDescriptor.columns()/table.partitionKeys(), which are returned as unmodifiable lists by the SDK), and update the tests to assert comments that were not already present on the input columns.Environment Description
Additional context
Found during review of #19289, which fixes the equivalent HMS paths. Related to HUDI-8843 / #17359.