Skip to content

[SUPPORT] AWSGlueCatalogSyncClient.updateTableComments applies no comments since AWS SDK v2 upgrade #19316

Description

@rangareddy

Describe the problem you faced

AWSGlueCatalogSyncClient.updateTableComments no longer applies any column or partition column comments since the AWS SDK v2 upgrade.

Its helper setComments builds a new Column and discards the result instead of replacing the element in the list:

private void setComments(List<Column> columns, Map<String, Option<String>> commentsMap) {
  columns.forEach(column -> {
    String comment = commentsMap.getOrDefault(column.name(), Option.empty()).orElse(null);
    Column.builder().comment(comment).build();   // result dropped, column is unchanged
  });
}

AWS SDK v2 model classes are immutable, so mutating in place is not possible. Before the SDK v2 upgrade (e9dd73f, #9347) this code called column.setComment(...) on the mutable v1 model, which worked. As a result updateTableComments never detects a change and always returns false, and no comments are synced to Glue on the update path.

The existing Glue comment tests do not catch this because they only assert comments that were pre-baked into the mocked Column objects.

Expected behavior

With hoodie.datasource.hive_sync.sync_comment=true, column and partition column comments should be applied to the Glue table.

Possible fix

Rebuild the column lists with the comment applied, e.g.:

for (int i = 0; i < columns.size(); i++) {
  Column column = columns.get(i);
  String comment = commentsMap.getOrDefault(column.name(), Option.empty()).orElse(null);
  columns.set(i, column.toBuilder().comment(comment).build());
}

(using mutable copies of storageDescriptor.columns() / table.partitionKeys(), which are returned as unmodifiable lists by the SDK), and update the tests to assert comments that were not already present on the input columns.

Environment Description

Additional context

Found during review of #19289, which fixes the equivalent HMS paths. Related to HUDI-8843 / #17359.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions