v0.51.0-public.rc1

Changelog

[0.51.0-public.rc1] - 2024-11-22

Features and Changes

• Column ColumnBuilder adds a new withGenerateExpression method for constructing auto-partition columns
• TableSchema
  • Added generatePartitionSpec method, used to generate partition information from Record
  • The setPartitionColumns method now accepts List<Column> instead of ArrayList<Column>
• TableCreator
  • Added support for GenerateExpression and introduced the method autoPartitionBy, which allows for the creation of AutoPartition tables.
  • Added support for ClusterInfo, enabling the creation of Hash/Range Cluster tables.
  • Added the option to specify TableFormat, allowing for the creation of tables in APPEND, TRANSACTION, DELTA, EXTERNAL, and VIEW formats.
  • Introduced the selectStatement parameter for create table as and create view as scenarios.
  • Added the getSql method to obtain the SQL statement for table creation.
  • Now quotes all Comment parameters to support those that contain special characters.
  • Integrated DataHub-related table creation parameters (hubLifecycle, shardNum) into DataHubInfo.
  • Renamed the withJars method to withResources to indicate it can use resources other than JAR files.
  • Renamed the withBucketNum method to withDeltaTableBucketNum to indicate this method is for Delta Tables only.
  • Modified the logic of withHints, withAlias, withTblProperties, and withSerdeProperties methods, now overwriting previous values instead of merging.
  • Removed the createExternal method; you can now use the create method instead.
• Table
  • Introduced the getSchemaVersion method, allowing users to retrieve the current schema version of the table. The version number is updated each time a Schema Evolution occurs, and this field is used primarily for specifying when creating a StreamTunnel.
  • Added setLifeCycle, changeOwner, changeComment, touch, changeClusterInfo, rename, addColumns, dropColumns methods to support modification of table structure.
• StreamTunnel Modified the initialization logic; if allowSchemaMismatch is set to false, it will automatically retry until the latest version of the table structure is used (with a timeout of 5 minutes).

Fixes

• GenerationExpression Fixed the issue where an exception would be thrown when the TruncTime was uppercase during table creation and reloading the table.
• TypeInfoParser Can now correctly handle Struct types, with fields quoted using backticks in TypeInfo.

---

更新日志

[0.51.0-public.rc1] - 2024-11-22

功能与变更

• Column ColumnBuilder 新增 withGenerateExpression 方法，用于构造 auto-partition 列
• TableSchema
  • 新增 generatePartitionSpec 方法，用于从Record中生成分区信息
  • setPartitionColumns 方法现在接收List<Column>，而不是ArrayList<Column>
• TableCreator
  • 新增对GenerateExpression的支持，新增方法autoPartitionBy，现在可以创建 AutoPartition 表了
  • 新增对ClusterInfo的支持，现在可以创建 Hash/Range Cluster 表了
  • 新增指定 TableFormat，现在可以指定创建APPEND,TRANSACTION,DELTA,EXTERNAL,VIEW格式的表
  • 新增selectStatement参数，用于create table as 和 create view as 场景
  • 新增getSql方法，用于获取创建表的 SQL 语句
  • 现在会对所有的 Comment 参数进行 quote，以支持包含特殊字符的 Comment 参数
  • 将 DataHub 相关的建表参数（hubLifecycle, shardNum) 整合为 DataHubInfo
  • 重命名withJars方法为withResources，以表示不仅可以使用JAR类型资源
  • 重命名withBucketNum方法为withDeltaTableBucketNum，以表示该方法仅用于 Delta Table
  • 修改了 withHints，withAlias，withTblProperties，withSerdeProperties 方法的逻辑，现在会覆盖之前设置的值，而不是合并
  • 移除了createExternal方法，现在使用create方法即可
• Table
  • 新增 getSchemaVersion 方法，用户获取当前表结构的版本，用户每次进行 SchemaEvolution 都会更新版本号，目前该字段仅用于在创建 StreamTunnel 时指定
  • 新增 setLifeCycle，changeOwner，changeComment，touch，changeClusterInfo，rename，addColumns，dropColumns方法，以支持对表结构进行修改
• StreamTunnel 修改初始化逻辑，当指定 allowSchemaMismatch 为 false 时，会自动重试直到使用最新版本的表结构(超时时间为5min)

修复

• GenerationExpression 修复了当建表时TruncTime为大写，reload table 会抛出异常的问题
• TypeInfoParser 能够正确处理 Struct 类型，字段被反引号quote的 TypeInfo了