Release-0.6.0 roadmap

[zhoujinsong]

This issue will track the roadmap of Amoro 0.6.0

Feature Development

• [Feature]: Support integration with Apache Paimon #1269
  - [Subtask]: Extract a common interface to support more table format. #1846
  - [Subtask]: Refactor code of "Table" and "Catalog" to support Paimon format integration. #1906
  - [Subtask]: Support registering for the Paimon Catalog. #1871
  - [Subtask]: Support operating the Paimon table in Terminal. #1872
  - [Subtask]: Support displaying Paimon lists on the Tables page. #1873
  - [Subtask]: Support display paimon table information in table details page #1874
  - [Subtask]: Support display paimon files in 'files' page #1875
  - [Subtask]: Support display paimon snapshots in 'Transactions' page #1876
  - [Subtask]: Dynamic chart in 'transactions' page #2118
  - [Subtask]: Support display paimon Operations in 'operations' page #1877
  - [Subtask]: Support display paimon optimizing information in optimizing page #1878
  - [Improvement] Polish the code to obtain the Paimon table compaction information #2169
• [Feature]: Integration with Kubernetes #1917
  - [Feature]: Support Kubernetes deployment for AMS #1335
  - [Feature]: Make FlinkOptimizerContainer to support running Optimizer on K8s. #508
  - [Improvement]: Documents about intergration with kubernetes #1918
  - [Subtask][Imporvement]: Helm script supports FlinkOptimizerContainer #2054
  - [Improvement]: Ingress configration for ams when deploying with Helm. #2020
• [Feature]: Official Amoro Docker Image. #2033
• [Feature]: Enrich the storage type registry when creating a catalog. #2007
  - [Subtask]: Support storage type Hadoop and S3 and authentication type AK/SK for Catalog #2084
  - [Subtask]: Dashboard Catalog page supports automatically filling of required properties #2099
  - [Subtask]: Support hiding some Catalog properties #2102
• [Feature]: Support for expiring partitions in Iceberg tables #1758
• [Feature]: Support ORC file format for Mixed format tables #740
• [Feature]: Upgrading Flink version based on Mixed-format table #1983
  - [Subtask]: Remove Flink 1.14 modules #2093
  - [Subtask]: Remove Flink 1.12 modules #2059
• [Improvement]: Add Plugin and PluginManager interface and implementation for better abstraction and extension #2041
  - [Subtask]: add PluginManager interface and SpiPluginManager implementation #2042
  - [Subtask]: ActivePluginManager interface and implementation #2065
  - [Subtask]: Metrics interface and MetricsManager Inheritted from ActivePluginManager #2069
• [Feature]: AMS Support Database Secert On K8S #1945
• [Subtask]: Upgrade the version description of Amoro Flink documentation #2068
• [Feature]: [Flink] Flink supports 1.16 version #2027
• [Feature]: New MetricsEmitter loading mechanism #2100

Improvement

• [Improvement]: modify the parameter transfer method for flink optimize #1789
• [Improvement]: Separate local-optimizer and flink-optimizer into different modules and package them independently. #2008
• [Improvement]: Support s3:// and s3a:// protocols in the default distribution. #2009
• [Improvement]: Spotless Maven Plugin Integration #1402
  - [Improvement]: Add basic configration of spotless #1926
  - [Subtask]: Apply spotless to Spark Modules #2017
  - [Subtask]: Apply spotless plugin to Flink modules #2018
  - [Imporvement]: Add spotless:validate to core and hive module #2108
  - [Subtask]: Apply spotless to Trino Modules #2113
  - [Imporvement]: Add spotless:validate to ams/api and ams/optimizer modules #2170
• [Improvement]: Make health status checks available without logging in #2081
• [Subtask]: Fix Flink LookUpJoin TestLookup Test Failed #2061
• [Improvement]: Replace commandline banner from Javalin to Amoro #2056
• [Subtask]: Flink Kafka unit testing improvements #2023
• [Improvement]: Support s3:// and s3a:// protocols in the default distribution. #2009
• [Improvement]: Separate local-optimizer and flink-optimizer into different modules and package them independently. #2008
• [Improvement]: Support list partitions via flink catalog #1940
• [Improvement]: Docker Images for quick-start are too large #868
• [Improvement]: Add conf/env.sh for jvm options configurations #2047
• [Improvement]: Change position delete index from set to bitmap of optimizing #2078
• [Subtask]: Remove dependency on mysql:mysql-connector-java #2036
• [Subtask]: Remove dependency on org.openjdk.jol:jol-core #2037
• [Improvement]: Critical vulnerabilities in the project dependencies need to be fixed #2040
• [Improvement] Polish code: fix typo and add unit test and add document #2164
• [Improvement] Fix double-checked locking field is not declared volatile #2153
• [Improvement]: Add TableFormat information to the return value of the Catalog listTable method. #2150
• [Improvement]: Support list partitions by filter via flink catalog #2111
• [Improvement]: Make AMS table sync with external catalogs multithreaded #2109
• [Polish] Code polish #2105
• [Workflow]: Auto build Official Docker Image #2136
• [CI] Limit concurrency of workflow #2138
• [Build] Remove duplicated dependency #2132
• [Improvement]Remove the unnecessary classes copied from the Iceberg project #2106

Bug Tracking

• [Bug]: The new fields are not compatible with the data of the old version #2097
• [Bug]: Failed to optimize Mixed Format KeyedTable with Timestamp(without zone) column as Primary key  #2091
• [Bug]: Log4j 1.2 is a SocketServer class that is vulnerable to deserialization of untrusted data #1548
• [Bug]: An infinite loop occurs in merge optimization, and the number of input files is equal to the number of output files. #2090
• [Hotfix] Fix typo in config.yaml example #2140
• [Bug]: Get incorrect partition field name when refresh tables #2165
• [Subtask]: Dashboard Catalog page supports automatically filling of required properties #2099
• [Fix] Fix the format of thread name in DefaultTableService #2163
• [DOC][hotfix] Remove redundant whitespace characters #2154
• [fix] specify options various is class field;use ThreadLocal remove instead of set null #2143
• [hotfix] Catalog type always null in TerminalSession #2146
• [Bug]: ams server can not initialized on HA mode #2083
• [Hotfix] Fix the location of metric emitter configuration file #2137
• [Bug]: Throws NPE if the data expiration config is not serialized in the table configuration #2208
• [Bug]: Flink optimizer scales out interaction error #2196
• [Bug]: The S3 related configurations not working when setting S3 as storage type #2192
• [Fix] Add hive-conf-dir option to paimon catalog #2187
• [Hotfix]Drop database after table test finished #2210
• [Bug]: Unable to configure JVM parameters during Helm deployment. #2190
• [AMORO-2190]: Fix unable to configure JVM parameters during Helm deployment #2202
• [Fix] Fix using the wrong summary map #2186
• [Improvement]: Change the default value of self-optimizing.major.trigger.duplicate-ratio #2225
• [docs] Update some images in docs to version 0.6.0 of Amoro #2231
• [Bug]: Optimizing process failed for Iceberg format table on MinIO #2223
• [Bug]: The table detail page does not display the change store metrics of Mixed KeyedTable #2227
• [AMORO-2168]:Fix-2183 FileNameRules adapt to hidden filename. #2229
• [Bug]: Throws NPE if the data expiration config is not serialized in the table configuration #2208
• [Hotfix]Delete sync tables when drop external catalogs #2235
• [AMORO-2253] Mixed Format optimized-sequence should not contains the skipped partitions #2249

Document Improvement

• [DOC] Fixed a counting error #2247
• [Doc] Add docs for paimon format #2159

[zhoujinsong]

Closed because version 0.6.0 has already been released: https://github.com/NetEase/amoro/releases/tag/v0.6.0