[AMORO-2115] Support displaying Optimizing tasks #2322

wangtaohz · 2023-11-18T02:36:37Z

Why are the changes needed?

Close #2115 .

Brief change log

add optimizing detail page with all optimizing taskes
get optimizing taskes http interface return optimizer token

How was this patch tested?

Add some test cases that check the changes thoroughly including negative and positive cases if possible
Add screenshots for manual tests if appropriate
Run test locally before making a pull request

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

wangtaohz · 2023-11-18T02:37:35Z

wangtaohz · 2023-11-18T02:41:40Z

wangtaohz · 2023-11-18T02:42:21Z

codecov · 2023-11-18T02:43:02Z

Codecov Report

Attention: 69 lines in your changes are missing coverage. Please review.

Comparison is base (24dbeed) 53.14% compared to head (15e0087) 53.16%.

Files	Patch %	Lines
...tease/arctic/server/optimizing/MetricsSummary.java	63.21%	32 Missing ⚠️
...tic/server/dashboard/model/OptimizingTaskInfo.java	0.00%	22 Missing ⚠️
...c/server/dashboard/controller/TableController.java	0.00%	5 Missing ⚠️
...rver/dashboard/MixedAndIcebergTableDescriptor.java	0.00%	4 Missing ⚠️
.../server/dashboard/model/OptimizingProcessInfo.java	0.00%	3 Missing ⚠️
.../arctic/server/dashboard/utils/OptimizingUtil.java	0.00%	2 Missing ⚠️
...arctic/server/dashboard/PaimonTableDescriptor.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #2322      +/-   ##
============================================
+ Coverage     53.14%   53.16%   +0.02%     
- Complexity     3785     3796      +11     
============================================
  Files           488      488              
  Lines         25641    25712      +71     
  Branches       2450     2453       +3     
============================================
+ Hits          13628    13671      +43     
- Misses        10953    10982      +29     
+ Partials       1060     1059       -1

Flag	Coverage Δ
core	`53.16% <51.06%> (+0.02%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

wangtaohz · 2023-11-18T02:44:59Z

shidayang · 2023-11-21T02:41:53Z

In the task details, I found some information that is not very clear. Could you please prefix the newly added files with "new" and the old files with "reWrite"?

zhongqishang · 2023-11-21T03:27:07Z

In the task details, I found some information that is not very clear. Could you please prefix the newly added files with "new" and the old files with "reWrite"?

@wangtaohz I suggest use new define parameter to display in page, this is also the reason why I added new parameters.

ams/server/src/main/java/com/netease/arctic/server/optimizing/TaskRuntime.java

…TaskRuntime.java Co-authored-by: Qishang Zhong <zhongqishang@gmail.com>

…timizing-page

wangtaohz · 2023-11-21T06:59:36Z

In the task details, I found some information that is not very clear. Could you please prefix the newly added files with "new" and the old files with "reWrite"?

Thanks for your report. After merging this PR, and modifying the order of metrics, the new metrics are more clear.

wangtaohz · 2023-11-21T07:01:38Z

@wangtaohz I suggest use new define parameter to display in page, this is also the reason why I added new parameters.

All the new define parameters are displayed now. You can refer to my last comment.

zhongqishang · 2023-11-21T08:40:32Z

@wangtaohz

Is it still necessary to display the deprecated parameters (reRowDeletedDataFileCnt, positionalDeleteSize, newFileSize, newFileCnt) on the page?
It will cause some doubts to users, I recommend not display them.

wangtaohz · 2023-11-22T07:35:01Z

There are two reasons for displaying the old parameters here.
Firstly, since the new parameters of the old task are all 0, omitting the display of old parameters would result in the loss of some information for the old task.
Secondly, I am not very familiar with how to remove these parameters on the front-end. 😅

tcodehuber · 2023-11-22T08:19:58Z

Looks great. Can we add an action to kill/cancel the running process in the optimizing process detail page?

zhoujinsong

@wangtaohz I left a comment, please take a look.

zhoujinsong · 2023-12-11T12:35:24Z

ams/server/src/main/java/com/netease/arctic/server/optimizing/TaskRuntime.java

@@ -345,7 +347,10 @@ private Set<Status> getNext() {
    }

    public synchronized boolean tryAccepting(Status targetStatus) {
-      if (owner.isClosed() || !getNext().contains(targetStatus)) {
+      if (owner.isClosed() && targetStatus != Status.CANCELED) {


I'm not sure if treating CANCELED as a special status here is a good approach, or if we should include CANCELED in the next status list for all possible statuses to express this logic.

The owner here is the optimizing process. When I try to close the optimizing process and its tasks, the optimizing process is set to be Closed first, and then the tasks are rejected changing the status to CANCELED.

zhongqishang · 2023-12-12T05:14:51Z

@wangtaohz After I canceled the optimize task, tasks without start time showed abnormal cost time values.

wangtaohz · 2023-12-12T05:41:17Z

@wangtaohz After I canceled the optimize task, tasks without start time showed abnormal cost time values.

Thanks for your report. I will fix it.

wangtaohz · 2023-12-12T06:42:15Z

2023-12-12.14.38.49.mov

zhongqishang

I left some comments.

ams/server/src/main/java/com/netease/arctic/server/dashboard/controller/TableController.java

ams/server/src/main/java/com/netease/arctic/server/optimizing/TaskRuntime.java

This reverts commit e469e71.

zhongqishang

LGTM.

zhoujinsong

LGTM.

* dashboard: rename optimized to optimizing * dashboard: support optimizing taskes * add optimizer token * dashboard: modify column width * dashboard: build * sort the metrics field and change record cnt to long * modify MetricsSummary Compatibility * dashbard: build * Update ams/server/src/main/java/com/netease/arctic/server/optimizing/TaskRuntime.java Co-authored-by: Qishang Zhong <zhongqishang@gmail.com> * fix * support input metrics and output metrics for optimizing process * dashboard: support optimizing metrics * dashbard: build * dashboard:rebuild * support MetricsSummary to map * optimizing task supports input output * dashboard: optimizing tasks support input and output * dashboard: not display seconds when longer than 1 hour * dashboard: optimizing process show summary * remove useless import * dashboard: build * as head * dashbard: build * change process status to CLOSED after cancel process * remove useless log * dashboard: refresh after cancelled * support cancel optimizing tasks * dashboard: handle exception when can't cancel optimizing process * throw exception when can't cancel optimizing process * dashboard: build * dashboard: refresh optimizing process when exist optimizing detail page * dashboard: build * fix cost time is 0ms * change metrics name * fix task startTime and endTime * fix costTime * using Preconditions.checkArgument * fix task reset * add comments * cancel tasks before closing optimizing process * fix unit test * fix cancel task * as head * Revert "as head" This reverts commit e469e71. * dashboard: build --------- Co-authored-by: Qishang Zhong <zhongqishang@gmail.com>

* [AMORO-1812] support spark-based external optimizer * resolve code style error * [AMORO-1951] Support parallelized planning in one optimizer group (#2282) * [AMORO-1951] Support parallelized planning in one optimizer group * [AMORO-1951] add unit test for OptimizingQueue and DefaultOptimizingService * [AMORO-1951] optimize default parameters * fix bugs * fix warnings and spotless issues * merge from #2290 * add apache license and fix spotless * fix config error * Update ams/server/src/main/java/com/netease/arctic/server/DefaultOptimizingService.java Co-authored-by: ZhouJinsong <zhoujinsong0505@163.com> * add annotations * fix compile errors * fix import problem * remove isDebugEnabled() * spotless apply * Update ArcticManagementConf.java * fix reboot bug and supply document content * use MoreObjects.toStringHelper for OptimizerThread.java * Merged from [AMORO-2376] Print right log info after calculating and sorting tables * fix import problem * remove unused codes * spotless * remove incorrect comments * add max-planning-parallelism to config --------- Co-authored-by: majin1102 <majin1102@163.com> Co-authored-by: ZhouJinsong <zhoujinsong0505@163.com> * [AMORO-2378] The optimizer based on Flink on YARN should prioritize loading the optimizer-job.jar (#2379) * load optimizer jar first * fix code style * change config name * add config taskmanager.memory.managed.fraction * fix * [AMORO-2222] [Improvement]: Skip cleaning up dangling delete files for Iceberg V1 table (#2361) * [AMORO-2222] [Improvement]: Skip cleaning up dangling delete files for Iceberg V1 table * Update IcebergTableMaintainer.java The `total-delete-files` could be 0. --------- Co-authored-by: wangtaohz <103108928+wangtaohz@users.noreply.github.com> * [AMORO-2404] fix Mixed Hive table mistakenly deletes hive files during expiring snapshots (#2405) get hive locations return the uri path * [AMORO-2407] Fix access data file from dashboard of non-partitioned table (#2408) * fix null partition * fix listing files of non-partitioned iceberg table * [AMORO-2383] Add serialVersionUID to RewriteFilesInput (#2384) * add serialVersionUID * fix comment * [AMORO-1720] Fix Mixed Format KeyedTable expiring all the snapshots with optimized sequence (#2394) * should not expire the latest snapshot contains optimized sequence * add visible for testing * add fetchLatestNonOptimizedSnapshotTime for base store * get hive locations return the uri path * refactor codes and fix comments * improve for exclude files is empty for expring snapshots --------- Co-authored-by: ZhouJinsong <zhoujinsong0505@163.com> * [AMORO-2386][AMS] Configure `iceberg.worker.num-threads` in the config.yaml (#2393) * [AMORO-2386][AMS] Configure `iceberg.worker.num-threads` in the config.yaml * Fix * [AMORO-2386][AMS] reuse config `table-manifest-io.thread-count` and reuse thread pool * Add comment * [AMORO-1716] [Improvement]: sort the table list returned by server (#2362) * improve: sort the table list returned by server * optimize: sort tables by format * optimize: optimiz tables sorting * style: udpate comment --------- Co-authored-by: chenyuzhi <chenyuzhi@corp.netease.com> Co-authored-by: ZhouJinsong <zhoujinsong0505@163.com> * [HotFix] Re-add table-filter to Server ExternalCatalog (#2310) * re add table filter * implement in external catalog * add ut case * fix comment * fix comment * fix comment * fix ut * fix update properties * roll back the engine side's filter * resolve conflicts * add ut --------- Co-authored-by: baiyangtx <xiangnebula@163.com> Co-authored-by: ZhouJinsong <zhoujinsong0505@163.com> * [AMORO-2299]: Cancel the running optimizing process from ams web console (#2297) * cancel the running opimizing process from ams web console * refact code to avoid NPE * add o comment for com.netease.arctic.server.table.TableService#getServerTableIdentifier * change the cancel post api to be more restful style * [AMORO-2415] Print GC date stamps (#2416) add gc timestamp * Update wrong comments in SnapshotsExpiringExecutor.java (#2422) * [AMORO-2276]: UnifiiedCatalog for Spark Engine (#2269) * Add UnifiedSparkCatalog under spark common module * Extract MixedSparkCatalogBase and MixedSparkSessionCatalogBase to spark common module * Refactor spark unit test framework to adapt unifed catalog tests and mixed format tests. * [AMORO-2261] Extract the deleting dangling files from the cleaning orphan files (#2403) * [Improvement]: Extract the deleting dangling files from the cleaning orphan files * [Improvement]: Extract the deleting dangling files from the cleaning orphan files * [Improvement]: Extract the deleting dangling files from the cleaning orphan files * [AMORO-1341] [Flink]: Support UnifiedCatalog to contain Mixed format table in Flink Engine (#2370) * [AMORO-1341] [Flink]: Support UnifiedCatalog to contain Mixed format table in Flink Engine * [AMORO-2413] Need to select the first db after switching to another Catalog (#2419) * fix: If the current catalog is not the one in the query, the first db is selected by default. * build dashboard frontend --------- Co-authored-by: wangtao <wangtao3@corp.netease.com> * [HotFix] Fix loading the optimizing snapshot id of change store for Mixed Format KeyedTable (#2430) fix load target change snapshot id * [AMORO-2260] Show the format version of iceberg table (#2425) [AMORO-2260] Show the format version of Iceberg Table Signed-off-by: tcodehuber <tcodehuber@gmail.com> * [AMORO-2115] Support displaying Optimizing tasks (#2322) * dashboard: rename optimized to optimizing * dashboard: support optimizing taskes * add optimizer token * dashboard: modify column width * dashboard: build * sort the metrics field and change record cnt to long * modify MetricsSummary Compatibility * dashbard: build * Update ams/server/src/main/java/com/netease/arctic/server/optimizing/TaskRuntime.java Co-authored-by: Qishang Zhong <zhongqishang@gmail.com> * fix * support input metrics and output metrics for optimizing process * dashboard: support optimizing metrics * dashbard: build * dashboard:rebuild * support MetricsSummary to map * optimizing task supports input output * dashboard: optimizing tasks support input and output * dashboard: not display seconds when longer than 1 hour * dashboard: optimizing process show summary * remove useless import * dashboard: build * as head * dashbard: build * change process status to CLOSED after cancel process * remove useless log * dashboard: refresh after cancelled * support cancel optimizing tasks * dashboard: handle exception when can't cancel optimizing process * throw exception when can't cancel optimizing process * dashboard: build * dashboard: refresh optimizing process when exist optimizing detail page * dashboard: build * fix cost time is 0ms * change metrics name * fix task startTime and endTime * fix costTime * using Preconditions.checkArgument * fix task reset * add comments * cancel tasks before closing optimizing process * fix unit test * fix cancel task * as head * Revert "as head" This reverts commit e469e71. * dashboard: build --------- Co-authored-by: Qishang Zhong <zhongqishang@gmail.com> * [AMORO-2385] Make the maximum input file size for per optimize thread configurable (#2387) * add config self-optimizing.max-input-file-size-per-thread * add doc * add resource group property max-input-file-size-per-thread * add doc * fix compile * [Hotfix] Add database filter to Server ExternalCatalog (#2414) * [Hotfix] Add database filter to Server ExternalCatalog * [Hotfix] Add database filter to Server ExternalCatalog * Rename config database.filter-regular-expression to database-filter --------- Co-authored-by: baiyangtx <xiangnebula@163.com> * [AMORO-2423] [Flink]: Using 'mixed_iceberg' and 'mixed_hive' indentifier to CREATE CATALOG and deprecate 'arctic' identifier (#2424) * [AMORO-2423] [Flink]: Using 'mixed_iceberg' and 'mixed_hive' identifiers to CREATE CATALOG and deprecate 'arctic' identifier * [AMORO-2316] The Files page supports filtering by partition name and sorting by dictionary value. (#2420) * AMORO-2316: The Files page supports filtering by partition name and sorting by dictionary value. * build dashboard frontend * dashboard: build --------- Co-authored-by: wangtao <wangtao3@corp.netease.com> * [AMORO-1892] Improve the SQL Shortcuts in Terminal Web UI (#2434) * [AMORO-1892] Improve the SQL Shortcuts in Terminal Web UI * refactor some code --------- Signed-off-by: tcodehuber <tcodehuber@gmail.com> * [AMORO-2440] Fix the batch deletion of Change Store files for Mixed Format Table (#2439) fix remove change files * [AMORO-2418] Exclude kryo dependency from flink-optimizer (#2437) exclude kryo * [AMORO-2441] Fix `TableEntriesScan` when no file format is specified in the file name (#2442) * fix TableEntriesScan without format suffix * using the file format from entries * [Hotfix] Fix fetchLatestNonOptimizedSnapshotTime (#2396) * fix fetchLatestNonOptimizedSnapshotTime * fix * spotless * fix ut * rename data-expire.since * using UTC zone for snapshot timestamp * resolve conflict * rerview * spotless * review * review * [AMORO-2344] [Flink]: Support UnifiedCatalog to contain Iceberg format table in Flink Engine (#2427) * [AMORO-2330] Improve major plan (#2332) * [AMORO-2330][AMS] Improve major plan * [AMORO-2330][AMS] remove filter dataFileWith1Pos * Fix comments * Fix * Move the rollback logic of undersized segment to the split task stage * Rename * Fix * Rollback method name `fileShouldRewrite` * Rollback mixed table format full trigger condition & reuse `isUndersizedSegmentFile` * Rollback testSegmentFilesBase() * TreeNodeTaskSplitter logic keep same with bin-pack * Improve code duplicate --------- Co-authored-by: ZhouJinsong <zhoujinsong0505@163.com> * [AMORO-1810] Check the validity of the heatbeat interval when an opti… (#2432) * [AMORO-1810] Check the validity of the heatbeat interval when an optimizer start * adjust the import way * resolve some logic code * refactor code * refactor code * fix ut error * resolve ut error * fix ut error * fix ut error * fix ut error * fix ci error * refactor code * refactor code * refactor code * [AMORO-1812] support spark-based external optimizer * resolve code style error * refactor code * refactor code * bugfix * refactor code * refactor code * code style * bugfix * bugfix --------- Signed-off-by: tcodehuber <tcodehuber@gmail.com> Co-authored-by: JinMat <majin1102@gmail.com> Co-authored-by: majin1102 <majin1102@163.com> Co-authored-by: ZhouJinsong <zhoujinsong0505@163.com> Co-authored-by: wangzeyu <hameizi369@gmail.com> Co-authored-by: ConradJam <jam.gzczy@gmail.com> Co-authored-by: wangtaohz <103108928+wangtaohz@users.noreply.github.com> Co-authored-by: yeatsliao <liaoyt66066@gmail.com> Co-authored-by: Qishang Zhong <zhongqishang@gmail.com> Co-authored-by: chenyuzhi459 <553673833@qq.com> Co-authored-by: chenyuzhi <chenyuzhi@corp.netease.com> Co-authored-by: HuangFru <68625618+HuangFru@users.noreply.github.com> Co-authored-by: baiyangtx <xiangnebula@163.com> Co-authored-by: xujiangfeng001 <104614523+xujiangfeng001@users.noreply.github.com> Co-authored-by: Xianxun Ye <yesorno828423@gmail.com> Co-authored-by: liuweimin <minteliu.l@gmail.com> Co-authored-by: wangtao <wangtao3@corp.netease.com> Co-authored-by: Xavier Bai <xuba@cisco.com>

wangtaohz added 5 commits November 17, 2023 22:36

dashboard: rename optimized to optimizing

fc81967

dashboard: support optimizing taskes

ebe745e

add optimizer token

f757da0

dashboard: modify column width

1247bfe

dashboard: build

3332948

github-actions bot added the module:ams-dashboard Ams dashboard module label Nov 18, 2023

wangtaohz changed the title ~~[AMORO-2115] Support displaying Optimizing taskes~~ [AMORO-2115] Support displaying Optimizing tasks Nov 20, 2023

wangtaohz and others added 6 commits November 20, 2023 13:57

Merge branch 'master' into optimizing-page

b9d0b93

Merge branch 'master' into optimizing-page

47d3c7d

sort the metrics field and change record cnt to long

875e4f8

modify MetricsSummary Compatibility

598ad15

Merge branch 'master' into optimizing-page

a92d509

dashbard: build

eb12796

zhongqishang reviewed Nov 21, 2023

View reviewed changes

ams/server/src/main/java/com/netease/arctic/server/optimizing/TaskRuntime.java Outdated Show resolved Hide resolved

wangtaohz and others added 3 commits November 21, 2023 14:52

Update ams/server/src/main/java/com/netease/arctic/server/optimizing/…

15c65d4

…TaskRuntime.java Co-authored-by: Qishang Zhong <zhongqishang@gmail.com>

Merge branch 'optimizing-page' of github.com:wangtaohz/arctic into op…

dad9754

…timizing-page

fix

f299ff6

Merge branch 'master' into optimizing-page

c0deca8

wangtaohz and others added 4 commits December 11, 2023 17:10

Merge branch 'master' into optimizing-page

29644e9

dashboard: refresh optimizing process when exist optimizing detail page

95a1330

dashboard: build

dc8f234

fix cost time is 0ms

51af0a5

zhoujinsong reviewed Dec 11, 2023

View reviewed changes

change metrics name

56a5e1a

wangtaohz added 2 commits December 12, 2023 14:14

fix task startTime and endTime

5cd6289

fix costTime

e0c3947

zhongqishang reviewed Dec 12, 2023

View reviewed changes

wangtaohz and others added 11 commits December 12, 2023 21:12

using Preconditions.checkArgument

b86121b

fix task reset

a22523d

add comments

0382167

cancel tasks before closing optimizing process

b294b2b

Merge branch 'master' into optimizing-page

b9e04c3

fix unit test

acb6087

fix cancel task

cae0414

as head

e469e71

Revert "as head"

a1689de

This reverts commit e469e71.

Merge remote-tracking branch 'upstream/master' into optimizing-page

326305d

dashboard: build

15e0087

zhongqishang approved these changes Dec 14, 2023

View reviewed changes

zhoujinsong approved these changes Dec 14, 2023

View reviewed changes

zhoujinsong merged commit ffefda2 into apache:master Dec 14, 2023
6 of 7 checks passed

wangtaohz deleted the optimizing-page branch December 14, 2023 05:48

zhoujinsong mentioned this pull request Jun 25, 2024

Release-0.7.0 roadmap #2176

Closed

66 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[AMORO-2115] Support displaying Optimizing tasks #2322

[AMORO-2115] Support displaying Optimizing tasks #2322

wangtaohz commented Nov 18, 2023

wangtaohz commented Nov 18, 2023 •

edited

Loading

wangtaohz commented Nov 18, 2023

wangtaohz commented Nov 18, 2023

codecov bot commented Nov 18, 2023 •

edited

Loading

wangtaohz commented Nov 18, 2023

shidayang commented Nov 21, 2023

zhongqishang commented Nov 21, 2023

wangtaohz commented Nov 21, 2023 •

edited

Loading

wangtaohz commented Nov 21, 2023

zhongqishang commented Nov 21, 2023

wangtaohz commented Nov 22, 2023

tcodehuber commented Nov 22, 2023 •

edited

Loading

zhoujinsong left a comment

zhoujinsong Dec 11, 2023

wangtaohz Dec 11, 2023

zhongqishang commented Dec 12, 2023

wangtaohz commented Dec 12, 2023

wangtaohz commented Dec 12, 2023

zhongqishang left a comment

zhongqishang left a comment

zhoujinsong left a comment

[AMORO-2115] Support displaying Optimizing tasks #2322

[AMORO-2115] Support displaying Optimizing tasks #2322

Conversation

wangtaohz commented Nov 18, 2023

Why are the changes needed?

Brief change log

How was this patch tested?

Documentation

wangtaohz commented Nov 18, 2023 • edited Loading

wangtaohz commented Nov 18, 2023

wangtaohz commented Nov 18, 2023

codecov bot commented Nov 18, 2023 • edited Loading

Codecov Report

wangtaohz commented Nov 18, 2023

shidayang commented Nov 21, 2023

zhongqishang commented Nov 21, 2023

wangtaohz commented Nov 21, 2023 • edited Loading

wangtaohz commented Nov 21, 2023

zhongqishang commented Nov 21, 2023

wangtaohz commented Nov 22, 2023

tcodehuber commented Nov 22, 2023 • edited Loading

zhoujinsong left a comment

Choose a reason for hiding this comment

zhoujinsong Dec 11, 2023

Choose a reason for hiding this comment

wangtaohz Dec 11, 2023

Choose a reason for hiding this comment

zhongqishang commented Dec 12, 2023

wangtaohz commented Dec 12, 2023

wangtaohz commented Dec 12, 2023

zhongqishang left a comment

Choose a reason for hiding this comment

zhongqishang left a comment

Choose a reason for hiding this comment

zhoujinsong left a comment

Choose a reason for hiding this comment

wangtaohz commented Nov 18, 2023 •

edited

Loading

codecov bot commented Nov 18, 2023 •

edited

Loading

wangtaohz commented Nov 21, 2023 •

edited

Loading

tcodehuber commented Nov 22, 2023 •

edited

Loading