[ARCTIC-946][AMS] Add balanced schedule policy for optimizer #993

shidayang · 2023-01-10T08:16:31Z

Why are the changes needed?

linked: #946

Brief change log

Add config of optimizer group to determine schedule policy of optimization
Add some schedule policy in com.netease.arctic.ams.server.service.impl.OptimizeQueueService

How was this patch tested?

Add some test cases that check the changes thoroughly including negative and positive cases if possible
Add screenshots for manual tests if appropriate
Run test locally before making a pull request

Documentation

Does this pull request introduces a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

shidayang · 2023-01-10T09:26:19Z

Under the quota strategy, the stock table will not perform optimization for a long time.
In balanced policy it look like this:

I also ran a benchmark.

hzluting · 2023-01-11T01:46:00Z

ams/ams-server/src/main/java/com/netease/arctic/ams/server/config/ConfigFileProperties.java

@@ -58,6 +58,9 @@ public class ConfigFileProperties {
  public static final String OPTIMIZE_GROUP_NAME = "name";
  public static final String OPTIMIZE_GROUP_CONTAINER = "container";
  public static final String OPTIMIZE_GROUP_PROPERTIES = "properties";
+  public static final String OPTIMIZE_SCHEDULING_POLICY = "scheduling_policy";
+  public static final String OPTIMIZE_SCHEDULING_POLICY_QUOTA = "quota";


I think using Enum to define specific policies will be better.

The config system need to reconstruct, such as introduce ConfigOption class to resolve all config, but it is another work

hzluting · 2023-01-11T01:52:09Z

...ms-server/src/main/java/com/netease/arctic/ams/server/service/impl/OptimizeQueueService.java

+     * A policy that determines how to schedule a table.
+     * Tables at the head of the list have higher optimizing priority
+     */
+    interface SchedulePolicy {


The interface and the implements will be clear in structure if they are external classes.
Otherwise, the OptimizeQueueService class contains too much code.

I don't understand what you mean, could you explain in detail

I don't understand what you mean, could you explain in detail

I mean the interface SchedulePolicy and implement defined as external classes rather than the inner class will be better?

SchedulePolicy is customized for queue，I still think it would be better as an inner class.

zhoujinsong

@shidayang Thanks a lot for your contribution. The implementaton looks good to me.
But I still found some code that could probably be improved, please take another look.

ams/ams-server/src/main/java/com/netease/arctic/ams/server/optimize/IOptimizeService.java

...ms-server/src/main/java/com/netease/arctic/ams/server/service/impl/OptimizeQueueService.java

site/docs/ch/concepts/self-optimizing.md

zhoujinsong · 2023-01-11T02:15:45Z

Under the quota strategy, the stock table will not perform optimization for a long time. In balanced policy it look like this:

I also ran a benchmark.

Cool! There are a great improvement in performance after we introduce the balanced optimizing schedule policy.
Should we change the performance test result in the Benchmark chapter on site?
Of course, we can do it in another PR.

shidayang · 2023-01-11T02:29:36Z

Under the quota strategy, the stock table will not perform optimization for a long time. In balanced policy it look like this:
I also ran a benchmark.

Cool! There are a great improvement in performance after we introduce the balanced optimizing schedule policy. Should we change the performance test result in the Benchmark chapter on site? Of course, we can do it in another PR.

OK! I will change the performance test result in this pr

zhoujinsong

LGTM.

majin1102 · 2023-01-12T06:04:42Z

site/docs/ch/concepts/self-optimizing.md

-
-如果你使用的是不可更新的表，如日志，传感器数据，并且已经习惯于 Iceberg 提供的 optimize 指令，可以考虑通过下面的配置关闭表上的 self-optimizing 功能：
+## Self-optimizing scheduling policy
+scheduling policy 表示 AMS 在分发任务给 optimizer 时候的策略，可以决定本次调度哪些表的任务给 optimizer 执行。


Scheduling policy 是 AMS 决定不同表执行 self-optimizing 先后顺序的调度策略，通过不同的调度策略，决定了每张表实际可以占用的 self-optimizing 的资源，Arctic 用 Quota 定义每张表的预期资源用量，Quota occupy 代表了相比预期用量，实际占用的资源百分比。

下面建议再加一个 quota 和 quota occupy 的图片截图，说明 quota 和 quota occupy 的含义

majin1102

modify documents

* Add balanced schedule policy for optimizer * Fix timezone error of int96 predicate push down * Fix timezone error of int96 predicate push down

github-actions bot added module:ams-server Ams server module module:ams-dashboard Ams dashboard module type:docs Improvements or additions to documentation labels Jan 10, 2023

shidayang closed this Jan 10, 2023

shidayang reopened this Jan 10, 2023

shidayang requested review from majin1102, zhoujinsong, hzluting and wangtaohz January 10, 2023 09:39

hzluting approved these changes Jan 10, 2023

View reviewed changes

hzluting reviewed Jan 11, 2023

View reviewed changes

zhoujinsong reviewed Jan 11, 2023

View reviewed changes

github-actions bot added the module:mixed-hive Hive moduel for Mixed Format label Jan 11, 2023

Add balanced schedule policy for optimizer

bae116b

shidayang force-pushed the auto-quota branch from e1f5d6e to bae116b Compare January 11, 2023 06:09

github-actions bot removed the module:mixed-hive Hive moduel for Mixed Format label Jan 11, 2023

shidayang requested a review from zhoujinsong January 11, 2023 06:13

Fix timezone error of int96 predicate push down

79a983a

zhoujinsong approved these changes Jan 12, 2023

View reviewed changes

majin1102 reviewed Jan 12, 2023

View reviewed changes

majin1102 approved these changes Jan 12, 2023

View reviewed changes

majin1102 reviewed Jan 12, 2023

View reviewed changes

Fix timezone error of int96 predicate push down

03c50a5

zhoujinsong merged commit daf0def into apache:master Jan 13, 2023

zhoujinsong mentioned this pull request Feb 2, 2023

[Improvement]: Optimizer has a performance bottleneck #975

Closed

3 tasks

This was referenced Apr 21, 2023

Issue for arctic Aireed/imgs#1

Open

[Subtask]: We need to perform regression testing on historical issues. #1411

Closed

shidayang deleted the auto-quota branch October 24, 2023 13:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ARCTIC-946][AMS] Add balanced schedule policy for optimizer #993

[ARCTIC-946][AMS] Add balanced schedule policy for optimizer #993

shidayang commented Jan 10, 2023 •

edited

Loading

shidayang commented Jan 10, 2023

hzluting Jan 11, 2023

shidayang Jan 11, 2023

hzluting Jan 11, 2023

shidayang Jan 11, 2023 •

edited

Loading

hzluting Jan 11, 2023

shidayang Jan 11, 2023

zhoujinsong left a comment

zhoujinsong commented Jan 11, 2023 •

edited

Loading

shidayang commented Jan 11, 2023

zhoujinsong left a comment

majin1102 Jan 12, 2023

majin1102 Jan 12, 2023

majin1102 left a comment

[ARCTIC-946][AMS] Add balanced schedule policy for optimizer #993

[ARCTIC-946][AMS] Add balanced schedule policy for optimizer #993

Conversation

shidayang commented Jan 10, 2023 • edited Loading

Why are the changes needed?

Brief change log

How was this patch tested?

Documentation

shidayang commented Jan 10, 2023

hzluting Jan 11, 2023

Choose a reason for hiding this comment

shidayang Jan 11, 2023

Choose a reason for hiding this comment

hzluting Jan 11, 2023

Choose a reason for hiding this comment

shidayang Jan 11, 2023 • edited Loading

Choose a reason for hiding this comment

hzluting Jan 11, 2023

Choose a reason for hiding this comment

shidayang Jan 11, 2023

Choose a reason for hiding this comment

zhoujinsong left a comment

Choose a reason for hiding this comment

zhoujinsong commented Jan 11, 2023 • edited Loading

shidayang commented Jan 11, 2023

zhoujinsong left a comment

Choose a reason for hiding this comment

majin1102 Jan 12, 2023

Choose a reason for hiding this comment

majin1102 Jan 12, 2023

Choose a reason for hiding this comment

majin1102 left a comment

Choose a reason for hiding this comment

shidayang commented Jan 10, 2023 •

edited

Loading

shidayang Jan 11, 2023 •

edited

Loading

zhoujinsong commented Jan 11, 2023 •

edited

Loading