Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[AMORO-1951] Support parallelized planning in one optimizer group #2282

Merged
merged 34 commits into from
Dec 5, 2023
Merged

[AMORO-1951] Support parallelized planning in one optimizer group #2282

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
a5ed5ad
[AMORO-1951] Support parallelized planning in one optimizer group
Oct 31, 2023
60aef48
Merge remote-tracking branch 'origin/master' into concurrent-plan
Oct 31, 2023
56d333b
[AMORO-1951] add unit test for OptimizingQueue and DefaultOptimizingS…
Nov 10, 2023
251811a
Merge remote-tracking branch 'origin/master' into concurrent-plan
Nov 10, 2023
f860d16
Merge pull request #1 from majin1102/master
majin1102 Nov 10, 2023
4ef61f3
Merge remote-tracking branch 'origin/concurrent-plan' into concurrent…
Nov 10, 2023
ba5344d
[AMORO-1951] optimize default parameters
Nov 10, 2023
9a94284
fix bugs
Nov 21, 2023
2528260
Merge remote-tracking branch 'origin/master' into concurrent-plan
Nov 21, 2023
3304254
fix warnings and spotless issues
Nov 21, 2023
0ec4b4a
Merge remote-tracking branch 'origin/master' into concurrent-plan
Nov 23, 2023
4cf8536
Merge remote-tracking branch 'origin/master' into concurrent-plan
Nov 23, 2023
990d0ef
merge from #2290
Nov 23, 2023
829dea2
add apache license and fix spotless
Nov 23, 2023
372c7ef
fix config error
Nov 24, 2023
1bf06f9
Merge branch 'master' into concurrent-plan
zhoujinsong Nov 28, 2023
927fdb6
Update ams/server/src/main/java/com/netease/arctic/server/DefaultOpti…
majin1102 Nov 30, 2023
3c1caca
add annotations
Nov 30, 2023
6369955
Merge remote-tracking branch 'origin/concurrent-plan' into concurrent…
Nov 30, 2023
9488cd5
fix compile errors
Nov 30, 2023
57d896b
Merge remote-tracking branch 'origin/master' into concurrent-plan
Nov 30, 2023
5b03bc8
fix import problem
Nov 30, 2023
4820eb5
remove isDebugEnabled()
Nov 30, 2023
c745b47
spotless apply
Nov 30, 2023
89c0ce6
Update ArcticManagementConf.java
majin1102 Dec 1, 2023
3d80143
fix reboot bug and supply document content
Dec 4, 2023
28f501f
use MoreObjects.toStringHelper for OptimizerThread.java
Dec 4, 2023
708a150
Merge remote-tracking branch 'origin/master' into concurrent-plan
Dec 4, 2023
93e7e2f
Merged from [AMORO-2376] Print right log info after calculating and s…
Dec 4, 2023
ebe5edc
fix import problem
Dec 4, 2023
f5caae5
remove unused codes
Dec 4, 2023
2b58c90
spotless
Dec 4, 2023
d846477
remove incorrect comments
Dec 5, 2023
e6efd97
add max-planning-parallelism to config
Dec 5, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions ams/api/src/main/java/com/netease/arctic/ams/api/resource/ResourceManager.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,4 +23,7 @@ public interface ResourceManager {
List<Resource> listResourcesByGroup(String groupName);

Resource getResource(String resourceId);

/** close this resource manager */
void dispose();
majin1102 marked this conversation as resolved.
Show resolved Hide resolved
}
2 changes: 2 additions & 0 deletions ams/dist/src/main/arctic-bin/conf/config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,8 @@ ams:
optimizer:
heart-beat-timeout: 60000 # 1min
task-ack-timeout: 30000 # 30s
polling-timeout: 3000 # 3s
max-planning-parallelism: 1 # default 1

blocker:
timeout: 60000 # 1min
Expand Down
18 changes: 12 additions & 6 deletions ams/server/src/main/java/com/netease/arctic/server/ArcticManagementConf.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -254,6 +254,18 @@ public class ArcticManagementConf {
.defaultValue(30000L)
.withDescription("Timeout duration for task acknowledgment.");

public static final ConfigOption<Integer> OPTIMIZER_MAX_PLANNING_PARALLELISM =
ConfigOptions.key("optimizer.max-planning-parallelism")
.intType()
.defaultValue(1)
.withDescription("Max planning parallelism in one optimizer group.");

public static final ConfigOption<Long> OPTIMIZER_POLLING_TIMEOUT =
ConfigOptions.key("optimizer.polling-timeout")
.longType()
.defaultValue(3000L)
.withDescription("Optimizer polling task timeout.");

/** config key prefix of terminal */
public static final String TERMINAL_PREFIX = "terminal.";

Expand Down Expand Up @@ -317,12 +329,6 @@ public class ArcticManagementConf {
public static final String CONTAINER_IMPL = "container-impl";
public static final String CONTAINER_PROPERTIES = "properties";

// optimizer config
public static final String OPTIMIZER_GROUP_LIST = "optimizer_groups";
public static final String OPTIMIZER_GROUP_NAME = "name";
public static final String OPTIMIZER_GROUP_CONTAINER = "container";
public static final String OPTIMIZER_GROUP_PROPERTIES = "properties";

public static final String DB_TYPE_DERBY = "derby";
public static final String DB_TYPE_MYSQL = "mysql";
public static final String DB_TYPE_POSTGRES = "postgres";
Expand Down
2 changes: 0 additions & 2 deletions ams/server/src/main/java/com/netease/arctic/server/ArcticServiceConstants.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,6 @@

public class ArcticServiceConstants {

public static final long OPTIMIZER_CHECK_INTERVAL = 10 * 1000;

public static final long INVALID_TIME = 0;

public static final long QUOTA_LOOK_BACK_TIME = 60 * 60 * 1000;
Expand Down
Loading