Skip to content

[DS-6849][MasterServer] fetch more commands and handle in parallel#6850

Merged
lenboo merged 5 commits intoapache:devfrom
caishunfeng:find_command
Nov 19, 2021
Merged

[DS-6849][MasterServer] fetch more commands and handle in parallel#6850
lenboo merged 5 commits intoapache:devfrom
caishunfeng:find_command

Conversation

@caishunfeng
Copy link
Contributor

@caishunfeng caishunfeng commented Nov 15, 2021

this pr close #6849

Purpose of the pull request

To improve the command handle rate by two parts:

  • Fetch more commands every time, it can reduce the IO between server and db;
  • Use pre exec threads to handle command2ProcessInstance in parallel;

When master slot change, slotCheck and deleteCommandWithTry can ensure that one command will not be processed multiple times.

@codecov-commenter
Copy link

Codecov Report

Merging #6850 (a095033) into dev (94f6c5c) will decrease coverage by 0.02%.
The diff coverage is 8.00%.

Impacted file tree graph

@@             Coverage Diff              @@
##                dev    #6850      +/-   ##
============================================
- Coverage     33.09%   33.06%   -0.03%     
- Complexity     1621     1622       +1     
============================================
  Files           433      433              
  Lines         14290    14310      +20     
  Branches       1426     1427       +1     
============================================
+ Hits           4729     4732       +3     
- Misses         9111     9127      +16     
- Partials        450      451       +1     
Impacted Files Coverage Δ
...r/server/master/runner/MasterSchedulerService.java 0.00% <0.00%> (ø)
...inscheduler/server/master/config/MasterConfig.java 69.76% <66.66%> (-0.51%) ⬇️
...er/master/dispatch/host/assign/RandomSelector.java 77.77% <0.00%> (-5.56%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 94f6c5c...a095033. Read the comment docs.

@caishunfeng caishunfeng changed the title [DS-6849][MasterServer] fetch more commands and handle in parallel [WIP] [DS-6849][MasterServer] fetch more commands and handle in parallel Nov 15, 2021
Comment on lines +24 to +26
master-fetch-command-num: 10
# master prepare execute thread number to limit handle commands in parallel
master-pre-exec-threads: 10
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here, strip the prefix master-

@caishunfeng caishunfeng changed the title [WIP] [DS-6849][MasterServer] fetch more commands and handle in parallel [DS-6849][MasterServer] fetch more commands and handle in parallel Nov 16, 2021
@sonarqubecloud
Copy link

SonarCloud Quality Gate failed.    Quality Gate failed

Bug C 1 Bug
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 2 Code Smells

16.8% 16.8% Coverage
0.0% 0.0% Duplication

@kezhenxu94
Copy link
Member

LGTM

Copy link
Contributor

@lenboo lenboo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@lenboo lenboo merged commit 595e484 into apache:dev Nov 19, 2021
@caishunfeng caishunfeng deleted the find_command branch December 15, 2021 04:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Improvement][MasterServer] improve master scan and handle commands

4 participants