Skip to content

[Optimization] Calculate global parameter and local parameter at master.#10704

Merged
lenboo merged 6 commits intoapache:devfrom
WangJPLeo:calendat-local-expand
Jun 30, 2022
Merged

[Optimization] Calculate global parameter and local parameter at master.#10704
lenboo merged 6 commits intoapache:devfrom
WangJPLeo:calendat-local-expand

Conversation

@WangJPLeo
Copy link
Contributor

Purpose of the pull request

  1. Add an external extension class to the service.
  2. The global parameters and local parameters are calculated and prepared in the master service.
  3. The local parameters in the worker service will be used directly.

Brief change log

close #10703

Verify this pull request

Manually verified the change by testing locally.

@SbloodyS SbloodyS added feature new feature backend labels Jun 30, 2022
@SbloodyS SbloodyS added this to the 3.1.0-alpha milestone Jun 30, 2022
@WangJPLeo WangJPLeo changed the title [Feature] Global parameter and local parameter calculation external expansion. [Optimization] Global parameter and local parameter calculation external expansion. Jun 30, 2022
@codecov-commenter
Copy link

codecov-commenter commented Jun 30, 2022

Codecov Report

Merging #10704 (efa2ccc) into dev (a90e574) will decrease coverage by 0.05%.
The diff coverage is 37.17%.

❗ Current head efa2ccc differs from pull request most recent head 3fd63b3. Consider uploading reports for the commit 3fd63b3 to get more accurate results

@@             Coverage Diff              @@
##                dev   #10704      +/-   ##
============================================
- Coverage     40.89%   40.84%   -0.06%     
+ Complexity     4830     4827       -3     
============================================
  Files           897      897              
  Lines         36157    36128      -29     
  Branches       4006     3987      -19     
============================================
- Hits          14788    14756      -32     
- Misses        19898    19907       +9     
+ Partials       1471     1465       -6     
Impacted Files Coverage Δ
...r/api/service/impl/ProcessInstanceServiceImpl.java 60.10% <ø> (ø)
.../org/apache/dolphinscheduler/common/Constants.java 81.81% <ø> (ø)
...er/server/builder/TaskExecutionContextBuilder.java 64.70% <0.00%> (-5.51%) ⬇️
...r/server/master/runner/MasterSchedulerService.java 0.00% <ø> (ø)
...pand/TimePlaceholderResolverExpandServiceImpl.java 100.00% <ø> (ø)
...nscheduler/service/process/ProcessServiceImpl.java 30.28% <ø> (ø)
...inscheduler/plugin/task/api/parser/ParamUtils.java 0.00% <ø> (ø)
...heduler/plugin/task/api/parser/ParameterUtils.java 0.00% <ø> (ø)
...lphinscheduler/plugin/task/dq/DataQualityTask.java 0.00% <0.00%> (ø)
.../dolphinscheduler/plugin/task/datax/DataxTask.java 0.00% <0.00%> (ø)
... and 29 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a90e574...3fd63b3. Read the comment docs.

Copy link
Member

@ruanwenjun ruanwenjun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see the whole, just add some comments about code style.

@sonarqubecloud
Copy link

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 34 Code Smells

33.9% 33.9% Coverage
0.0% 0.0% Duplication

1 similar comment
@sonarqubecloud
Copy link

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 34 Code Smells

33.9% 33.9% Coverage
0.0% 0.0% Duplication

Copy link
Member

@ruanwenjun ruanwenjun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ruanwenjun ruanwenjun changed the title [Optimization] Global parameter and local parameter calculation external expansion. [Optimization] Calculate global parameter and local parameter at master. Jun 30, 2022
@lenboo
Copy link
Contributor

lenboo commented Jun 30, 2022

+1

Copy link
Contributor

@lenboo lenboo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@lenboo lenboo merged commit 8f621ff into apache:dev Jun 30, 2022
welsh-wen pushed a commit to welsh-wen/dolphinscheduler that referenced this pull request Jul 2, 2022
…er. (apache#10704)

* Global parameter and local parameter calculation external expansion.

* k8s task ut fix.

* TimePlaceholderUtils import DateUtils fix

* follow the review comments to fix.

* follow the review comments to fix.

* e2e rerun
zhongjiajie pushed a commit to zhongjiajie/dolphinscheduler that referenced this pull request Jul 5, 2022
…er. (apache#10704)

* Global parameter and local parameter calculation external expansion.

* k8s task ut fix.

* TimePlaceholderUtils import DateUtils fix

* follow the review comments to fix.

* follow the review comments to fix.

* e2e rerun
ruanwenjun pushed a commit to ruanwenjun/dolphinscheduler that referenced this pull request Jul 12, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Optimization] Global parameter and local parameter calculation external expansion.

6 participants