[Task] Add Dinky task to better support the development and execution of FlinkSQL by aiwenmo · Pull Request #10640 · apache/dolphinscheduler

aiwenmo · 2022-06-27T16:00:23Z

Purpose of the pull request

Add Dinky task to better support the development and execution of FlinkSQL.

close #10444

Brief change log

Verify this pull request

This pull request is code cleanup without any test coverage.

(or)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

…t and execution of FlinkSQL

aiwenmo · 2022-06-28T13:31:39Z

demo-get-dinky-task-id(../../../../img/tasks/demo/dinky_task_id.png)
At present, Dinky is not internationalized, and English pictures will be added later.

codecov-commenter · 2022-06-28T16:15:45Z

Codecov Report

Merging #10640 (464ac6e) into dev (719a9d4) will decrease coverage by 0.04%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##                dev   #10640      +/-   ##
============================================
- Coverage     40.94%   40.90%   -0.05%     
+ Complexity     4876     4830      -46     
============================================
  Files           895      897       +2     
  Lines         36212    36158      -54     
  Branches       3987     4006      +19     
============================================
- Hits          14826    14789      -37     
+ Misses        19921    19898      -23     
- Partials       1465     1471       +6

Impacted Files	Coverage Δ
...inscheduler/server/master/config/MasterConfig.java	`35.00% <0.00%> (-41.75%)`	⬇️
...inscheduler/server/worker/config/WorkerConfig.java	`52.38% <0.00%> (-24.09%)`	⬇️
.../dolphinscheduler/common/utils/ParameterUtils.java	`58.53% <0.00%> (-14.60%)`	⬇️
...cheduler/plugin/task/api/TaskExecutionContext.java	`93.87% <0.00%> (-5.38%)`	⬇️
...ache/impl/ProcessInstanceExecCacheManagerImpl.java	`83.33% <0.00%> (-1.29%)`	⬇️
...heduler/server/master/service/FailoverService.java	`63.80% <0.00%> (-1.10%)`	⬇️
...he/dolphinscheduler/server/utils/ProcessUtils.java	`28.23% <0.00%> (-1.04%)`	⬇️
...olphinscheduler/plugin/alert/email/MailSender.java	`51.03% <0.00%> (-0.36%)`	⬇️
...r/api/service/impl/WorkFlowLineageServiceImpl.java	`33.01% <0.00%> (-0.32%)`	⬇️
...nscheduler/service/process/ProcessServiceImpl.java	`30.28% <0.00%> (-0.30%)`	⬇️
... and 34 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 719a9d4...464ac6e. Read the comment docs.

...eduler-task-dinky/src/main/java/org/apache/dolphinscheduler/plugin/task/dinky/DinkyTask.java

SbloodyS · 2022-06-29T11:14:31Z

SonarCloud Quality Gate failed.

1 Bug 0 Vulnerabilities 0 Security Hotspots 0 Code Smells

0.0% Coverage 0.0% Duplication

Please fix the bugs.

...task-dinky/src/main/java/org/apache/dolphinscheduler/plugin/task/dinky/DinkyTaskChannel.java

SbloodyS

Backend part LGTM. Thanks.

dolphinscheduler-ui/src/views/projects/task/components/node/tasks/use-dinky.ts

aiwenmo · 2022-07-04T02:29:44Z

Hi. Please review the new code. thx. @Amy0104

Amy0104

The front end part LGTM.

SbloodyS · 2022-07-04T03:42:58Z

@zhongjiajie PTAL.

zhongjiajie

And, please add content in https://github.com/apache/dolphinscheduler/blob/dev/docs/configs/docsdev.js#L169-L171 and https://github.com/apache/dolphinscheduler/blob/dev/docs/configs/docsdev.js#L548-L551

zhongjiajie · 2022-07-04T05:53:23Z

docs/docs/en/guide/task/dinky.md

+- Node name: The node name in a workflow definition is unique.
+- Run flag: Identifies whether this node can be scheduled normally, if it does not need to be executed, you can turn on the prohibition switch.
+- Descriptive information: Describe the function of the node.
+- Task priority: When the number of worker threads is insufficient, execute in the order of priority from high to low, and tasks with the same priority will execute in a first-in first-out order.
+- Worker grouping: Assign tasks to the machines of the worker group to execute. If `Default` is selected, randomly select a worker machine for execution.
+- Number of failed retry attempts: The failure task resubmitting times. It supports drop-down and hand-filling.
+- Failed retry interval: The time interval for resubmitting the task after a failed task. It supports drop-down and hand-filling.
+- Timeout alarm: Check the timeout alarm and timeout failure. When the task exceeds the "timeout period", an alarm email will send and the task execution will fail.
+- Dinky Address: The url for a dinky server.
+- Dinky Task ID: The unique task id for a dinky task.
+- Online Task: Specify whether the current dinky job is online. If yes, the submitted job can only be submitted successfully when it is published and there is no corresponding Flink job instance running.


currently, all our English version tasks parameters convert from list style to table format style, which you can see example in https://github.com/apache/dolphinscheduler/blob/dev/docs/docs/en/guide/task/shell.md. So please change to a table format in English version of docs. And it is up to you whether change Chinese version of docs

sonarqubecloud · 2022-07-04T13:05:04Z

SonarCloud Quality Gate failed.

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

0.0% Coverage
0.0% Duplication

aiwenmo · 2022-07-04T13:24:45Z

Hi. Please review the new docs. thx. @zhongjiajie

zhongjiajie

LGTM, thanks

zhongjiajie · 2022-07-05T02:08:01Z

Thanks for bringing this up

… of FlinkSQL (apache#10640)

… of FlinkSQL (apache#10640) # Conflicts: # docs/configs/docsdev.js # dolphinscheduler-task-plugin/dolphinscheduler-task-all/pom.xml # dolphinscheduler-task-plugin/pom.xml # dolphinscheduler-ui/src/views/projects/task/components/node/fields/index.ts # dolphinscheduler-ui/src/views/projects/task/components/node/format-data.ts # dolphinscheduler-ui/src/views/projects/task/components/node/tasks/index.ts # dolphinscheduler-ui/src/views/projects/task/components/node/types.ts # dolphinscheduler-ui/src/views/projects/task/constants/task-type.ts # dolphinscheduler-ui/src/views/projects/workflow/components/dag/dag.module.scss 3.0.1 合并dinky的代码.

[Feature-10444][Task] Add Dinky task to better support the developmen…

eeb4a01

…t and execution of FlinkSQL

aiwenmo requested review from Amy0104, SbloodyS, caishunfeng, kezhenxu94, songjianet and zhuangchong as code owners June 27, 2022 16:00

SbloodyS assigned aiwenmo Jun 28, 2022

SbloodyS added feature new feature UI ui and front end related backend labels Jun 28, 2022

SbloodyS added this to the 3.1.0-alpha milestone Jun 28, 2022

[Feature-10444][Task] Add dinky task docs

252bdd0

aiwenmo requested review from Tianqi-Dotes and zhongjiajie as code owners June 28, 2022 10:27

[Feature-10444][Task] Add dinky task icon

308c0a9

[Feature-10444][Task] Fix bugs and optimize code smells

464ac6e

caishunfeng reviewed Jun 29, 2022

View reviewed changes

[Feature-10444][Task] Fix a bug and optimize a code smell

e7288c4

[Feature-10444][Task] Optimize DinkyTask

0ff07d9

SbloodyS closed this Jun 30, 2022

SbloodyS reopened this Jun 30, 2022

SbloodyS reviewed Jun 30, 2022

View reviewed changes

...task-dinky/src/main/java/org/apache/dolphinscheduler/plugin/task/dinky/DinkyTaskChannel.java Show resolved Hide resolved

SbloodyS previously approved these changes Jun 30, 2022

View reviewed changes

Amy0104 requested changes Jul 1, 2022

View reviewed changes

dolphinscheduler-ui/src/views/projects/task/components/node/tasks/use-dinky.ts Outdated Show resolved Hide resolved

dolphinscheduler-ui/src/views/projects/task/components/node/tasks/use-dinky.ts Outdated Show resolved Hide resolved

[Feature-10444][Task] Format the codes

27d7594

aiwenmo dismissed SbloodyS’s stale review via 27d7594 July 1, 2022 14:34

Amy0104 previously approved these changes Jul 4, 2022

View reviewed changes

zhongjiajie reviewed Jul 4, 2022

View reviewed changes

zhongjiajie mentioned this pull request Jul 4, 2022

[Feature][CI] CI about add new task type but not add to docsdev.js #10761

Open

3 tasks

[Feature-10444][Task] Modify the docs

0165af3

aiwenmo dismissed Amy0104’s stale review via 0165af3 July 4, 2022 12:42

zhongjiajie approved these changes Jul 5, 2022

View reviewed changes

zhongjiajie merged commit 5b0347e into apache:dev Jul 5, 2022

zhongjiajie changed the title ~~[Feature-10444][Task] Add Dinky task to better support the development and execution of FlinkSQL~~ [Task] Add Dinky task to better support the development and execution of FlinkSQL Jul 5, 2022

zhongjiajie pushed a commit to zhongjiajie/dolphinscheduler that referenced this pull request Jul 5, 2022

[Task] Add Dinky task to better support the development and execution…

1fcec55

… of FlinkSQL (apache#10640)

Conversation

aiwenmo commented Jun 27, 2022 • edited by SbloodyS Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose of the pull request

Brief change log

Verify this pull request

Uh oh!

aiwenmo commented Jun 28, 2022

Uh oh!

codecov-commenter commented Jun 28, 2022

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SbloodyS commented Jun 29, 2022

Uh oh!

Uh oh!

SbloodyS left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

aiwenmo commented Jul 4, 2022

Uh oh!

Amy0104 left a comment

Choose a reason for hiding this comment

Uh oh!

SbloodyS commented Jul 4, 2022

Uh oh!

zhongjiajie left a comment

Choose a reason for hiding this comment

Uh oh!

zhongjiajie Jul 4, 2022

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud bot commented Jul 4, 2022

Uh oh!

aiwenmo commented Jul 4, 2022

Uh oh!

zhongjiajie left a comment

Choose a reason for hiding this comment

Uh oh!

zhongjiajie commented Jul 5, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

aiwenmo commented Jun 27, 2022 •

edited by SbloodyS

Loading