[Feature][SQLTask] Suggest SQL task support variables in anywhere by zhuxt2015 · Pull Request #10746 · apache/dolphinscheduler

zhuxt2015 · 2022-07-03T11:48:01Z

Purpose of the pull request

fix #10745

Brief change log

Verify this pull request

This pull request is code cleanup without any test coverage.

(or)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

codecov-commenter · 2022-07-03T12:41:58Z

Codecov Report

Merging #10746 (18afe65) into dev (13d400c) will increase coverage by 0.00%.
The diff coverage is n/a.

@@            Coverage Diff            @@
##                dev   #10746   +/-   ##
=========================================
  Coverage     39.36%   39.36%           
- Complexity     4631     4632    +1     
=========================================
  Files           987      987           
  Lines         37566    37566           
  Branches       4183     4183           
=========================================
+ Hits          14787    14788    +1     
  Misses        21227    21227           
+ Partials       1552     1551    -1

Impacted Files	Coverage Δ
...e/dolphinscheduler/remote/NettyRemotingClient.java	`52.77% <0.00%> (+0.69%)`	⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

sonarqubecloud · 2022-07-03T12:49:28Z

SonarCloud Quality Gate failed.

0 Bugs
0 Vulnerabilities
0 Security Hotspots
1 Code Smell

0.0% Coverage
0.0% Duplication

...hinscheduler-task-sql/src/main/java/org/apache/dolphinscheduler/plugin/task/sql/SqlTask.java

caishunfeng · 2022-08-11T01:43:16Z

Hi @zhuxt2015 please resolve the conflicts.

caishunfeng · 2022-08-15T07:46:42Z

...hinscheduler-task-sql/src/main/java/org/apache/dolphinscheduler/plugin/task/sql/SqlTask.java

-     */
-    private PreparedStatement prepareStatementAndBind(Connection connection, SqlBinds sqlBinds) {
-        // is the timeout set
-        boolean timeoutFlag = taskExecutionContext.getTaskTimeoutStrategy() == TaskTimeoutStrategy.FAILED


It seems miss the timeout action.

caishunfeng · 2022-08-15T07:55:04Z

...hinscheduler-task-sql/src/main/java/org/apache/dolphinscheduler/plugin/task/sql/SqlTask.java

-        sql = ParameterUtils.replaceScheduleTime(sql, taskExecutionContext.getScheduleTime());
-        // special characters need to be escaped, ${} needs to be escaped
-        setSqlParamsMap(sql, rgex, sqlParamsMap, paramsMap,taskExecutionContext.getTaskInstanceId());
-        //Replace the original value in sql ！{...} ，Does not participate in precompilation
-        String rgexo = "['\"]*\\!\\{(.*?)\\}['\"]*";
-        sql = replaceOriginalValue(sql, rgexo, paramsMap);
-        // replace the ${} of the SQL statement with the Placeholder
-        String formatSql = sql.replaceAll(rgex, "?");
-        // Convert the list parameter
-        formatSql = ParameterUtils.expandListParameter(sqlParamsMap, formatSql);
-        sqlBuilder.append(formatSql);


Why remove these logic?

caishunfeng · 2022-08-15T07:55:43Z

BTW, @zhuxt2015 can you add some UT for it?

* [doc] Use Rainbond deployment DolphinScheduler Cluster Signed-off-by: Qi Zhang <smallqi1@163.com> * Update docs/docs/en/guide/installation/rainbond.md * Update docs/docs/en/guide/installation/rainbond.md * Update docs/docs/zh/guide/installation/rainbond.md * Update docs/docs/zh/guide/installation/rainbond.md * [doc] modify rainbond doc and add image Signed-off-by: Qi Zhang <smallqi1@163.com> * fix: img link Signed-off-by: Qi Zhang <smallqi1@163.com> * fix: modify proposal Signed-off-by: Qi Zhang <smallqi1@163.com> * docs: move rainbond docs to integration Signed-off-by: Qi Zhang <smallqi1@163.com> Signed-off-by: Qi Zhang <smallqi1@163.com> Co-authored-by: Jiajie Zhong <zhongjiajie955@gmail.com>

* Update instructions on worker groups

…ponent. (apache#11485) * [Refactor][UI] Refactor task definition batch stream using NSpace component. * [Refactor][UI] Refactor task definition batch stream using NSpace component. * [Refactor][UI] Refactor task definition batch stream using NSpace component.

…t. (apache#11495)

…ache#11402)

In the process of realizing datax synchronization, use custom json for better connection to enable kerberos-authenticated clusters, such as using plug-ins such as hdfs and hbase

* [fix] Database change not sync * Make del sql upgrade same as release version * Correct data quality dml sql * Upgrade database DDL to avoid log path too long error * Add missing if exists

…e end of each java file (apache#11520)

…lines. (apache#11484)

* [test][HttpTask] Add unit tests to http task * [test][HttpTask] fix http task test code format problem * [test][HttpTask] fix http task test code replace http code to constants

* [Improvement] File deletion logic optimization

…pache#11489) * Support re running historical version workflow Support re running historical version workflow * Update ExecutorServiceImpl.java * Update ExecutorServiceImpl.java

Co-authored-by: Eric Gao <ericgao.apache@gmail.com>

…che#11527)

…example (apache#11384)

…tion (apache#11543) * add can stop for execute state machine * add execute type check for pause operation

…eal null currently. (apache#11325)

apache#11441) and set the resource.storage.type default to hdfs

sonarqubecloud · 2022-08-21T11:42:29Z

SonarCloud Quality Gate failed.

1 Bug
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

0.0% Coverage
0.0% Duplication

zhuxt2015 requested review from SbloodyS, caishunfeng and zhuangchong as code owners July 3, 2022 11:48

SbloodyS assigned zhuxt2015 Jul 3, 2022

SbloodyS added feature new feature backend labels Jul 3, 2022

SbloodyS added this to the 3.0.0-beta-3 milestone Jul 3, 2022

caishunfeng removed this from the 3.0.0-release milestone Jul 20, 2022

ruanwenjun reviewed Aug 1, 2022

View reviewed changes

...hinscheduler-task-sql/src/main/java/org/apache/dolphinscheduler/plugin/task/sql/SqlTask.java Outdated Show resolved Hide resolved

zhuxt2015 added 5 commits August 14, 2022 20:56

fix

735b90a

fix

7667415

close ResultSet

1140ac3

fix checkstyle

fe51dd6

resolve conflicts

0416351

zhuxt2015 force-pushed the sql_task_support_variables_anywhere branch from b6b42fa to 0416351 Compare August 14, 2022 12:56

zhuxt2015 added 2 commits August 14, 2022 22:57

remove unuse function

7cc5ee5

use SQLException instead of Exception

18afe65

caishunfeng reviewed Aug 15, 2022

View reviewed changes

zhuangchong and others added 8 commits August 21, 2022 19:17

update PyDolphinScheduler documentation link. (apache#11474)

e4d0882

[Doc][Security] Update instructions on worker groups (apache#11483)

face998

* Update instructions on worker groups

[Fix][UI] Fix the table header in the workflow instance. (apache#11487)

bf085a0

[Refactor][UI] Refactor task definition version using NSpace componen…

ad2aa75

…t. (apache#11495)

[Doc] Update UI readme. (apache#11496)

dd08c32

Support reading application properties from kubernetes config map (ap…

fa0e15f

…ache#11402)

lugela and others added 18 commits August 21, 2022 19:17

[Feature-10182][task-datax] (apache#11387)

7486986

In the process of realizing datax synchronization, use custom json for better connection to enable kerberos-authenticated clusters, such as using plug-ins such as hdfs and hbase

[fix] Database change not sync (apache#11258)

28f8573

* [fix] Database change not sync * Make del sql upgrade same as release version * Correct data quality dml sql * Upgrade database DDL to avoid log path too long error * Add missing if exists

[Feature][Style] Make sure there is one and only one empty line at th…

e06f51f

…e end of each java file (apache#11520)

[DS-11481][docs] Add log specification document to contribution guide…

b986ea9

…lines. (apache#11484)

[test-11432][HttpTask] Add unit tests to http task (apache#11453)

ad378aa

* [test][HttpTask] Add unit tests to http task * [test][HttpTask] fix http task test code format problem * [test][HttpTask] fix http task test code replace http code to constants

[Improvement] File deletion logic optimization (apache#11519)

14b2da5

* [Improvement] File deletion logic optimization

[Improvement-11468] Support re running historical version workflow (a…

672c435

…pache#11489) * Support re running historical version workflow Support re running historical version workflow * Update ExecutorServiceImpl.java * Update ExecutorServiceImpl.java

[Feature] New task plugin Pytorch (apache#11498)

2fa8dd9

Co-authored-by: Eric Gao <ericgao.apache@gmail.com>

[Fix][UI] Remove unsupported datasource types in the datax task. (apa…

d2fa8a7

…che#11527)

Set process instance and task priority default value as 2 (apache#11539)

c28644f

Enable spotless to fix markdown checkstyle and refactor faq.md as an …

1ce1d62

…example (apache#11384)

[Improvement]Add a stop function when you are ready to do pause opera…

79ad318

…tion (apache#11543) * add can stop for execute state machine * add execute type check for pause operation

[Improvement] [LDAP] Bug fix: Config value return "null" instead of r…

af4d591

…eal null currently. (apache#11325)

Remove logger header in task log file (apache#11555)

49982ff

[fix-11404]: make the common.properties to configurable on values.yaml (

89360fb

apache#11441) and set the resource.storage.type default to hdfs

[feature-11408] add hive option (apache#11493)

f709cfc

Fix some resource leak (apache#11576)

9e9f868

add query timeout logic

4ca923a

zhuxt2015 requested review from Amy0104, EricGao888, Tianqi-Dotes, kezhenxu94, songjianet and zhongjiajie as code owners August 21, 2022 11:18

github-actions bot added CI&CD document Pyscheduler UI ui and front end related labels Aug 21, 2022

zhuxt2015 closed this Aug 21, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature][SQLTask] Suggest SQL task support variables in anywhere #10746

[Feature][SQLTask] Suggest SQL task support variables in anywhere #10746
zhuxt2015 wants to merge 34 commits intoapache:devfrom
zhuxt2015:sql_task_support_variables_anywhere

zhuxt2015 commented Jul 3, 2022

Uh oh!

codecov-commenter commented Jul 3, 2022 •

edited

Loading

Uh oh!

sonarqubecloud bot commented Jul 3, 2022

Uh oh!

Uh oh!

caishunfeng commented Aug 11, 2022

Uh oh!

caishunfeng Aug 15, 2022

Uh oh!

caishunfeng Aug 15, 2022

Uh oh!

caishunfeng commented Aug 15, 2022

Uh oh!

sonarqubecloud bot commented Aug 21, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Conversation

zhuxt2015 commented Jul 3, 2022

Purpose of the pull request

Brief change log

Verify this pull request

Uh oh!

codecov-commenter commented Jul 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sonarqubecloud bot commented Jul 3, 2022

Uh oh!

Uh oh!

caishunfeng commented Aug 11, 2022

Uh oh!

caishunfeng Aug 15, 2022

Choose a reason for hiding this comment

Uh oh!

caishunfeng Aug 15, 2022

Choose a reason for hiding this comment

Uh oh!

caishunfeng commented Aug 15, 2022

Uh oh!

sonarqubecloud bot commented Aug 21, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

codecov-commenter commented Jul 3, 2022 •

edited

Loading