[Feature][plugin] Support Kyuubi datasource #13642

zhaohehuhu · 2023-02-28T03:30:06Z

Purpose of the pull request

close #12535
This PR is to Introduce Kyuubi as datasource, so users can use Kyuubi as SQL proxy instead of Hive datasource, which can get more feature support powered by Kyuubi.

Brief change log

Verify this pull request

This pull request is code cleanup with some test coverage.

(or)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(or)

If your pull request contain incompatible change, you should also add it to docs/docs/en/guide/upgrede/incompatible.md

SbloodyS

Please check the failed CI. Thanks.

fuchanghai

where did you use this parameter datawarehouse ?

zhaohehuhu · 2023-03-01T09:29:38Z

where did you use this parameter datawarehouse ?

will check later

EricGao888 · 2023-03-03T14:45:04Z

@zhaohehuhu CI failed by style check. Could u plz run mvn spotless:apply? Spotless will automatically fix those errors.

EricGao888 · 2023-03-03T14:47:03Z

...c/main/java/org/apache/dolphinscheduler/plugin/datasource/kyuubi/KyuubiDataSourceClient.java

+import org.apache.dolphinscheduler.common.utils.PropertyUtils;
+import org.apache.dolphinscheduler.plugin.datasource.api.client.CommonDataSourceClient;
+import org.apache.dolphinscheduler.plugin.datasource.api.provider.JDBCDataSourceProvider;
+import org.apache.dolphinscheduler.plugin.datasource.kyuubi.security.*;


Please avoid using wildcard imports and fix it before running spotless cmd, or spotless will directly remove it and lead to compilation failure.

zhaohehuhu · 2023-03-06T06:09:42Z

@zhaohehuhu CI failed by style check. Could u plz run mvn spotless:apply? Spotless will automatically fix those errors.

OK

zhaohehuhu · 2023-03-06T06:21:33Z

where did you use this parameter datawarehouse ?

fixed

codecov-commenter · 2023-03-06T07:32:23Z

Codecov Report

Merging #13642 (1ab382c) into dev (5f94c8c) will increase coverage by 0.04%.
The diff coverage is 58.82%.

❗ Current head 1ab382c differs from pull request most recent head 50efaba. Consider uploading reports for the commit 50efaba to get more accurate results

@@             Coverage Diff              @@
##                dev   #13642      +/-   ##
============================================
+ Coverage     38.87%   38.91%   +0.04%     
- Complexity     4455     4476      +21     
============================================
  Files          1158     1164       +6     
  Lines         42429    42513      +84     
  Branches       4776     4780       +4     
============================================
+ Hits          16496    16546      +50     
- Misses        24118    24151      +33     
- Partials       1815     1816       +1

Impacted Files	Coverage Δ
...cheduler/common/constants/DataSourceConstants.java	`0.00% <ø> (ø)`
.../org/apache/dolphinscheduler/spi/enums/DbType.java	`0.00% <0.00%> (ø)`
...ugin/datasource/kyuubi/KyuubiDataSourceClient.java	`4.76% <4.76%> (ø)`
...gin/datasource/kyuubi/KyuubiDataSourceChannel.java	`50.00% <50.00%> (ø)`
...source/kyuubi/param/KyuubiDataSourceProcessor.java	`76.92% <76.92%> (ø)`
...asource/kyuubi/KyuubiDataSourceChannelFactory.java	`100.00% <100.00%> (ø)`
...datasource/kyuubi/param/KyuubiConnectionParam.java	`100.00% <100.00%> (ø)`
...asource/kyuubi/param/KyuubiDataSourceParamDTO.java	`100.00% <100.00%> (ø)`

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

...c/main/java/org/apache/dolphinscheduler/plugin/datasource/kyuubi/KyuubiDataSourceClient.java

...va/org/apache/dolphinscheduler/plugin/datasource/kyuubi/param/KyuubiDataSourceProcessor.java

+        }
+        hosts.deleteCharAt(hosts.length() - 1);
+        kyuubiDataSourceParamDTO.setHost(hosts.toString());
+        kyuubiDataSourceParamDTO.setPort(Integer.parseInt(hostPortArray[0].split(Constants.COLON)[1]));


zhaohehuhu · 2023-03-06T07:57:46Z

Hi All, what should I do next ? @EricGao888 @fuchanghai

fuchanghai · 2023-03-06T08:04:01Z

Hi All, what should I do next ? @EricGao888 @fuchanghai

can you add related license?

fuchanghai · 2023-03-06T08:06:41Z

@zhaohehuhu You can refer to this #12379

zhaohehuhu · 2023-03-06T09:40:46Z

Hi All, what should I do next ? @EricGao888 @fuchanghai

can you add related license?

Sure.

fuchanghai · 2023-03-06T10:06:49Z

dolphinscheduler-spi/src/main/java/org/apache/dolphinscheduler/spi/enums/DbType.java

@@ -43,7 +43,8 @@ public enum DbType {
    TRINO(12, "trino"),
    STARROCKS(13, "starrocks"),
    AZURESQL(14, "azuresql"),
-    DAMENG(15, "dameng");
+    DAMENG(15, "dameng"),
+    KYUUBI(18, "kyuubi");


why not set 16?

why not set 16?

cause 16 is occupied by other datasources implemented by ourselves

how about use 17?

17 is used by ssh now

zhaohehuhu · 2023-03-07T01:53:45Z

@zhaohehuhu You can refer to this #12379

should be fixed. Please hep check it @fuchanghai

fuchanghai · 2023-03-07T03:00:12Z

@zhaohehuhu You can refer to this #12379

should be fixed. Please hep check it @fuchanghai
maybe you need to add kyuubi in this file

fix kyuubi datasource doc remove indention

pan3793

Thanks @zhaohehuhu, LGTM on my side

zhaohehuhu · 2023-04-10T05:17:25Z

Thanks @zhaohehuhu, LGTM on my side

Thanks. Buddy.

zhaohehuhu · 2023-04-10T05:18:23Z

@SbloodyS plz help run CI

SbloodyS · 2023-04-10T05:32:48Z

@SbloodyS plz help run CI

Done.

zhongjiajie · 2023-04-11T01:36:03Z

approval to run CI

zhaohehuhu · 2023-04-11T04:22:28Z

approval to run CI
plz help run

zhaohehuhu · 2023-04-13T02:40:08Z

Hi All, what should I do next ? @EricGao888 @fuchanghai @Radeity

sonarcloud · 2023-04-13T06:53:58Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
28 Code Smells

64.2% Coverage
0.0% Duplication

zhaohehuhu · 2023-04-17T14:30:09Z

Hi @EricGao888 @fuchanghai @Radeity, it seems there is no error anymore for this PR.

fuchanghai

LGTM on my side

already fix

zhaohehuhu · 2023-04-18T06:52:07Z

Thanks all

zhaohehuhu requested review from kezhenxu94, songjianet, Amy0104 and caishunfeng as code owners February 28, 2023 03:30

github-actions bot added backend UI ui and front end related labels Feb 28, 2023

EricGao888 self-requested a review February 28, 2023 05:24

EricGao888 assigned zhaohehuhu Feb 28, 2023

EricGao888 added feature new feature plug-in labels Feb 28, 2023

EricGao888 added this to the 3.2.0 milestone Feb 28, 2023

SbloodyS reviewed Feb 28, 2023

View reviewed changes

fuchanghai reviewed Feb 28, 2023

View reviewed changes

EricGao888 reviewed Mar 3, 2023

View reviewed changes

zhaohehuhu requested review from zhongjiajie and Tianqi-Dotes as code owners March 6, 2023 06:40

github-actions bot added the document label Mar 6, 2023

github-advanced-security bot found potential problems Mar 6, 2023

View reviewed changes

fuchanghai reviewed Mar 6, 2023

View reviewed changes

zhaohehuhu added 3 commits April 10, 2023 13:03

delete unnecessary dependencies

71fbaae

fix kyuubi datasource doc remove indention

change kyuubi dependency name

8168d89

format code

19f9ffc

pan3793 previously approved these changes Apr 10, 2023

View reviewed changes

Merge remote-tracking branch 'ds/dev' into Feature-13600

970577d

add more UTs

ae27b50

zhaohehuhu dismissed pan3793’s stale review via ae27b50 April 11, 2023 04:05

add test annotation

0c5ece9

zhaohehuhu closed this Apr 11, 2023

zhaohehuhu reopened this Apr 11, 2023

zhaohehuhu added 3 commits April 12, 2023 10:06

add UTs

d8fceda

apply spotless

8680350

apply spotless

50efaba

zhaohehuhu closed this Apr 13, 2023

zhaohehuhu reopened this Apr 13, 2023

fix UTs

97afdda

fuchanghai approved these changes Apr 18, 2023

View reviewed changes

EricGao888 approved these changes Apr 18, 2023

View reviewed changes

zhongjiajie merged commit c0126b7 into apache:dev Apr 18, 2023
1 check passed

SbloodyS mentioned this pull request Apr 19, 2023

[Feature][plugin] Add databend datasource in datasource plugin #13866

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature][plugin] Support Kyuubi datasource #13642

[Feature][plugin] Support Kyuubi datasource #13642

zhaohehuhu commented Feb 28, 2023 •

edited

Loading

SbloodyS left a comment

fuchanghai left a comment

zhaohehuhu commented Mar 1, 2023

EricGao888 commented Mar 3, 2023

EricGao888 Mar 3, 2023

zhaohehuhu Mar 6, 2023

zhaohehuhu Mar 6, 2023

zhaohehuhu commented Mar 6, 2023

zhaohehuhu commented Mar 6, 2023

codecov-commenter commented Mar 6, 2023 •

edited

Loading

zhaohehuhu commented Mar 6, 2023

fuchanghai commented Mar 6, 2023

fuchanghai commented Mar 6, 2023

zhaohehuhu commented Mar 6, 2023

fuchanghai Mar 6, 2023

zhaohehuhu Mar 7, 2023

zhongjiajie Mar 30, 2023

zhaohehuhu Mar 31, 2023

zhaohehuhu commented Mar 7, 2023

fuchanghai commented Mar 7, 2023 •

edited

Loading

pan3793 left a comment

zhaohehuhu commented Apr 10, 2023

zhaohehuhu commented Apr 10, 2023

SbloodyS commented Apr 10, 2023

zhongjiajie commented Apr 11, 2023

zhaohehuhu commented Apr 11, 2023

zhaohehuhu commented Apr 13, 2023

sonarcloud bot commented Apr 13, 2023

zhaohehuhu commented Apr 17, 2023

fuchanghai left a comment

zhaohehuhu commented Apr 18, 2023

[Feature][plugin] Support Kyuubi datasource #13642

[Feature][plugin] Support Kyuubi datasource #13642

Conversation

zhaohehuhu commented Feb 28, 2023 • edited Loading

Purpose of the pull request

Brief change log

Verify this pull request

SbloodyS left a comment

Choose a reason for hiding this comment

fuchanghai left a comment

Choose a reason for hiding this comment

zhaohehuhu commented Mar 1, 2023

EricGao888 commented Mar 3, 2023

EricGao888 Mar 3, 2023

Choose a reason for hiding this comment

zhaohehuhu Mar 6, 2023

Choose a reason for hiding this comment

zhaohehuhu Mar 6, 2023

Choose a reason for hiding this comment

zhaohehuhu commented Mar 6, 2023

zhaohehuhu commented Mar 6, 2023

codecov-commenter commented Mar 6, 2023 • edited Loading

Codecov Report

zhaohehuhu commented Mar 6, 2023

fuchanghai commented Mar 6, 2023

fuchanghai commented Mar 6, 2023

zhaohehuhu commented Mar 6, 2023

fuchanghai Mar 6, 2023

Choose a reason for hiding this comment

zhaohehuhu Mar 7, 2023

Choose a reason for hiding this comment

zhongjiajie Mar 30, 2023

Choose a reason for hiding this comment

zhaohehuhu Mar 31, 2023

Choose a reason for hiding this comment

zhaohehuhu commented Mar 7, 2023

fuchanghai commented Mar 7, 2023 • edited Loading

pan3793 left a comment

Choose a reason for hiding this comment

zhaohehuhu commented Apr 10, 2023

zhaohehuhu commented Apr 10, 2023

SbloodyS commented Apr 10, 2023

zhongjiajie commented Apr 11, 2023

zhaohehuhu commented Apr 11, 2023

zhaohehuhu commented Apr 13, 2023

sonarcloud bot commented Apr 13, 2023

zhaohehuhu commented Apr 17, 2023

fuchanghai left a comment

Choose a reason for hiding this comment

zhaohehuhu commented Apr 18, 2023

zhaohehuhu commented Feb 28, 2023 •

edited

Loading

codecov-commenter commented Mar 6, 2023 •

edited

Loading

fuchanghai commented Mar 7, 2023 •

edited

Loading