Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DSIP-19] Support zeppelin connections in the connection center, as well as external connections to the connection center in zeppelin tasks #14434

Merged
merged 31 commits into from
Oct 30, 2023
Merged

[DSIP-19] Support zeppelin connections in the connection center, as well as external connections to the connection center in zeppelin tasks #14434

merged 31 commits into from
Oct 30, 2023

Conversation

xdu-chenrj
Copy link
Contributor

Support zeppelin connections in the connection center, as well as external connections to the connection center in zeppelin tasks

issue: #14338

Related issues: [DSIP-19][Feature] Add connection center feature for DS #10283

Mail: https://lists.apache.org/thread/xl6pb3sbrt0ffrf1fltcph39s9w1pjlx

@github-actions github-actions bot added UI ui and front end related backend e2e e2e test labels Jun 30, 2023
@xdu-chenrj
Copy link
Contributor Author

Tested locally.

image

image

@xdu-chenrj
Copy link
Contributor Author

@EricGao888
I want to obtain the type parameter of the zeppelin task, but I am not sure how it is transmitted from the front-end. In order to quickly advance, I directly wrote a fixed value.

When building connection parameters in DataSourceUtils, it will be built based on the type of task.

 DataSourceUtils.buildConnectionParams(DbType.valueOf("ZEPPELIN"), zeppelinTaskExecutionContext.getConnectionParams());

@Radeity
Copy link
Member

Radeity commented Jul 1, 2023 •
edited
Loading

Hi, @xdu-chenrj , thanks for your contribution :D

Some suggestions for you before taking further look of this PR :

  • You should better follow the PR template to describe this PR.
  • Please add missing license header for dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-zeppelin/pom.xml.
  • Please avoid using wild card import in codes, e.g. import org.apache.dolphinscheduler.plugin.task.api.*;.
  • Please run mvn spotless:apply to format your code first.
  • Please handle conflicts first.

@xdu-chenrj
Copy link
Contributor Author

Hi, @xdu-chenrj , thanks for your contribution :D

Some suggestions for you before taking further look of this PR :

  • You should better follow the PR template to describe this PR.
  • Please add missing license header for dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-zeppelin/pom.xml.
  • Please avoid using wild card import in codes, e.g. import org.apache.dolphinscheduler.plugin.task.api.*;.
  • Please run mvn spotless:apply to format your code first.
  • Please handle conflicts first.

Okay, I got it

@xdu-chenrj xdu-chenrj changed the title [DSIP-2019] Support zeppelin connections in the connection center, as well as external connections to the connection center in zeppelin tasks [DSIP-19] Support zeppelin connections in the connection center, as well as external connections to the connection center in zeppelin tasks Jul 3, 2023
@EricGao888 EricGao888 added improvement make more easy to user or prompt friendly 3.2.0 for 3.2.0 version labels Jul 6, 2023
@EricGao888 EricGao888 added this to the 3.2.0 milestone Jul 6, 2023
EricGao888
EricGao888 reviewed Jul 6, 2023
View reviewed changes
.asf.yaml Outdated Show resolved Hide resolved
EricGao888
EricGao888 reviewed Jul 6, 2023
View reviewed changes
Copy link
Member

@EricGao888 EricGao888 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW, could u plz update related UT cases for Zeppelin tasks?

songjianet
songjianet previously approved these changes Jul 7, 2023
View reviewed changes
@kezhenxu94 kezhenxu94 removed their request for review July 7, 2023 01:23
@xdu-chenrj
Copy link
Contributor Author

BTW, could u plz update related UT cases for Zeppelin tasks?

Okay, I got it

@codecov-commenter
Copy link

codecov-commenter commented Jul 21, 2023 •
edited
Loading

Codecov Report

Merging #14434 (28aaf14) into dev (c0ed681) will increase coverage by 0.05%.
The diff coverage is 59.79%.

❗ Current head 28aaf14 differs from pull request most recent head 2a3b90f. Consider uploading reports for the commit 2a3b90f to get more accurate results

@@             Coverage Diff              @@
##                dev   #14434      +/-   ##
============================================
+ Coverage     38.09%   38.15%   +0.05%     
- Complexity     4642     4660      +18     
============================================
  Files          1262     1269       +7     
  Lines         45155    45238      +83     
  Branches       4944     4947       +3     
============================================
+ Hits          17203    17259      +56     
- Misses        26060    26087      +27     
  Partials       1892     1892
Files Coverage Δ
...rce/zeppelin/param/ZeppelinDataSourceParamDTO.java 0.00% <0.00%> (ø)
...uler/plugin/task/zeppelin/ZeppelinTaskChannel.java 0.00% <0.00%> (ø)
...in/task/zeppelin/ZeppelinTaskExecutionContext.java 80.00% <80.00%> (ø)
.../org/apache/dolphinscheduler/spi/enums/DbType.java 0.00% <0.00%> (ø)
...datasource/zeppelin/ZeppelinDataSourceChannel.java 0.00% <0.00%> (ø)
...rce/zeppelin/ZeppelinDataSourceChannelFactory.java 0.00% <0.00%> (ø)
...gin/datasource/zeppelin/ZeppelinClientWrapper.java 71.42% <71.42%> (ø)
...uler/plugin/datasource/zeppelin/ZeppelinUtils.java 0.00% <0.00%> (ø)
...duler/plugin/task/zeppelin/ZeppelinParameters.java 58.33% <60.00%> (+25.00%) ⬆️
...inscheduler/plugin/task/zeppelin/ZeppelinTask.java 55.33% <62.50%> (+5.33%) ⬆️
... and 1 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@sonarcloud
Copy link

sonarcloud bot commented Jul 21, 2023

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

No Coverage information No Coverage information
No Duplication information No Duplication information

@zhongjiajie zhongjiajie modified the milestones: 3.2.0, 3.3.0 Aug 30, 2023
@EricGao888
Copy link
Member

@xdu-chenrj Could please resolve the conflicts? Together with other PRs related to DSIP-19. Thanks : )

EricGao888
EricGao888 approved these changes Oct 30, 2023
View reviewed changes
Copy link
Member

@EricGao888 EricGao888 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Waiting for CI to pass

@sonarcloud
Copy link

sonarcloud bot commented Oct 30, 2023

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

No Coverage information No Coverage information
No Duplication information No Duplication information

@EricGao888 EricGao888 merged commit e99c595 into apache:dev Oct 30, 2023
53 of 55 checks passed
@EricGao888
Copy link
Member

Thanks for helping improve zeppelin task plugin @xdu-chenrj

github-advanced-security[bot]
github-advanced-security bot found potential problems Oct 30, 2023
View reviewed changes
...va/org/apache/dolphinscheduler/plugin/datasource/zeppelin/param/ZeppelinConnectionParam.java

import com.fasterxml.jackson.annotation.JsonInclude;

@Data

Check notice

Code scanning / CodeQL

Missing Override annotation Note

This method overrides
ConnectionParam.setPassword
Loading
; it is advisable to add an Override annotation.
...va/org/apache/dolphinscheduler/plugin/datasource/zeppelin/param/ZeppelinConnectionParam.java

protected String username;

protected String password;

Check notice

Code scanning / CodeQL

Missing Override annotation Note

This method overrides
ConnectionParam.getPassword
Loading
; it is advisable to add an Override annotation.
xdu-chenrj added a commit to xdu-chenrj/dolphinscheduler that referenced this pull request Oct 30, 2023
@xdu-chenrj @EricGao888
[DSIP-19] Support zeppelin connections in the connection center, as w…
8418842 
…ell as external connections to the connection center in zeppelin tasks (apache#14434)

* Refactoring zeppelin task plugin with connections managed in connection center

---------

Co-authored-by: Eric Gao <ericgao.apache@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@EricGao888 EricGao888 EricGao888 approved these changes

@Amy0104 Amy0104 Awaiting requested review from Amy0104 Amy0104 is a code owner

@caishunfeng caishunfeng Awaiting requested review from caishunfeng caishunfeng is a code owner

@SbloodyS SbloodyS Awaiting requested review from SbloodyS SbloodyS is a code owner

@zhuangchong zhuangchong Awaiting requested review from zhuangchong zhuangchong is a code owner

@songjianet songjianet Awaiting requested review from songjianet songjianet is a code owner

Assignees

@xdu-chenrj xdu-chenrj

Labels
3.2.0 for 3.2.0 version backend e2e e2e test improvement make more easy to user or prompt friendly ready-to-merge UI ui and front end related
Projects
None yet
Milestone
3.3.0
Development

Successfully merging this pull request may close these issues.

None yet
6 participants
@xdu-chenrj @Radeity @codecov-commenter @EricGao888 @songjianet @zhongjiajie