Skip to content

[feature-6071][plugin] add datasource plugin. (#6071)#6361

Merged
CalvinKirs merged 46 commits intoapache:devfrom
Narcasserun:feature-6071
Nov 9, 2021
Merged

[feature-6071][plugin] add datasource plugin. (#6071)#6361
CalvinKirs merged 46 commits intoapache:devfrom
Narcasserun:feature-6071

Conversation

@Narcasserun
Copy link
Contributor

@Narcasserun Narcasserun commented Sep 26, 2021

  • remove the dependency of datasource in common and task-plugin module.

  • the new datasource-api module provides data source connection acquisition.

  • add all data source plug-in.

this closes #6071

* remove the dependency of datasource in common and task-plugin module.

* the new datasource-api module provides data source connection acquisition.

* add sqlserver data source plug-in.
@CalvinKirs CalvinKirs self-requested a review September 26, 2021 10:48
…duler into feature-6071

� Conflicts:
�	dolphinscheduler-dist/src/main/provisio/dolphinscheduler.xml
�	dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/runner/TaskExecuteThread.java
�	dolphinscheduler-server/src/test/java/org/apache/dolphinscheduler/server/master/consumer/TaskPriorityQueueConsumerTest.java
…duler into feature-6071

� Conflicts:
�	dolphinscheduler-dist/src/main/provisio/dolphinscheduler.xml
…inscheduler into feature-6071

� Conflicts:
�	dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/HadoopUtils.java
�	dolphinscheduler-task-plugin/dolphinscheduler-task-sql/src/main/java/org/apache/dolphinscheduler/plugin/task/sql/SqlTask.java
@codecov-commenter
Copy link

codecov-commenter commented Oct 29, 2021

Codecov Report

Merging #6361 (b8f2d0b) into dev (05bae0b) will decrease coverage by 0.35%.
The diff coverage is 26.12%.

Impacted file tree graph

@@             Coverage Diff              @@
##                dev    #6361      +/-   ##
============================================
- Coverage     41.43%   41.08%   -0.36%     
- Complexity     3600     3631      +31     
============================================
  Files           625      659      +34     
  Lines         26103    26425     +322     
  Branches       2847     2863      +16     
============================================
+ Hits          10816    10856      +40     
- Misses        14310    14585     +275     
- Partials        977      984       +7     
Impacted Files Coverage Δ
...scheduler/api/controller/DataSourceController.java 50.00% <ø> (ø)
.../org/apache/dolphinscheduler/common/Constants.java 79.16% <ø> (ø)
...che/dolphinscheduler/common/utils/CommonUtils.java 14.00% <ø> (-60.00%) ⬇️
...e/dolphinscheduler/common/utils/PropertyUtils.java 41.26% <ø> (ø)
.../org/apache/dolphinscheduler/dao/MonitorDBDao.java 0.00% <ø> (ø)
...apache/dolphinscheduler/dao/entity/DataSource.java 73.17% <ø> (ø)
...che/dolphinscheduler/dao/entity/MonitorRecord.java 0.00% <ø> (ø)
...scheduler/dao/upgrade/DolphinSchedulerManager.java 0.00% <ø> (ø)
...pache/dolphinscheduler/dao/upgrade/UpgradeDao.java 2.34% <ø> (ø)
...e/dolphinscheduler/dao/utils/MysqlPerformance.java 0.00% <ø> (ø)
... and 112 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 05bae0b...b8f2d0b. Read the comment docs.

CalvinKirs
CalvinKirs previously approved these changes Nov 3, 2021
Copy link
Member

@CalvinKirs CalvinKirs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@CalvinKirs
Copy link
Member

…duler into feature-6071

� Conflicts:
�	dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/upgrade/UpgradeDao.java
�	dolphinscheduler-spi/src/main/java/org/apache/dolphinscheduler/spi/DolphinSchedulerPlugin.java
�	pom.xml
requireNonNull(datasourceChannelFactory, "taskChannelFactory is null");

if (datasourceClientFactoryMap.putIfAbsent(datasourceChannelFactory.getName(), datasourceChannelFactory) != null) {
throw PluginException.getInstance(format("Task Plugin '%s' is already registered", datasourceChannelFactory.getName()));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DataSource Plugin

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

@caishunfeng
Copy link
Contributor

LGTM. Thank's for your great work.

…duler into feature-6071

� Conflicts:
�	dolphinscheduler-spi/src/main/java/org/apache/dolphinscheduler/spi/DolphinSchedulerPlugin.java
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class DataSourcePluginManager extends AbstractDolphinPluginManager {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please do not use AbstractDolphinPluginManager anymore, we've been migrating to Java native SPI and abandoning the AbstractDolphinPluginManager for easy development and usage. see #6723 #6712

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In order to facilitate development and use, it is also what I want. I look forward to further communication with you

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe you can take a look at this PR and look forward to your suggestions #6093

@sonarqubecloud
Copy link

sonarqubecloud bot commented Nov 9, 2021

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot E 1 Security Hotspot
Code Smell A 17 Code Smells

41.3% 41.3% Coverage
1.0% 1.0% Duplication

Copy link
Member

@CalvinKirs CalvinKirs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Narcasserun
Copy link
Contributor Author

#6071

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature][SPI] datasource plugin loading

5 participants