[ISSUE #367] Refactor connector #433

ruanwenjun · 2021-07-13T15:01:51Z

1.support load plugin from eventMeshPluginDir
2.remove connector plugin from runtime

In local develop, we can add dependency of eventmesh-connector-rocketmq in build.gradle.
And remove the plugin when packing.

codecov-commenter · 2021-07-13T15:24:07Z

Codecov Report

Merging #433 (a8a761e) into develop (db42296) will increase coverage by 0.04%.
The diff coverage is 42.27%.

❗ Current head a8a761e differs from pull request most recent head 5fc6c97. Consider uploading reports for the commit 5fc6c97 to get more accurate results

@@             Coverage Diff              @@
##             develop    #433      +/-   ##
============================================
+ Coverage       9.80%   9.84%   +0.04%     
- Complexity       282     283       +1     
============================================
  Files            228     228              
  Lines          10785   10829      +44     
  Branches         919     923       +4     
============================================
+ Hits            1057    1066       +9     
- Misses          9633    9666      +33     
- Partials          95      97       +2

Impacted Files	Coverage Δ
...h/connector/rocketmq/MessagingAccessPointImpl.java	`38.46% <ø> (ø)`
...pl/consumer/ConsumeMessageConcurrentlyService.java	`0.00% <ø> (ø)`
...eventmesh/connector/rocketmq/common/Constants.java	`0.00% <ø> (ø)`
.../connector/rocketmq/common/EventMeshConstants.java	`0.00% <ø> (ø)`
...ntmesh/connector/rocketmq/config/ClientConfig.java	`44.92% <ø> (ø)`
...connector/rocketmq/config/ClientConfiguration.java	`0.00% <ø> (ø)`
...onnector/rocketmq/config/ConfigurationWrapper.java	`0.00% <ø> (ø)`
.../connector/rocketmq/consumer/PushConsumerImpl.java	`40.56% <ø> (ø)`
...nector/rocketmq/consumer/RocketMQConsumerImpl.java	`0.00% <ø> (ø)`
...mesh/connector/rocketmq/domain/ConsumeRequest.java	`0.00% <ø> (ø)`
... and 20 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update db42296...5fc6c97. Read the comment docs.

qqeasonchen · 2021-07-15T11:15:30Z

@ruanwenjun Please hellp to fix these new coming up conflicts.

ruanwenjun · 2021-07-15T12:02:53Z

@ruanwenjun Please hellp to fix these new coming up conflicts.

Done

qqeasonchen · 2021-07-15T04:42:08Z

eventmesh-starter/build.gradle

@@ -16,7 +16,7 @@
 */

 dependencies {
-    implementation project(":eventmesh-runtime"), project(":eventmesh-connector-rocketmq")
-    testImplementation project(":eventmesh-runtime"), project(":eventmesh-connector-rocketmq")
+    implementation project(":eventmesh-runtime")


The starter moudle is useless after refactor, right?

Yes, if the starter module is only used to package dependencies, we can remove this module, but not sure if other things will need to be done in the future before runtime starting.

qqeasonchen · 2021-07-15T04:44:56Z

settings.gradle

 include 'eventmesh-sdk-java'
 include 'eventmesh-common'
 include 'eventmesh-connector-api'
 include 'eventmesh-starter'
 include 'eventmesh-test'
 include 'eventmesh-spi'
+include 'eventmesh-connector-plugin'
+include 'eventmesh-connector-plugin:eventmesh-connector-rocketmq'



Is it possible to build all plugins without dependencies in gradle file and run specific plugin use configuration?

Yes, we will not package the plugin into runtime, the all plugins will be packaged into jars, and copy to plugin directory.
We can configure the plugin directory.

qqeasonchen · 2021-07-16T08:03:23Z

@ruanwenjun Did you test this already?

ruanwenjun · 2021-07-16T09:33:07Z

@ruanwenjun Did you test this already?

Yes, the documentation of SPI will be added latter.

qqeasonchen · 2021-07-19T07:53:28Z

@ruanwenjun conflicts needs to be resolved.

Jackzeng1224 · 2021-07-19T11:21:47Z

@ruanwenjun hello,
When I was using eventmesh-connector-plugin, I found that rocketMq could not be loaded. I called rocketMq according to the previous method. Is there any other places that need to be configured?

1.support load plugin from eventMeshPluginDir 2.remove connector plugin from runtime

ruanwenjun · 2021-07-19T12:03:14Z

@Jackzeng1224 Which branch do you use? Can you get the log

initialize extension instance success, extensionType: interface org.apache.eventmesh.api.producer.MeshMQProducer, extensionName: rocketmq

at StartUp log ? You can get some doc in this pr.

ruanwenjun · 2021-07-19T12:04:42Z

@ruanwenjun conflicts needs to be resolved.

@qqeasonchen Thanks, done.

Jackzeng1224 · 2021-07-20T06:58:56Z

initialize extension instance success,

@ruanwenjun hello, The branch I am in is https://github.com/ruanwenjun/incubator-eventmesh/tree/dev_wenjun_refactorPlugin. I haven't seen it so far “initialize extension instance success, extensionType: interface org.apache.eventmesh.api.producer.MeshMQProducer, extensionName: rocketmq”

Jackzeng1224 · 2021-07-20T07:22:22Z

initialize extension instance success,
@ruanwenjun hello, The branch I am in is https://github.com/ruanwenjun/incubator-eventmesh/tree/dev_wenjun_refactorPlugin. I haven't seen it so far “initialize extension instance success, extensionType: interface org.apache.eventmesh.api.producer.MeshMQProducer, extensionName: rocketmq”

@ruanwenjun hello，I think this file configuration is not added.

ruanwenjun · 2021-07-20T07:22:57Z

@Jackzeng1224 Yes, now the SPI module will load plugin from dist/plugin/connector(this can be changed by environment) and classpath. In local develop, you can execute dist task and copyConnectorPlugin to install the connector plugin.
Or you can add the rocketmq plugin module in build.gradle.

ruanwenjun · 2021-07-20T07:26:43Z

@Jackzeng1224 In our design, we don't want to package all plugins in runtime jar.
The plugin is aim to separate the implementation and provide better scalability, user can choose to load the plugin by config.

Jackzeng1224 · 2021-07-20T11:05:56Z

@Jackzeng1224 Yes, now the SPI module will load plugin from dist/plugin/connector(this can be changed by environment) and classpath. In local develop, you can execute dist task and copyConnectorPlugin to install the connector plugin.
Or you can add the rocketmq plugin module in build.gradle.
OK！In the future, add the plug-in to eventmesh's build.gradle.

ruanwenjun · 2021-07-20T11:30:13Z

@Jackzeng1224 How is it going, do you meet other problem?

Jackzeng1224 · 2021-07-20T12:06:32Z

@Jackzeng1224 How is it going, do you meet other problem?

No. Very well designed 👍

qqeasonchen · 2021-07-20T12:20:35Z

When is the doc update?

@ruanwenjun Did you test this already?

Yes, the documentation of SPI will be added latter.

Have the docs updated now?

ruanwenjun · 2021-07-20T12:22:27Z

@qqeasonchen Yes, the docs has been updated.

qqeasonchen · 2021-07-20T12:43:06Z

docs/cn/instructions/eventmesh-runtime-quickstart.zh-CN.md

- eventmesh-connector-api : eventmesh插件接口定义模块
- eventmesh-connector-rocketmq : eventmesh rocketmq插件模块
+- eventmesh-connector-api : eventmesh connector插件接口定义模块
+- eventmesh-connector-plugin : eventmesh connector插件模块


eventmesh-connector-api was moved into eventmesh-connector-plugin，the description and picture are not updated

Ok, I will submit a pr to fix.

Ok, I will submit a pr to fix.

See #323 , i want to let the student to fix it :)

SteveYurongSu · 2021-07-21T01:29:43Z

style/checkStyle.xml

@@ -44,4 +50,9 @@
      <property name="format" value="^(org)\.apache(\.[a-zA-Z][a-zA-Z0-9]*)+$"/>
    </module>
  </module>
+
+  <module name="BeforeExecutionExclusionFileFilter">
+    <property name="fileNamePattern" value="./eventmesh-runtime/conf/sChat2.jks$"/>


Don't forget to remove this module :D

* [ISSUE apache#367] Refactor connector 1.support load plugin from eventMeshPluginDir 2.remove connector plugin from runtime * Add docs of plugin * Remove connector-api in connector-plugin

* [ISSUE #367] Refactor connector 1.support load plugin from eventMeshPluginDir 2.remove connector plugin from runtime * Add docs of plugin * Remove connector-api in connector-plugin

ruanwenjun force-pushed the dev_wenjun_refactorPlugin branch from 0f93c69 to 9620f8f Compare July 13, 2021 15:16

ruanwenjun mentioned this pull request Jul 14, 2021

[ISSUE #453] OpenSchema integration with EventMesh #434

Closed

ruanwenjun force-pushed the dev_wenjun_refactorPlugin branch 2 times, most recently from 7afa346 to 3606d55 Compare July 15, 2021 12:02

qqeasonchen reviewed Jul 16, 2021

View reviewed changes

ruanwenjun force-pushed the dev_wenjun_refactorPlugin branch 13 times, most recently from 26e9071 to d936d6c Compare July 17, 2021 03:12

ruanwenjun mentioned this pull request Jul 17, 2021

[ISSUE #442] Fix findings filtered by Checkstyle workflow #443

Merged

ruanwenjun force-pushed the dev_wenjun_refactorPlugin branch from d936d6c to 979dd51 Compare July 17, 2021 08:41

ruanwenjun added 2 commits July 19, 2021 19:25

[ISSUE apache#367] Refactor connector

9f169c3

1.support load plugin from eventMeshPluginDir 2.remove connector plugin from runtime

Add docs of plugin

15c51f7

ruanwenjun force-pushed the dev_wenjun_refactorPlugin branch from 979dd51 to b0c09e5 Compare July 19, 2021 11:45

ruanwenjun force-pushed the dev_wenjun_refactorPlugin branch from b0c09e5 to bf6de22 Compare July 19, 2021 11:48

Remove connector-api in connector-plugin

5fc6c97

ruanwenjun force-pushed the dev_wenjun_refactorPlugin branch from bf6de22 to 5fc6c97 Compare July 19, 2021 11:59

qqeasonchen merged commit 7850620 into apache:develop Jul 20, 2021

qqeasonchen reviewed Jul 20, 2021

View reviewed changes

SteveYurongSu reviewed Jul 21, 2021

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ISSUE #367] Refactor connector #433

[ISSUE #367] Refactor connector #433

ruanwenjun commented Jul 13, 2021

codecov-commenter commented Jul 13, 2021 •

edited

Loading

qqeasonchen commented Jul 15, 2021

ruanwenjun commented Jul 15, 2021

qqeasonchen Jul 15, 2021

ruanwenjun Jul 16, 2021

qqeasonchen Jul 15, 2021

ruanwenjun Jul 16, 2021

qqeasonchen commented Jul 16, 2021

ruanwenjun commented Jul 16, 2021 •

edited

Loading

qqeasonchen commented Jul 19, 2021

Jackzeng1224 commented Jul 19, 2021

ruanwenjun commented Jul 19, 2021 •

edited

Loading

ruanwenjun commented Jul 19, 2021

Jackzeng1224 commented Jul 20, 2021

Jackzeng1224 commented Jul 20, 2021

ruanwenjun commented Jul 20, 2021

ruanwenjun commented Jul 20, 2021

Jackzeng1224 commented Jul 20, 2021 •

edited

Loading

ruanwenjun commented Jul 20, 2021

Jackzeng1224 commented Jul 20, 2021 •

edited

Loading

qqeasonchen commented Jul 20, 2021

ruanwenjun commented Jul 20, 2021

qqeasonchen Jul 20, 2021

ruanwenjun Jul 21, 2021

qqeasonchen Jul 21, 2021

ruanwenjun Jul 21, 2021

SteveYurongSu Jul 21, 2021

ruanwenjun Jul 21, 2021

[ISSUE #367] Refactor connector #433

[ISSUE #367] Refactor connector #433

Conversation

ruanwenjun commented Jul 13, 2021

codecov-commenter commented Jul 13, 2021 • edited Loading

Codecov Report

qqeasonchen commented Jul 15, 2021

ruanwenjun commented Jul 15, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qqeasonchen commented Jul 16, 2021

ruanwenjun commented Jul 16, 2021 • edited Loading

qqeasonchen commented Jul 19, 2021

Jackzeng1224 commented Jul 19, 2021

ruanwenjun commented Jul 19, 2021 • edited Loading

ruanwenjun commented Jul 19, 2021

Jackzeng1224 commented Jul 20, 2021

Jackzeng1224 commented Jul 20, 2021

ruanwenjun commented Jul 20, 2021

ruanwenjun commented Jul 20, 2021

Jackzeng1224 commented Jul 20, 2021 • edited Loading

ruanwenjun commented Jul 20, 2021

Jackzeng1224 commented Jul 20, 2021 • edited Loading

qqeasonchen commented Jul 20, 2021

ruanwenjun commented Jul 20, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-commenter commented Jul 13, 2021 •

edited

Loading

ruanwenjun commented Jul 16, 2021 •

edited

Loading

ruanwenjun commented Jul 19, 2021 •

edited

Loading

Jackzeng1224 commented Jul 20, 2021 •

edited

Loading

Jackzeng1224 commented Jul 20, 2021 •

edited

Loading