[Node Mgr] Suppot Node-MQ Topic Mapping #532

VanderChen · 2021-01-06T19:21:43Z

Implement #490 as Port Manager style and there still be a problem to solve.

chenpiaoping · 2021-01-12T09:38:53Z

...ces/data_plane_manager/src/main/java/com/futurewei/alcor/dataplane/cache/LocalCacheImpl.java

+
+    @Override
+    public void addNodeInfo(NodeInfo nodeInfo) throws Exception {
+        nodeInfoCache.addNodeInfo(nodeInfo);


Do we need to cache all the fields of NodeInfo here?

chenpiaoping · 2021-01-12T09:40:23Z

...ices/data_plane_manager/src/main/java/com/futurewei/alcor/dataplane/cache/NodeInfoCache.java

+            nodeInfo = nodeInfoCache.get(nodeIp);
+            assert nodeInfo != null;
+        } catch (Exception e) {
+            NodeInfo newNodeInfo = nodeManagerRestClient.getNodeInfo(nodeIp).getNodeInfo();


newNodeInfo may be null here?

A NodeInfoNotFound Exception has been added here.

chenpiaoping · 2021-01-12T09:41:02Z

...ices/data_plane_manager/src/main/java/com/futurewei/alcor/dataplane/cache/NodeInfoCache.java

+    }
+
+    @DurationStatistics
+    public synchronized void addNodeInfo(NodeInfo nodeInfo) throws Exception {


Do we need synchronized here?

I have remove it.

chenpiaoping · 2021-01-12T09:42:58Z

...e_manager/src/main/java/com/futurewei/alcor/dataplane/client/pulsar/DataPlaneClientImpl.java

-            }
-
+        for (String hostIp : hostIps) {
+            String groupTopic = localCache.getNodeInfo(hostIp).getGroupTopic();


localCache.getNodeInfo(hostIp) may rerturn null?

A Exception has been added here.

chenpiaoping · 2021-01-12T09:44:44Z

...ata_plane_manager/src/main/java/com/futurewei/alcor/dataplane/controller/NodeController.java

+            throw new ParameterNullOrEmptyException("");
+        }
+        try {
+            nodeService.createNodeInfo(nodeInfoJson);


updateNodeInfo()?

I have fixed this bug.

chenpiaoping · 2021-01-12T09:45:02Z

...ata_plane_manager/src/main/java/com/futurewei/alcor/dataplane/controller/NodeController.java

+            throw new ParameterNullOrEmptyException("");
+        }
+        try {
+            nodeService.createNodeInfo(nodeInfoJson);


deleteNodeInfo()?

I have fixed this bug.

chenpiaoping · 2021-01-12T09:45:52Z

..._plane_manager/src/main/java/com/futurewei/alcor/dataplane/service/impl/NodeServiceImpl.java

+    public void createNodeInfoBulk(BulkNodeInfoJson bulkNodeInfoJson) throws Exception {
+        List<NodeInfo> nodeInfos = bulkNodeInfoJson.getNodeInfos();
+        for (NodeInfo nodeInfo : nodeInfos) {
+            localCache.addNodeInfo(nodeInfo);


localCache should support batch operation?

addNodeInfoBulk method has been added to LocalCache

chenpiaoping · 2021-01-12T09:50:55Z

...ata_plane_manager/src/main/java/com/futurewei/alcor/dataplane/controller/NodeController.java

+    @DurationStatistics
+    public void createNodeInfo(@RequestBody NodeInfoJson nodeInfoJson) throws Exception {
+        if (nodeInfoJson == null) {
+            throw new ParameterNullOrEmptyException("");


More checks need to be done here?

More checks have been added here.

…o mq-mapping

codecov-io · 2021-01-26T14:41:13Z

Codecov Report

Merging #532 (0e45596) into master (17226df) will decrease coverage by 0.42%.
The diff coverage is 12.84%.

@@             Coverage Diff              @@
##             master     #532      +/-   ##
============================================
- Coverage     35.20%   34.78%   -0.43%     
- Complexity     1234     1241       +7     
============================================
  Files           491      502      +11     
  Lines         12157    12373     +216     
  Branches       1545     1557      +12     
============================================
+ Hits           4280     4304      +24     
- Misses         7303     7489     +186     
- Partials        574      580       +6

Impacted Files	Coverage Δ	Complexity Δ
...rewei/alcor/nodemanager/processor/NodeContext.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (?)`
...wei/alcor/nodemanager/request/AbstractRequest.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (?)`
...nodemanager/request/BulkCreateNodeInfoRequest.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (?)`
...cor/nodemanager/request/CreateNodeInfoRequest.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (?)`
...cor/nodemanager/request/DeleteNodeInfoRequest.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (?)`
...ewei/alcor/nodemanager/request/RequestManager.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (?)`
...cor/nodemanager/request/UpdateNodeInfoRequest.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (?)`
.../nodemanager/service/implement/NodeFileLoader.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
...nodemanager/service/implement/NodeServiceImpl.java	`1.35% <0.00%> (-0.44%)`	`2.00 <0.00> (ø)`
...i/alcor/nodemanager/utils/NodeManagerConstant.java	`0.00% <ø> (ø)`	`0.00 <0.00> (ø)`
... and 18 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 17226df...0e45596. Read the comment docs.

…o mq-mapping

chenpiaoping · 2021-02-04T08:05:09Z

LGTM

VanderChen · 2021-02-17T13:42:13Z

@xieus Excuse me. I wonder whether this PR is ready for merge

xieus

LGTM.

VanderChen added 7 commits December 23, 2020 22:20

Add dpm proxy in nm

690a9fd

Modify NM and DPM to implement MQ-Topic Mapping

8ebe5ef

Compelete MQ Topic mapping as PM style

920e7ae

Compelete MQ Topic mapping as PM style

6b325da

Merge remote-tracking branch 'upstream/master' into mq-mapping

3e077c8

Fix NodeInfo bug in PortManager UT

7d6ee3e

Fix NodeInfo bug in PortManager

2f37f3c

xieus requested a review from chenpiaoping January 7, 2021 01:31

xieus assigned VanderChen Jan 7, 2021

xieus added the feature feature development label Jan 7, 2021

xieus added this to the Version 1.0.2021.02.28 milestone Jan 7, 2021

chenpiaoping reviewed Jan 12, 2021

View reviewed changes

Merge branch 'master' of https://github.com/futurewei-cloud/alcor int…

3ce29ee

…o mq-mapping

VanderChen added 6 commits January 28, 2021 00:11

Implement get NodeInfo by NodeIp in local ignite

7e84ae0

Fix reviews

d94dcf4

Fix some error

0a80795

Implement get NodeInfo from NM

a559174

Implement get NodeInfo from NM

0a3fec0

Merge branch 'master' of https://github.com/futurewei-cloud/alcor int…

0e45596

…o mq-mapping

chenpiaoping approved these changes Feb 4, 2021

View reviewed changes

xieus approved these changes Feb 17, 2021

View reviewed changes

xieus changed the title ~~Suppot Node-MQ Topic Mapping~~ [Node Mgr] Suppot Node-MQ Topic Mapping Feb 17, 2021

xieus merged commit 91d6298 into futurewei-cloud:master Feb 17, 2021

xieus mentioned this pull request Feb 25, 2021

[MQ Scaling Path] Support production grade deployment for MQ-specific mapping. #490

Closed

VanderChen deleted the mq-mapping branch October 18, 2021 02:25

[Node Mgr] Suppot Node-MQ Topic Mapping #532

[Node Mgr] Suppot Node-MQ Topic Mapping #532

Uh oh!

Conversation

VanderChen commented Jan 6, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov-io commented Jan 26, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

chenpiaoping commented Feb 4, 2021

Uh oh!

VanderChen commented Feb 17, 2021

Uh oh!

xieus left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov-io commented Jan 26, 2021 •

edited

Loading