NIFI-1897: Refactoring to allow requests to be replicated from a node to other nodes #488
Conversation
|
Reviewing... |
|
@markap14 I've been reviewing this PR and have some initial findings. As I'm reviewing I'm updating various comments and naming that still reflect the previous clustering model. I've gone ahead and updated those. Also, had some trouble starting the application in standalone mode but was able to resolve by updating the Spring Context where necessary. You can check out these modifications here [1]. Additionally, I noticed that none of the user documentation has been updated to reflect the new clustering model and a number of test resources have become stale. I've create separate JIRAs for addressing these issues [2] [3] at a later time. Will be doing some additional reviewing but so far so good. [1] mcgilman@05cbb92 |
|
Ran into a small issue with node (re)connection regarding Templates. It appears that Templates local to that node are replaced with the cluster set of Templates. There is an outstanding JIRA for allowing nodes to have a superset of the cluster Templates [1]. This issue needs to be resolved there. |
|
Just ran into an issue with the fields that are used in the flow fingerprint. I think we need to reconsider which fields are included. Specifically I had modified the penalty duration. This behavior is not new to this PR but warrants attention [1]. |
|
Ran into another issue around cluster coordinator selection. When the selected coordinator has a unheritable flow it is treated as the 'golden' copy as they are the coordinator. Other nodes fail to start up with different behavior as described in this PR [1]. |
|
Made a few more modifications to the handling of connection failure to ensure the coordinator is nodified. Also ensured the appropriate classloader when serializing templates, and that we are returning the replicated response in the Controller Resource. Please review them here [1]. Going merge this into master. Identified some issues (both new and old) that we'll need to address but can do so in a later PR. Thanks! [1] mcgilman@e2c13a8 |
|
Also just created another PR to consider building something to automated the conversion of the nifi.properties [1]. |
|
Modifications that you proposed look good to me. +1 |
|
This has been merged into master. Discovered a few new items and identified a few old items that we'll still need to knock out. Another item that we still need to tackle is ensuring that all node responses are merged according to the most restrictive view [1]. |
- Refactoring to allow requests to be replicated from a node to other nodes - Renaming cluster node connection/read timeout properties. - Renaming NCM DN to Cluster Coordinator DN. - Fixing default values in properties. - Starting to fix Spring context to load correctly in standalone mode. - Using the cluster protocol to handle connection failures instead of heartbeats. - Ensured replicate call is returned from ControllerResource. - Ensure the appropriate classloader when serializing templates. - Handling when the flow contents are null. - This closes apache#488
- Refactoring to allow requests to be replicated from a node to other nodes - Renaming cluster node connection/read timeout properties. - Renaming NCM DN to Cluster Coordinator DN. - Fixing default values in properties. - Starting to fix Spring context to load correctly in standalone mode. - Using the cluster protocol to handle connection failures instead of heartbeats. - Ensured replicate call is returned from ControllerResource. - Ensure the appropriate classloader when serializing templates. - Handling when the flow contents are null. - This closes apache#488
Nodes are now capable of replicating requests across the cluster and a Cluster Coordinator is auto-elected to monitor heartbeats and provide the up-to-date flow to newly joining nodes. The WebClusterManager and associated components has been removed!