Add Customized State View Aggregation Feature#963
Merged
jiajunwang merged 23 commits intoapache:masterfrom Apr 23, 2020
Merged
Conversation
This commit contains the bare minimum properties for CustomizedView Helix property.
Add CustomizedViewAggregation as a cluster level config. This config defines the types of customized states that will be aggregated by Helix controller to generate a customized view. If Helix customers would like to have an aggregated view generated for their own states, they will need to add the type of the state to the list view in this config.
…e#792) In this commit the below APIs have been added. 1- addCustomizedStateAggregationConfig 2- removeCustomizedStateAggregationConfig. 3- addTypeToCustomizedStateAggregationConfig 4- removeTypeFromCustomizedStateAggregationConfig Tests have been added to check the functionality of these APIs.
…ig (apache#797) In this commit the below REST APIs have been added. 1- addCustomizedStateAggregationConfig 2- removeCustomizedStateAggregationConfig 3- getCustomizedStateAggregationConfig 4- updateCustomizedStateAggregationConfig Tests have been added to check the functionality of these REST APIs. Also some of the depricated calls have been updated.
Implement Helix APIs in CustomizedStateProvider for customers to operate on their own customized state. The available operations include update, get, and delete. To use CustomizedStateProvider, Helix user should initialize its factory and pass required parameters.
…apache#814) This commit contains the basic functionalities for CustomizedView RoutingTableProvider. Here are the new added functionalities: 1- CustomizedViewChangeListener 2- Addition of CustomizedView to helix PropertyType and PropertyKey 3- Implementation of CustomizedView cache. 4- Registering CallbackHandler for CustomizedView.
Add listeners for customized state and customized state aggregation config in Helix managers
Add tests to make CustomizedStateProvider tests comprehensive.
1. Implement a participant state cache for generalizing functions in both current state cache and customized state cache. 2. Implement an intermediate data structure to store the result of customized state computation and prepare the data to be the format that can be used by customized view computation later.
…pache#885) Rename customized state aggregation config to customized state config for future extendibility. Co-authored-by: Meng Zhang <mnzhang@mnzhang-mn1.linkedin.biz>
…e#859) Currently the update customized state method is made synchronized for concurrency control. This commit modifies the implementation of update to leave the responsibility of concurrency control to ZooKeeper by using updater to update the customize state. With delete method already implemented with updater, we can prevent unexpected change of the customize state data.
Update some cache functions for customized view aggregation
1. One stage is the computation stage for customized state. It takes the Zookeeper data of customized states and converts them to the formatted output used by the other stage. 2. The other stage is customized view aggregation stage. It will take the output from the customized state computation stage, and output the customized view to Zookeeper. 3. The two stages together compute the customized view from the customized states. 4. Unit tests are added to verify the correctness of the two stages.
In this commit, the routing table provider has been changed in a way to include customized view feature.
…ation. (apache#851) Add extra stages and pipelines in controller for customized state computation and customized view aggregation. Add refresh logic in resource data provider for customized view related data refresh. Add customized state event handling in CallbackHandler. Add integration test for customized view aggregation. Modify existing tests to verify new logic. Co-authored-by: Meng Zhang <mnzhang@mnzhang-mn1.linkedin.biz>
Fix minor issues in customized view aggregation logic and add some more tests. Co-authored-by: Meng Zhang <mnzhang@mnzhang-mn1.linkedin.biz>
In this commit, the custom implementation of customized view cache has been replaced with property cache implementation.
The integration test involves components: update customized state using customized view provider, and use routing table provider to get customized view snapshots which are aggregated in controller.
Modify customized state provider factory. The new factory can build a customized state provider with either Helix own manager or a customer input manager.
In this commit, a new logic is added which target the scenario where user disables and enables specific types. In this case, since the CustomizedView path for that type is removed by the controller, the router looses its listener. In this commit, we added root change lister and re-registers the listens again.
This PR moves the initialization of routing table provider to before class so it is initialized before any updates. Also, added several checks into the validation method to cover some edge cases.
jiajunwang
approved these changes
Apr 23, 2020
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issues
(#728 )
Description
This PR implements Helix customized state view aggregation feature. This feature allow Helix users to have their own defined states for each partition, provide them APIs to update the states when there's any change, and also help them aggregate the customized states into a customized view similar to existing external view. Finally, users could get the aggregated view from routing table provider.
Tests
(List the names of added unit/integration tests)
TestCustomizedStateUpdate
TestCustomizedStateComputationStage
TestCustomizedViewStage
TestComputeAndCleanupCustomizedView
TestCustomizedViewAggregation
[WARNING] Tests run: 1144, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 4,645.631 s - in TestSuite
[INFO]
[INFO] Results:
[INFO]
[WARNING] Tests run: 1144, Failures: 0, Errors: 0, Skipped: 1
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:17 h
[INFO] Finished at: 2020-04-23T12:02:40-07:00
helix-rest: (known issues, will be fixed in the following PRs)
[ERROR] Failures:
[ERROR] TestZkRoutingDataReader.testGetRoutingData:81 » ZkMarshalling java.io.NotSeria...
[ERROR] TestZkRoutingDataWriter.testAddMetadataStoreRealm:87 » ZkMarshalling java.io.S...
[INFO]
[ERROR] Tests run: 159, Failures: 2, Errors: 0, Skipped: 11
[INFO]
Commits
Code Quality
(helix-style-intellij.xml if IntelliJ IDE is used)