HDDS-2924. Fix Pipeline#nodeIdsHash collision issue. #478
Conversation
|
The checkstyle issue should be fixed in next commit. However, there is an issue when interop multi-raft with rackawareness, which is detected by the topology acceptance test. It is not related to this change but I will look into it. |
I have a JIRA https://issues.apache.org/jira/browse/HDDS-2923 to fix rack awareness issue,. We may track there. |
|
Thanks @timmylicheng. In this one, I include some minor fixes around PipelinePlacementPolicy to cover the non-fallback case where we have enough nodes on different racks but are not chosen properly. Agree, we should have the fall back fix in HDDS-2923. |
|
That is a pretty smart fix. You are absolutely right, we need no magic to find if a three node set is equal to another. +1, once @xiaoyuyao is happy. |
|
/retest after rebase on github |
|
push the correct rebase. |
|
Thanks @anuengineer and @timmylicheng for the reviews. I will merge it shortly. |
* HDDS-1577. Add default pipeline placement policy implementation. (#1366) (cherry picked from commit b640a5f6d53830aee4b9c2a7d17bf57c987962cd) * HDDS-1571. Create an interface for pipeline placement policy to support network topologies. (#1395) (cherry picked from commit 753fc6703a39154ed6013e44dbae572391748906) * HDDS-2089: Add createPipeline CLI. (#1418) (cherry picked from commit 326b5acd4a63fe46821919322867f5daff30750c) * HDDS-1569 Support creating multiple pipelines with same datanode. Contributed by Li Cheng. This closes #28 * HDDS-1572 Implement a Pipeline scrubber to clean up non-OPEN pipeline. (#237) * Rebase Fix * HDDS-2650 Fix createPipeline CLI. (#340) * HDDS-2035 Implement datanode level CLI to reveal pipeline relation. (#348) * Revert "HDDS-2650 Fix createPipeline CLI. (#340)" This reverts commit 7c71710. * HDDS-2650 Fix createPipeline CLI and make it message based. (#370) * HDDS-1574 Average out pipeline allocation on datanodes and add metrcs/test (#291) * Resolve rebase conflict. * HDDS-2756. Handle pipeline creation failure in different way when it exceeds pipeline limit Closes #401 * HDDS-2115 Add acceptance test for createPipeline CLI and datanode list CLI (#375) * HDDS-2115 Add acceptance test for createPipeline CLI and datanode list CLI. * HDDS-2772 Better management for pipeline creation limitation. (#410) * HDDS-2913 Update config names and CLI for multi-raft feature. (#462) * HDDS-2924. Fix Pipeline#nodeIdsHash collision issue. (#478) * HDDS-2923 Add fall-back protection for rack awareness in pipeline creation. (#516) * HDDS-3007 Fix CI test failure for TestSCMNodeManager. (#550) Co-authored-by: Sammi Chen <sammichen@apache.org> Co-authored-by: Xiaoyu Yao <xyao@apache.org>
What changes were proposed in this pull request?
Change to use the set of datanodes that form the pipeline to detect overlapped pipeline allocation.
Remove the nodeIdHash that could collide.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-2924
How was this patch tested?
unit tests