[Segment Replication] Allocation and rebalancing based on average primary shard count per index #6422

dreamer-89 · 2023-02-21T19:57:05Z

Description

This change adds average primary shard count per node constraint to both allocation and rebalancing operations and basically implements the approach 1 as mentioned in #6210

Changes

Adds a new dynamic setting cluster.routing.allocation.balance.prefer_primary defaults to false. This setting defines if primary shard balance is needed or not.
Adds a new constraint isPrimaryShardsPerIndexPerNodeBreached which breaches when a node contains more than average primary shards of an index. When breached, the node weight calculation adds 100k to the node's weight resulting in lower chances for the node to be selected as target for unassigned shard allocation or relocation from rebalancing. Pleaset note, this is a soft limit which means even if node breaches constraint it can still be selected for allocation compared to hard limit from different allocation & rebalance deciders which removes node from allocation consideration.
Restructure the constraint classes to consume changes to dynamic setting. When setting is enabled, constraints are applied during allocation and rebalancing
Copies code from [Segment Replication] Consider primary shard balancing first #6325 to move primaries first for balancing. This is essential to converge the balancing round quickly and minimize relocations.
Unit and integration tests
There are few tests which currently are flaky due to limitation called out in [Segment Replication] Allocation changes to distribute primaries - Benchmark & document improvements using opensearch-benchmark. #6210 (comment) where optimal primary shard distribution can't be attained due to SameShardAllocationDecider allocation decider.

Issues Resolved

#6210

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Suraj Singh <surajrider@gmail.com>

mch2

LGTM, thanks @dreamer-89.

github-actions · 2023-03-02T00:24:46Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/11880/
CommitID: b28fe2d

dreamer-89 · 2023-03-02T00:25:25Z

@shwetathareja @nknize @Bukhtawar @vigyasharma : Need help with the review. CC @anasalkouz @mch2

As this is core change, it will be good to have one more review. @shwetathareja @nknize @Bukhtawar @vigyasharma ping for review. Will wait for couple of days for review.

github-actions · 2023-03-02T00:30:51Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.search.SearchWeightedRoutingIT.testSearchAggregationWithNetworkDisruption_FailOpenEnabled

URL: https://build.ci.opensearch.org/job/gradle-check/11881/
CommitID: f1ef7fc
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

CHANGELOG.md

Signed-off-by: Suraj Singh <surajrider@gmail.com>

dreamer-89 · 2023-03-02T00:56:30Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕

TEST FAILURES:
      1 org.opensearch.search.SearchWeightedRoutingIT.testSearchAggregationWithNetworkDisruption_FailOpenEnabled
URL: https://build.ci.opensearch.org/job/gradle-check/11881/

CommitID: f1ef7fc
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Tracked in #5957

github-actions · 2023-03-02T01:26:06Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/11888/
CommitID: 12e5021

dreamer-89 · 2023-03-03T23:43:19Z

Merging the changes now.
@shwetathareja @nknize @Bukhtawar @vigyasharma : Please feel free to comment if you have any questions/concerns.

…mary shard count per index (#6422) * [Segment Replication] Move primary shard first during rebalancing Signed-off-by: Suraj Singh <surajrider@gmail.com> * Address review comments Signed-off-by: Suraj Singh <surajrider@gmail.com> * Add primary weight constraint Signed-off-by: Suraj Singh <surajrider@gmail.com> * Add average primary shard count constraint for allocation and rebalancing operation Signed-off-by: Suraj Singh <surajrider@gmail.com> * Spotless fix and javadocs Signed-off-by: Suraj Singh <surajrider@gmail.com> * Add failing tests Signed-off-by: Suraj Singh <surajrider@gmail.com> * Fix unit tests Signed-off-by: Suraj Singh <surajrider@gmail.com> * Add unit test for nodes breaching multiple constraints Signed-off-by: Suraj Singh <surajrider@gmail.com> * Add comments and update unit test Signed-off-by: Suraj Singh <surajrider@gmail.com> * Address review comments Signed-off-by: Suraj Singh <surajrider@gmail.com> * Remove auto-expand replicas integration test Signed-off-by: Suraj Singh <surajrider@gmail.com> * Address review comments Signed-off-by: Suraj Singh <surajrider@gmail.com> * Update comments for tests Signed-off-by: Suraj Singh <surajrider@gmail.com> * Add changelog entry Signed-off-by: Suraj Singh <surajrider@gmail.com> * Remove extra changelog entry Signed-off-by: Suraj Singh <surajrider@gmail.com> --------- Signed-off-by: Suraj Singh <surajrider@gmail.com> (cherry picked from commit f61402a) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

nknize · 2023-03-04T00:26:51Z

No concerns, here. I haven't had a chance to dig deep on this but I did give it a quick once over. I like where this is going and great suggestion with the use of constraints.

I think merging is fine so we can let it bake. I'll be giving another pass when I get back to teasing out the common server components.

…mary shard count per index (#6422) (#6541) * [Segment Replication] Move primary shard first during rebalancing * Address review comments * Add primary weight constraint * Add average primary shard count constraint for allocation and rebalancing operation * Spotless fix and javadocs * Add failing tests * Fix unit tests * Add unit test for nodes breaching multiple constraints * Add comments and update unit test * Address review comments * Remove auto-expand replicas integration test * Address review comments * Update comments for tests * Add changelog entry * Remove extra changelog entry --------- (cherry picked from commit f61402a) Signed-off-by: Suraj Singh <surajrider@gmail.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…mary shard count per index (opensearch-project#6422) * [Segment Replication] Move primary shard first during rebalancing Signed-off-by: Suraj Singh <surajrider@gmail.com> * Address review comments Signed-off-by: Suraj Singh <surajrider@gmail.com> * Add primary weight constraint Signed-off-by: Suraj Singh <surajrider@gmail.com> * Add average primary shard count constraint for allocation and rebalancing operation Signed-off-by: Suraj Singh <surajrider@gmail.com> * Spotless fix and javadocs Signed-off-by: Suraj Singh <surajrider@gmail.com> * Add failing tests Signed-off-by: Suraj Singh <surajrider@gmail.com> * Fix unit tests Signed-off-by: Suraj Singh <surajrider@gmail.com> * Add unit test for nodes breaching multiple constraints Signed-off-by: Suraj Singh <surajrider@gmail.com> * Add comments and update unit test Signed-off-by: Suraj Singh <surajrider@gmail.com> * Address review comments Signed-off-by: Suraj Singh <surajrider@gmail.com> * Remove auto-expand replicas integration test Signed-off-by: Suraj Singh <surajrider@gmail.com> * Address review comments Signed-off-by: Suraj Singh <surajrider@gmail.com> * Update comments for tests Signed-off-by: Suraj Singh <surajrider@gmail.com> * Add changelog entry Signed-off-by: Suraj Singh <surajrider@gmail.com> * Remove extra changelog entry Signed-off-by: Suraj Singh <surajrider@gmail.com> --------- Signed-off-by: Suraj Singh <surajrider@gmail.com> Signed-off-by: Mingshi Liu <mingshl@amazon.com>

dreamer-89 requested review from reta, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, gbbafna, setiah, kartg, kotwanikunal, mch2, nknize, owaiskazi19, adnapibar, Rishikesh1159, ryanbogan, saratvemulapalli, shwetathareja, tlfeng, VachaShah and xuezhou25 as code owners February 21, 2023 19:57

dreamer-89 mentioned this pull request Feb 21, 2023

[Segment Replication] Consider primary shard balancing first #6325

Closed

6 tasks

dreamer-89 added the skip-changelog label Feb 21, 2023

opensearch-project deleted a comment from github-actions bot Feb 21, 2023

This comment was marked as outdated.

Sign in to view

dreamer-89 mentioned this pull request Feb 22, 2023

[Segment Replication] Allocation changes to distribute primaries - Benchmark & document improvements using opensearch-benchmark. #6210

Closed

dreamer-89 force-pushed the primary_allocation_with_constraint branch from fd7d113 to df7c663 Compare February 23, 2023 06:22

dreamer-89 changed the title ~~[Segment Replication] Average primary shard count per index constraint for allocation and rebalancing~~ [Segment Replication] Allocation and rebalancing based on average primary shard count per index Feb 23, 2023

This comment was marked as outdated.

Sign in to view

dreamer-89 marked this pull request as draft February 23, 2023 07:15

dreamer-89 added 5 commits March 1, 2023 15:06

Add comments and update unit test

d253109

Signed-off-by: Suraj Singh <surajrider@gmail.com>

Address review comments

bf8b4ad

Signed-off-by: Suraj Singh <surajrider@gmail.com>

Remove auto-expand replicas integration test

fab15bf

Signed-off-by: Suraj Singh <surajrider@gmail.com>

Address review comments

57754a5

Signed-off-by: Suraj Singh <surajrider@gmail.com>

Update comments for tests

b28fe2d

Signed-off-by: Suraj Singh <surajrider@gmail.com>

dreamer-89 force-pushed the primary_allocation_with_constraint branch from 5771038 to b28fe2d Compare March 1, 2023 23:55

Add changelog entry

f1ef7fc

Signed-off-by: Suraj Singh <surajrider@gmail.com>

dreamer-89 added v2.7.0 and removed skip-changelog labels Mar 1, 2023

mch2 approved these changes Mar 2, 2023

View reviewed changes

andrross reviewed Mar 2, 2023

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

Remove extra changelog entry

12e5021

Signed-off-by: Suraj Singh <surajrider@gmail.com>

dreamer-89 merged commit f61402a into opensearch-project:main Mar 3, 2023

dreamer-89 added the backport 2.x Backport to 2.x branch label Mar 3, 2023

opensearch-trigger-bot bot mentioned this pull request Mar 3, 2023

[Backport 2.x] [Segment Replication] Allocation and rebalancing based on average primary shard count per index #6541

Merged

This was referenced Mar 13, 2023

[Segment Replication] Primary shard based rebalancing across all indices #6642

Open

[Segment Replication] Add global primary shard balance constraint during allocation #6643

Merged

dreamer-89 mentioned this pull request Jun 14, 2023

primary weight factor should exchange primary and replica at the same in the smalll cluster in some case #8060

Closed

Arpit-Bandejiya mentioned this pull request Feb 8, 2024

[Feature Request] [Segment Replication] Balanced primary count across all nodes during rebalancing #12250

Closed

hlcianfagna mentioned this pull request Apr 26, 2024

Improve primary shards balancing/reduce primary shard write overhead crate/crate#15919

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Segment Replication] Allocation and rebalancing based on average primary shard count per index #6422

[Segment Replication] Allocation and rebalancing based on average primary shard count per index #6422

dreamer-89 commented Feb 21, 2023 •

edited

This comment was marked as outdated.

This comment was marked as outdated.

mch2 left a comment

github-actions bot commented Mar 2, 2023

dreamer-89 commented Mar 2, 2023 •

edited

github-actions bot commented Mar 2, 2023

dreamer-89 commented Mar 2, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Mar 2, 2023

dreamer-89 commented Mar 3, 2023

nknize commented Mar 4, 2023

[Segment Replication] Allocation and rebalancing based on average primary shard count per index #6422

[Segment Replication] Allocation and rebalancing based on average primary shard count per index #6422

Conversation

dreamer-89 commented Feb 21, 2023 • edited

Description

Changes

Issues Resolved

Check List

This comment was marked as outdated.

This comment was marked as outdated.

mch2 left a comment

Choose a reason for hiding this comment

github-actions bot commented Mar 2, 2023

Gradle Check (Jenkins) Run Completed with:

dreamer-89 commented Mar 2, 2023 • edited

github-actions bot commented Mar 2, 2023

Gradle Check (Jenkins) Run Completed with:

dreamer-89 commented Mar 2, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Mar 2, 2023

Gradle Check (Jenkins) Run Completed with:

dreamer-89 commented Mar 3, 2023

nknize commented Mar 4, 2023

dreamer-89 commented Feb 21, 2023 •

edited

dreamer-89 commented Mar 2, 2023 •

edited