Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Created new ReplicaShardBatchAllocator #8992

Merged
merged 19 commits into from
Mar 20, 2024
Merged

Created new ReplicaShardBatchAllocator #8992

merged 19 commits into from
Mar 20, 2024

Conversation

shiv0408
Copy link
Member

@shiv0408 shiv0408 commented Jul 30, 2023 •
edited
Loading

to be used instead of ReplicaShardAllocator for batch calls

Description

Sub PR part for overall improvement of #5098

Note :

  1. UTs and ITs will be added in upcoming commits.
  2. Gradle check will fail on compilation because this PR is dependent on multiple other PRs
  3. Not creating a helper class for common code as old PrimaryShardAllocator class will be deprecated in future.
  4. Most of the code is copy pasted from PrimaryShardAllocator class, only changes are handling of multiple shards metadata and handling of single DiscoveryNode object in response for a batch of shards.
  5. CHANGELOG will be updated only after last PR is raised(being merged) for this project.

Related Issues

Resolves #8960

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@shiv0408
Created new ReplicaShardBatchAllocator to be used instead of ReplicaS…
0d0e5c2 
…hardAllocator for batch calls

Signed-off-by: Shivansh Arora <shivansh.arora@protonmail.com>
@shiv0408
fixed import path
8b47472 
Signed-off-by: Shivansh Arora <shivansh.arora@protonmail.com>
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@amkhar amkhar mentioned this pull request Jul 31, 2023
Open
13 tasks
@shiv0408
Cleaned up code and removed unnecessary ineligible shards
aa68e29 
Signed-off-by: Shivansh Arora <shivansh.arora@protonmail.com>
@shiv0408
Removed GatewayAllocator from PR
e3354b4 
Signed-off-by: Shivansh Arora <shivansh.arora@protonmail.com>
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@opensearch-trigger-bot
Copy link
Contributor

Compatibility status:

Checks if related components are compatible with change dd75a22

Incompatible components

Incompatible components: [https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/security-analytics.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

@github-actions
Copy link
Contributor

github-actions bot commented Aug 23, 2023 •
edited
Loading

Compatibility status:

Checks if related components are compatible with change ca53a44

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/flow-framework.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/performance-analyzer.git]

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Compatibility status:

Checks if related components are compatible with change 0c81e3f

Incompatible components

Skipped components

Compatible components

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@shiv0408 shiv0408 mentioned this pull request Sep 27, 2023
Merged
6 tasks
@opensearch-trigger-bot
Copy link
Contributor

This PR is stalled because it has been open for 30 days with no activity.

@opensearch-trigger-bot opensearch-trigger-bot bot added the stalled Issues that have stalled label Sep 27, 2023
@github-actions github-actions bot added bug Something isn't working Cluster Manager labels Oct 3, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Oct 3, 2023

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

github-actions bot commented Oct 3, 2023

Gradle Check (Jenkins) Run Completed with:

@shiv0408 shiv0408 marked this pull request as ready for review October 4, 2023 05:43
@github-actions github-actions bot added the enhancement Enhancement or improvement to existing feature or request label Mar 19, 2024
@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for 0d23917: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for aa7f288: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for 0d23917: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@shiv0408
Copy link
Member Author

Failed test org.opensearch.search.geo.GeoPointShapeQueryTests.testQueryLinearRing is already identified as flaky in #11688

failing again in https://build.ci.opensearch.org/job/gradle-check/35377/testReport/junit/

@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for 0d23917: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@shwetathareja shwetathareja added the v2.13.0 Issues and PRs related to version 2.13.0 label Mar 19, 2024
@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for 0d23917: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for 0d23917: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@amkhar
Copy link
Contributor

amkhar commented Mar 20, 2024

❌ Gradle check result for 0d23917: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Flaky #11688

@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for 0d23917: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@shwetathareja shwetathareja self-requested a review March 20, 2024 14:36
@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for ca53a44: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@shwetathareja shwetathareja mentioned this pull request Mar 20, 2024
Closed
@github-actions GitHub Actions
Copy link
Contributor

❕ Gradle check result for ca53a44: UNSTABLE

  • TEST FAILURES:
      1 org.opensearch.search.SearchWeightedRoutingIT.testStrictWeightedRoutingWithCustomString_FailOpenEnabled

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

@shwetathareja shwetathareja merged commit 4dbd6fa into opensearch-project:main Mar 20, 2024
31 checks passed
@shwetathareja shwetathareja added the backport 2.x Backport to 2.x branch label Mar 20, 2024
@opensearch-trigger-bot opensearch-trigger-bot bot mentioned this pull request Mar 20, 2024
Closed
opensearch-trigger-bot bot pushed a commit that referenced this pull request Mar 20, 2024
@github-actions
Created new ReplicaShardBatchAllocator (#8992)
124a699 
* Created new ReplicaShardBatchAllocator to be used instead of ReplicaShardAllocator for batch calls

Signed-off-by: Shivansh Arora <shivansh.arora@protonmail.com>
(cherry picked from commit 4dbd6fa)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@shwetathareja
Copy link
Member

shwetathareja commented Mar 20, 2024 •
edited
Loading

Out of last 7/10 gradle-check failures were due to #11688

@shwetathareja
Copy link
Member

org.opensearch.repositories.azure.AzureBlobContainerRetriesTests.testReadRangeBlobWithRetries - #6090

@amkhar amkhar added backport 2.x Backport to 2.x branch and removed backport 2.x Backport to 2.x branch labels Mar 21, 2024
opensearch-trigger-bot bot pushed a commit that referenced this pull request Mar 21, 2024
@github-actions
Created new ReplicaShardBatchAllocator (#8992)
de3dc99 
* Created new ReplicaShardBatchAllocator to be used instead of ReplicaShardAllocator for batch calls

Signed-off-by: Shivansh Arora <shivansh.arora@protonmail.com>
(cherry picked from commit 4dbd6fa)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@opensearch-trigger-bot opensearch-trigger-bot bot mentioned this pull request Mar 21, 2024
Merged
shwetathareja pushed a commit that referenced this pull request Mar 21, 2024
@opensearch-trigger-bot @github-actions
Created new ReplicaShardBatchAllocator (#8992) (#12814)
d371a3d 
* Created new ReplicaShardBatchAllocator to be used instead of ReplicaShardAllocator for batch calls


(cherry picked from commit 4dbd6fa)

Signed-off-by: Shivansh Arora <shivansh.arora@protonmail.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
shiv0408 added a commit to Gaurav614/OpenSearch that referenced this pull request Apr 25, 2024
@shiv0408
Created new ReplicaShardBatchAllocator (opensearch-project#8992)
a1df35b 
* Created new ReplicaShardBatchAllocator to be used instead of ReplicaShardAllocator for batch calls

Signed-off-by: Shivansh Arora <shivansh.arora@protonmail.com>
Signed-off-by: Shivansh Arora <hishiv@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vikasvb90 vikasvb90 vikasvb90 left review comments

@amkhar amkhar amkhar approved these changes

@shwetathareja shwetathareja shwetathareja approved these changes

@reta reta Awaiting requested review from reta reta is a code owner

@anasalkouz anasalkouz Awaiting requested review from anasalkouz anasalkouz is a code owner

@andrross andrross Awaiting requested review from andrross andrross is a code owner

@Bukhtawar Bukhtawar Awaiting requested review from Bukhtawar Bukhtawar is a code owner

@CEHENKLE CEHENKLE Awaiting requested review from CEHENKLE CEHENKLE is a code owner

@dblock dblock Awaiting requested review from dblock dblock is a code owner

@gbbafna gbbafna Awaiting requested review from gbbafna gbbafna is a code owner

@setiah setiah Awaiting requested review from setiah

@kartg kartg Awaiting requested review from kartg

@kotwanikunal kotwanikunal Awaiting requested review from kotwanikunal kotwanikunal is a code owner

@mch2 mch2 Awaiting requested review from mch2 mch2 is a code owner

@nknize nknize Awaiting requested review from nknize nknize is a code owner

@owaiskazi19 owaiskazi19 Awaiting requested review from owaiskazi19 owaiskazi19 is a code owner

@peternied peternied Awaiting requested review from peternied

@Rishikesh1159 Rishikesh1159 Awaiting requested review from Rishikesh1159 Rishikesh1159 is a code owner

@ryanbogan ryanbogan Awaiting requested review from ryanbogan

@saratvemulapalli saratvemulapalli Awaiting requested review from saratvemulapalli saratvemulapalli is a code owner

@dreamer-89 dreamer-89 Awaiting requested review from dreamer-89

@VachaShah VachaShah Awaiting requested review from VachaShah VachaShah is a code owner

@dbwiddis dbwiddis Awaiting requested review from dbwiddis dbwiddis is a code owner

@sachinpkale sachinpkale Awaiting requested review from sachinpkale sachinpkale is a code owner

@sohami sohami Awaiting requested review from sohami sohami is a code owner

@msfroh msfroh Awaiting requested review from msfroh msfroh is a code owner

@abbashus abbashus Awaiting requested review from abbashus

@adnapibar adnapibar Awaiting requested review from adnapibar

@tlfeng tlfeng Awaiting requested review from tlfeng

@khushbr khushbr Awaiting requested review from khushbr

Assignees
No one assigned
Labels
backport 2.x Backport to 2.x branch Cluster Manager enhancement Enhancement or improvement to existing feature or request skip-changelog v2.13.0 Issues and PRs related to version 2.13.0
Projects
Cluster Manager Project Board
Status: ✅ Done
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Enhancement] Add a batch allocator for building allocation decisions for multiple primary & replica shards
5 participants
@shiv0408 @amkhar @shwetathareja @vikasvb90 @khushbr