Fix Indices Request Cache ITs from flaking and timing out

[kiranprakash154]

Description

There were multiple issues created for Indices Request Cache's IT tests flaking & the suite timing out.

Addressing suite timing out

I believe the reason the tests were timing out because the new stalekey mgmt related tests added are spinning up an internal test cluster for every run (since we want different refresh intervals in some of the tests) and each test is run 4 times due to concurrent segment search and tiered caching parameters here

So i have moved the stalekey mgmt related tests out into its own class since stalekey mgmt is agnostic of cache type underneath and concurrent segment search.

Addressing Flakiness

Upon looking into the failures, the tests were failing on cache size assertions.

I think its due to the background refreshes and segment merges happening that is not in our control.
Hence I have gone ahead and addressed them in this PR by

• Disabling Refresh Intervals
• Disable segment merges

    // Disable index refreshing to avoid cache being invalidated mid-test
    .put(IndexSettings.INDEX_REFRESH_INTERVAL_SETTING.getKey(), TimeValue.timeValueMillis(-1))
	// Disable background segment merges invalidating the cache
    .put(MergePolicyProvider.INDEX_MERGE_ENABLED, false)

and also asserting the segment sizes

	// assert segment counts stay the same
	assertEquals(1, getSegmentCount(client, index));

Local Testing

I configured the test suite to run each test 100 times.

Related Issues

Resolves
#13711
#13437
#13540
#13503
#13949
#14173
#14242
#11374
#14243
#14288

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for e76c9b6: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[kiranprakash154]

❌ Gradle check result for e76c9b6: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

#12639

[github-actions]

❕ Gradle check result for fdec19c: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 71.75%. Comparing base (b15cb0c) to head (3c4e2a3).
Report is 455 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #14369      +/-   ##
============================================
+ Coverage     71.42%   71.75%   +0.33%     
- Complexity    59978    62120    +2142     
============================================
  Files          4985     5118     +133     
  Lines        282275   291878    +9603     
  Branches      40946    42189    +1243     
============================================
+ Hits         201603   209450    +7847     
- Misses        63999    65140    +1141     
- Partials      16673    17288     +615     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

✅ Gradle check result for 43940a8: SUCCESS

[dblock]

@kiranprakash154 Want to describe what this does and mark it ready for CR?

[github-actions]

✅ Gradle check result for ab90861: SUCCESS

[github-actions]

❕ Gradle check result for a4a3c34: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[kiranprakash154]

❕ Gradle check result for a4a3c34: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

AzureBlobStoreRepositoryTests.testSnapshotAndRestore - #14291

[github-actions]

✅ Gradle check result for 3c4e2a3: SUCCESS

[kiranprakash154]

@andrross / @sohami this is now ready for your review.

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-14369-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 bd56456baab283570da9b423152125124d482db3
# Push it to GitHub
git push --set-upstream origin backport/backport-14369-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-14369-to-2.x.

[andrross]

@kiranprakash154 Can you take care of the manual backport?

[kiranprakash154]

@kiranprakash154 Can you take care of the manual backport?

I will, thanks !

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-14369-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 bd56456baab283570da9b423152125124d482db3
# Push it to GitHub
git push --set-upstream origin backport/backport-14369-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-14369-to-2.x.

[kiranprakash154]

Created a manual backport - #14482