Make IndicesRequestCacheCleanupIT more robust

[kiranprakash154]

Description

This is a followup to #14369
and has 3 changes

• use setup_index in the test testCacheWithInvalidation to keep it consistent with other tests.
• Increase refresh interval to rule out flakiness by keeping it way high - TimeValue.timeValueDays(1)
• Increase max wait time to 5 iteration cycles of cache cleanup cycles to consider for events like GC stop the world that is out of control and runs long. This would not increase the test run time since the backoff in assertBusy is logarithmic and should return after all the asserts passes.

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[kiranprakash154]

@andrross can you review this ?

[github-actions]

✅ Gradle check result for 6c9d08f: SUCCESS

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-14478-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 e8b7913926a9428a0a737569e6df44cfeff3defb
# Push it to GitHub
git push --set-upstream origin backport/backport-14478-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-14478-to-2.x.

[kiranprakash154]

@andrross I will create a manual backport, thanks for merging.

[andrross]

@kiranprakash154 The backport shouldn't fail unless we've missed a previous one. I don't think there's any reason this code should be different across the main and 2.x branches.

[kiranprakash154]

@kiranprakash154 The backport shouldn't fail unless we've missed a previous one. I don't think there's any reason this code should be different across the main and 2.x branches.

It failed because the previous backport was not merged yet, Jay merged it yesterday - #14482

I retried and looks like the auto backport worked - #14498