test: increase await assertion timeout in OpensearchExporterIT IndexSettingsTest to overcome flaky test runs

[mustafadagher]

Description

Increase await assertion timeout to overcome flaky test runs

Policy change is an asynchronous background process in opensearch, that's why we use awaits before asserts to reduce flaky results

Related issues

closes #17279

Definition of Done

Not all items need to be done depending on the issue and the pull request.

Code changes:

• The changes are backwards compatibility with previous versions
• If it fixes a bug then PRs are created to backport the fix to the last two minor versions. You can trigger a backport by assigning labels (e.g. backport stable/1.3) to the PR, in case that fails you need to create backports manually.

Testing:

• There are unit/integration tests that verify all acceptance criterias of the issue
• New tests are written to ensure backwards compatibility with further versions
• The behavior is tested manually
• The change has been verified by a QA run
• The impact of the changes is verified by a benchmark

Documentation:

• The documentation is updated (e.g. BPMN reference, configuration, examples, get-started guides, etc.)
• If the PR changes how BPMN processes are validated (e.g. support new BPMN element) then the Camunda modeling team should be informed to adjust the BPMN linting.

Other teams:
If the change impacts another team an issue has been created for this team, explaining what they need to do to support this change.

• Tasklist
• Web Modeler
• Desktop Modeler
• Optimize

Please refer to our review guidelines.

[megglos]

👍

[mustafadagher]

While policy change is indeed asynchronous, and this change is trying to increase the timeout to avoid flakiness in test runs. this specific failure here, and the one that triggered that PR should never have happened.

It's failing at this line, in the beginning of the test.
At this point above we are trying to create an index with retention initially NOT enabled. This assert is just making sure that this index has no lifecycle policy. which should always be the case. we are not adding or removing a policy at this point, so waiting can't help this failure.

I've been trying to reproduce this locally to see what might have been the problem, but unfortunately, I can't. it never fails locally!

CC: @megglos

[megglos]

While policy change is indeed asynchronous, and this change is trying to increase the timeout to avoid flakiness in test runs. this specific failure here, and the one that triggered that PR should never have happened.

It's failing at this line, in the beginning of the test. At this point above we are trying to create an index with retention initially NOT enabled. This assert is just making sure that this index has no lifecycle policy. which should always be the case. we are not adding or removing a policy at this point, so waiting can't help this failure.

I've been trying to reproduce this locally to see what might have been the problem, but unfortunately, I can't. it never fails locally!

CC: @megglos

hmm could it be that there are template leftovers from a previous test? meaning maybe we need to extend the cleanup to not just cleanup indices but also potentially existing templates

Edit: however that should be overridden by the previous export

[mustafadagher]

While policy change is indeed asynchronous, and this change is trying to increase the timeout to avoid flakiness in test runs. this specific failure here, and the one that triggered that PR should never have happened.
It's failing at this line, in the beginning of the test. At this point above we are trying to create an index with retention initially NOT enabled. This assert is just making sure that this index has no lifecycle policy. which should always be the case. we are not adding or removing a policy at this point, so waiting can't help this failure.
I've been trying to reproduce this locally to see what might have been the problem, but unfortunately, I can't. it never fails locally!
CC: @megglos

hmm could it be that there are template leftovers from a previous test? meaning maybe we need to extend the cleanup to not just cleanup indices but also potentially existing templates

Edit: however that should be overridden by the previous export

I made one try at expanding wildcards to all (open and closed) indices when deleting indices before each test. I'm not sure this is the problem though. it's just a guess.

This is also one other API where opensearch diverges from elasticsearch.

[mustafadagher]

should this be backported to 8.4, 8.5 too?
I don't see the need, except for consistency.

[mustafadagher]

/backport stable/8.4

[backport-action]

Successfully created backport PR for stable/8.4:

• [Backport stable/8.4] test: increase await assertion timeout in OpensearchExporterIT IndexSettingsTest to overcome flaky test runs #17309

[backport-action]

Backport failed for stable/8.4, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin stable/8.4
git worktree add -d .worktree/backport-17282-to-stable/8.4 origin/stable/8.4
cd .worktree/backport-17282-to-stable/8.4
git switch --create backport-17282-to-stable/8.4
git cherry-pick -x 3197e92ced339fce6266acb3ae790383969d2a0b 16cfe26acc27fcee899525d5ddd569b6a0d9d64c

[backport-action]

Successfully created backport PR for stable/8.5:

• [Backport stable/8.5] test: increase await assertion timeout in OpensearchExporterIT IndexSettingsTest to overcome flaky test runs #17312