[BUG] testCancellationDuringQueryPhaseUsingRequestParameter flakey test

[dblock]

Describe the bug

testCancellationDuringQueryPhaseUsingRequestParameter

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.search.SearchCancellationIT.testCancellationDuringQueryPhaseUsingRequestParameter" -Dtests.seed=181F3F6D79C48BB2 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=lv -Dtests.timezone=Africa/Monrovia -Druntime.java=17

org.opensearch.search.SearchCancellationIT > testCancellationDuringQueryPhaseUsingRequestParameter FAILED
    java.lang.AssertionError: At least one shard should have failed. Actual: 0
        at __randomizedtesting.SeedInfo.seed([181F3F6D79C48BB2:9A8D0FA37A08DEA1]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failEquals(Assert.java:187)
        at org.junit.Assert.assertNotEquals(Assert.java:201)
        at org.opensearch.search.SearchCancellationIT.ensureSearchWasCancelled(SearchCancellationIT.java:167)
        at org.opensearch.search.SearchCancellationIT.testCancellationDuringQueryPhaseUsingRequestParameter(SearchCancellationIT.java:247)

#2654

[owaiskazi19]

Tried running the test for 400 times. It passed successfully

./gradlew ':server:internalClusterTest' --tests "org.opensearch.search.SearchCancellationIT.testCancellationDuringQueryPhaseUsingRequestParameter" -Dtests.seed=181F3F6D79C48BB2 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=lv -Dtests.timezone=Africa/Monrovia -Druntime.java=17 -Dtests.iters=400

> Configure project :qa:os
Cannot add task 'destructiveDistroTest.docker' as a task with that name already exists.
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 7.6
  OS Info               : Linux 5.11.0-1020-aws (amd64)
  Runtime JDK Version   : 17 (Oracle JDK)
  Runtime java.home     : /usr/lib/jvm/java-17/jdk-17.0.1
  Gradle JDK Version    : 17 (Oracle JDK)
  Gradle java.home      : /usr/lib/jvm/java-17/jdk-17.0.1
  Random Testing Seed   : 181F3F6D79C48BB2
  In FIPS 140 mode      : false
=======================================

> Task :server:internalClusterTest
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.BootstrapForTesting (file:/home/ubuntu/OpenSearch/test/framework/build/distributions/framework-3.0.0-SNAPSHOT.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.BootstrapForTesting
WARNING: System::setSecurityManager will be removed in a future release
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.gradle.api.internal.tasks.testing.worker.TestWorker (file:/home/ubuntu/.gradle/wrapper/dists/gradle-7.6-all/9f832ih6bniajn45pbmqhk2cw/gradle-7.6/lib/plugins/gradle-testing-base-7.6.jar)
WARNING: Please consider reporting this to the maintainers of org.gradle.api.internal.tasks.testing.worker.TestWorker
WARNING: System::setSecurityManager will be removed in a future release

BUILD SUCCESSFUL in 20m 6s

[minalsha]

Closed this issue until it resurfaces next time.