[CI] RemoteClusterConnectionTests testSlowNodeCanBeCancelled failing

[jkakavas]

This has failed 3 times in the last month with the same error

Build scan:
https://gradle-enterprise.elastic.co/s/mdcb6j6zngrbu/tests/:server:test/org.elasticsearch.transport.RemoteClusterConnectionTests/testSlowNodeCanBeCancelled

Reproduction line:
./gradlew ':server:test' --tests "org.elasticsearch.transport.RemoteClusterConnectionTests.testSlowNodeCanBeCancelled" -Dtests.seed=4D5816178B377259 -Dtests.locale=ru-RU -Dtests.timezone=America/Guadeloupe -Druntime.java=8

Applicable branches:
7.x

Reproduces locally?:
No

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.transport.RemoteClusterConnectionTests&tests.test=testSlowNodeCanBeCancelled

Failure excerpt:

junit.framework.AssertionFailedError: Unexpected exception type, expected AlreadyClosedException but got ConnectTransportException[[][207.254.60.47:60530] connect_exception]; nested: SocketException[Connection reset by peer];

  at __randomizedtesting.SeedInfo.seed([4D5816178B377259:82857EEC4445DD99]:0)
  at org.apache.lucene.util.LuceneTestCase.expectThrows(LuceneTestCase.java:2746)
  at org.apache.lucene.util.LuceneTestCase.expectThrows(LuceneTestCase.java:2734)
  at org.elasticsearch.transport.RemoteClusterConnectionTests.testSlowNodeCanBeCancelled(RemoteClusterConnectionTests.java:204)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-2)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
  at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
  at java.lang.Thread.run(Thread.java:748)

  Caused by: org.elasticsearch.transport.ConnectTransportException: [][207.254.60.47:60530] connect_exception

    at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:990)
    at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$0(ActionListener.java:277)
    at org.elasticsearch.core.CompletableContext.lambda$addListener$0(CompletableContext.java:31)
    at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
    at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
    at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
    at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
    at org.elasticsearch.core.CompletableContext.completeExceptionally(CompletableContext.java:46)
    at org.elasticsearch.nio.SocketChannelContext.connect(SocketChannelContext.java:131)
    at org.elasticsearch.nio.EventHandler.handleConnect(EventHandler.java:106)
    at org.elasticsearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:112)
    at org.elasticsearch.nio.NioSelector.attemptConnect(NioSelector.java:417)
    at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:226)
    at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:163)
    at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:120)
    at java.lang.Thread.run(Thread.java:748)

    Caused by: java.net.SocketException: Connection reset by peer

      at sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java:-2)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:715)
      at org.elasticsearch.nio.SocketChannelContext.connect(SocketChannelContext.java:128)
      at org.elasticsearch.nio.EventHandler.handleConnect(EventHandler.java:106)
      at org.elasticsearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:112)
      at org.elasticsearch.nio.NioSelector.attemptConnect(NioSelector.java:417)
      at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:226)
      at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:163)
      at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:120)
      at java.lang.Thread.run(Thread.java:748)

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[jkakavas]

Got this again on 7.16 https://gradle-enterprise.elastic.co/s/qaq7hwq22lnwe/tests/:server:test/org.elasticsearch.transport.RemoteClusterConnectionTests/testSlowNodeCanBeCancelled, can/should #79069 be backported @tbrooks8 ?