Wait until client receives member shutdown in ClientCacheThroughHazel…

…castInstanceTest (#18658) (#19195)

In `getCache_whenOwnerInstanceIsShutdown_thenOperateOnCacheFails`, we shutdown the owner
instance, and than make a put request from the client. On a sunny day, we expect to
get HazelcastClientNotActiveException since the request is made after calling
the instance shutdown.

But on some occasions, it might happen that although the member shutdown process
is complete, the client might not be aware of it yet. If we make a put request
in this scenario, the client will try to route put request to shutdown member,
get the connection close event some time after, set the invocation with
the TargetDisconnectedError. Since the put request is not retryable, the client
will finalize the put request with this exception and the test will fail.

As a solution, before making the put request, we wait until the connection
related to the shutdown member is removed from the client.

hazelcast/src/test/java/com/hazelcast/client/cache/instance/ClientCacheThroughHazelcastInstanceTest.java

@@ -31,7 +31,10 @@
 
 import javax.cache.spi.CachingProvider;
 
+import java.util.UUID;
+
 import static com.hazelcast.cache.CacheTestSupport.createClientCachingProvider;
+import static com.hazelcast.client.test.ClientTestSupport.makeSureDisconnectedFromServer;
 
 @RunWith(HazelcastParallelClassRunner.class)
 @Category({QuickTest.class, ParallelJVMTest.class})
@@ -83,7 +86,9 @@ protected HazelcastInstance createInstance(Config config) {
     @Override
     protected void shutdownOwnerInstance(HazelcastInstance instance) {
         if (ownerInstance != null) {
+            UUID memberUuid = ownerInstance.getLocalEndpoint().getUuid();
             ownerInstance.shutdown();
+            makeSureDisconnectedFromServer(instance, memberUuid);
         } else {
             throw new IllegalStateException("");
         }

hazelcast/src/test/java/com/hazelcast/client/test/ClientTestSupport.java

@@ -79,12 +79,12 @@ protected void unblockMessagesToInstance(HazelcastInstance instance, HazelcastIn
         ((TestClientRegistry.MockTcpClientConnectionManager) connectionManager).unblockTo(address);
     }
 
-    protected HazelcastClientInstanceImpl getHazelcastClientInstanceImpl(HazelcastInstance client) {
+    protected static HazelcastClientInstanceImpl getHazelcastClientInstanceImpl(HazelcastInstance client) {
         HazelcastClientProxy clientProxy = (HazelcastClientProxy) client;
         return clientProxy.client;
     }
 
-    protected void makeSureDisconnectedFromServer(final HazelcastInstance client, UUID memberUUID) {
+    public static void makeSureDisconnectedFromServer(final HazelcastInstance client, UUID memberUUID) {
         assertTrueEventually(() -> {
             ClientConnectionManager connectionManager = getHazelcastClientInstanceImpl(client).getConnectionManager();
             assertNull(connectionManager.getConnection(memberUUID));