Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BACKPORT] Wait until client receives member shutdown in ClientCacheThroughHazelcastInstanceTest #19195

Merged
merged 1 commit into from
Jul 28, 2021

Conversation

mdumandag
Copy link
Contributor

@mdumandag mdumandag commented Jul 28, 2021

In getCache_whenOwnerInstanceIsShutdown_thenOperateOnCacheFails, we shutdown the owner
instance, and than make a put request from the client. On a sunny day, we expect to
get HazelcastClientNotActiveException since the request is made after calling
the instance shutdown.

But on some occasions, it might happen that although the member shutdown process
is complete, the client might not be aware of it yet. If we make a put request
in this scenario, the client will try to route put request to shutdown member,
get the connection close event some time after, set the invocation with
the TargetDisconnectedError. Since the put request is not retryable, the client
will finalize the put request with this exception and the test will fail.

As a solution, before making the put request, we wait until the connection
related to the shutdown member is removed from the client.

Fixes: #18656
Backport of: #18658

…castInstanceTest (hazelcast#18658)

In `getCache_whenOwnerInstanceIsShutdown_thenOperateOnCacheFails`, we shutdown the owner
instance, and than make a put request from the client. On a sunny day, we expect to
get HazelcastClientNotActiveException since the request is made after calling
the instance shutdown.

But on some occasions, it might happen that although the member shutdown process
is complete, the client might not be aware of it yet. If we make a put request
in this scenario, the client will try to route put request to shutdown member,
get the connection close event some time after, set the invocation with
the TargetDisconnectedError. Since the put request is not retryable, the client
will finalize the put request with this exception and the test will fail.

As a solution, before making the put request, we wait until the connection
related to the shutdown member is removed from the client.
@mdumandag mdumandag added this to the 4.2.2 milestone Jul 28, 2021
@mdumandag mdumandag self-assigned this Jul 28, 2021
@hz-devops-test
Copy link

The job Hazelcast-pr-builder of your PR failed. (Hazelcast internal details: build log, artifacts).
Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log file
--------------------------
-------TEST FAILURE-------
--------------------------
[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   EvictionMaxSizePolicyTest.testPerNodePolicy_afterGracefulShutdown:120->HazelcastTestSupport.assertTrueEventually:1349->HazelcastTestSupport.assertTrueEventually:1247->lambda$testPerNodePolicy_afterGracefulShutdown$1:131 initialMapSize=2971, evictedKeyCount=874, map size is 2097 and it should be smaller than maxPartitionSize * PARTITION_COUNT which is 3000
[INFO] 
[ERROR] Tests run: 36356, Failures: 1, Errors: 0, Skipped: 988
[INFO] 

[ERROR] There are test failures.

@mdumandag
Copy link
Contributor Author

Unrelated test failure reported on #19206

@mdumandag
Copy link
Contributor Author

Merging this one approval since this is a clean backport of a test fix. The test failure is unrelated to the fixed test

@mdumandag mdumandag merged commit 9534b4c into hazelcast:4.2.z Jul 28, 2021
@mdumandag mdumandag deleted the client-cache-test-fix-4.2.z branch July 28, 2021 10:27
@mmedenjak mmedenjak changed the title Wait until client receives member shutdown in ClientCacheThroughHazelcastInstanceTest [BACKPORT] Wait until client receives member shutdown in ClientCacheThroughHazelcastInstanceTest Aug 30, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants