GEODE-9204: thread hung waiting for response

[kamilla1201]

The original PR #6392 was closed because I took over the work on this ticket.

A not serializable exception can cause a ServerConnection thread to get stuck
waiting for a reply from another member. This is caused by ReplyMessage
not handling a non-serializable "returnValue".

The fix is to catch and log the exception in the sending node and then
handle the problem in the receiving node.

I would have liked to serialize the "returnValue" field into a separate buffer
so as not to transmit a partial serialization of the object but that would require
an extra buffer copy for every ReplyMessage sent, and we send a lot of them.

Thank you for submitting a contribution to Apache Geode.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

For all changes:

• Is there a JIRA ticket associated with this PR? Is it referenced in the commit message?

• Has your PR been rebased against the latest commit within the target branch (typically develop)?

• Is your initial contribution a single, squashed commit?

• Does gradlew build run cleanly?

• Have you written or updated unit tests to verify your changes?

• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?

Note:

Please ensure that once the PR is submitted, check Concourse for build issues and
submit an update to your PR as soon as possible. If you need help, please send an
email to dev@geode.apache.org.

[Bill]

Approved!

[kamilla1201]

@BenjaminPerryRoss @boglesby @gesterzhou @kirklund @nabarunnag @pivotal-eshu Could you please take a look at this PR?

[agingade]

@kamilla1201
Other options myself and @dschneider-pivotal were considering about:
Is this issue very specific to cache loader? in that case, let's check if the value is serializable as part of load, if not then send error response back to the caller.
@dschneider-pivotal Is it a safe assumption to make, anything we load through the cache-loader needs to be serializable...If not, then we may have to see if its serializable based on the thread/caller ?

[dschneider-pivotal]

It is possible to have a loader on a local region in which case it would not need to be serializable. It is safer to defer the check until it is being serialized.

[gesterzhou]

Don't worry of the testEventIdOutOfOrderInPartitionRegionSingleHop failure in precheckin. It will be fixed in other geode tickets: GEODE-9242, GEODE-9373.

[lgtm-com]

This pull request introduces 1 alert when merging ec97322 into c9d4f68 - view on LGTM.com

new alerts:

• 1 for Potential output resource leak

[lgtm-com]

This pull request introduces 1 alert when merging c8efba6 into 2271b7f - view on LGTM.com

new alerts:

• 1 for Potential output resource leak

[kirklund]

Need to annotate this with @Rule or it won't be torn down between tests.

[kirklund]

Not a request for change, just a note... I recommend not importing and using SerializableRunnableIF. Best way to use VM.invoke with methods is to use void as the return type:

private void createRegionWithBadCacheLoader(Properties properties)

And then have the caller invoke it with a lambda:

cacheLoaderVM.invoke("create a region with a bad cache loader",
        () -> createRegionWithBadCacheLoader(properties));

[kamilla1201]

Need to run performance tests before merging this PR