The client gets stuck on server restart

[ihsandemir]

It has been observed that the client may be hung when the server is restarted. The stack trace shows:

(gdb) bt
#0 0x00007fe53a5f17ce in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00000000004c6de3 in hazelcast::util::ConditionVariable::waitFor(hazelcast::util::Mutex&, long) ()
#2 0x00000000004bd205 in hazelcast::client::connection::CallFuture::get(long) ()
#3 0x00000000004bdb25 in hazelcast::client::connection::CallFuture::get() ()
#4 0x000000000044317e in hazelcast::client::proxy::ProxyImpl::invoke(std::auto_ptrhazelcast::client::protocol::ClientMessage, int) ()
#5 0x0000000000449dc3 in hazelcast::client::proxy::IMapImpl::get(hazelcast::client::serialization::pimpl::Data const&) ()
#6 0x0000000000406dc1 in hazelcast::client::IMap<std::string, std::string>::get (this=0x7ffe02f39a10, key=...)

at ./hazelcast-cpp-client/Linux_64_3.6-RC1/hazelcast/include/hazelcast/client/IMap.h:85

[ihsandemir]

I was able to produce a case in 3.6 client when restarting the servers on heavy client load. I observed that the input io thread was frozen while trying to set the future for the received response message. The response was the response to get partition request. This suggested that there is a problem with partition refresh. It was indeed fixed in #62 . My theory of how the problem occured is: PartitionRefreshTask destructor calling pthread_cancel which cancelled the thread after acquiring the call promise future lock and the lock was being held without being released. When the IO thread receives the response to partition refresh request, it gets stuck while trying to obtain the lock for the future to set the response.

[ihsandemir]

We fixed a minor error in ByteBuffer readBytes beside other fixes.