Deadlock happens in scenario where a read through also making a call to a cluster-singleton service

[hongchen2000]

Multiple clients using thin client connections to request the same data simultaneously.
The cache has read-through enabled and the data is to be loaded by a custom class that extends CacheStoreAdapter<>.

The following conditions results in a dead lock:

1. Number of clients simultaneously request the same data is more than the number of system threads. This results in all system threads are blocked and waiting for the first thread to finish loading the data
2. In the load() method, there is a call to a cluster-singleton service.

The load() method will wait for the service call to come back (blocked at the GridFutureAdapter.get0(), which is called by the GridServiceProxy.invokeMethod()).

However, it seems that the callback will require the availability of a system thread. However, since all system threads are wait for the callback, a dead lock happens.

Could you confirm why the service call back need a System thread be available? and what is the recommended way to get around this?

From this document, https://ignite.apache.org/docs/latest/perf-and-troubleshooting/thread-pools-tuning , It seems service call should use the Service Pool. But when I check the JMX when this deadlock happened, the service pool has available thread in both caller and callee node.