core: fix retriablestream deadlock

[YifeiZhuang]

fix #10314

The deadlock can be reproduced with the UT in this change, and diff:

@@ -863,6 +870,12 @@ abstract class RetriableStream<ReqT> implements ClientStream {
         headers.discardAll(GRPC_PREVIOUS_RPC_ATTEMPTS);
         headers.put(GRPC_PREVIOUS_RPC_ATTEMPTS, String.valueOf(substream.previousAttemptCount));
       }
+      try  {
+        Thread.sleep(500);
+      } catch (InterruptedException ex) {
+        log.log(Level.INFO, "nothing");
+      }

The issue was that two streams from two transports held their transport lock while waiting for the other, thus deadlock. In this particular case, what happens is that in one subListener.close() it creates a new substream on other transports that requires that transport thread. Meanwhile another subListener.close() it receives headersRead() and tries to cancel all other streams that requires the corresponding transport lock. It is believe that it won't happen in netty, only in okhttp.

Solutions

We should break the deadlock in both places.
This fix is in headersRead() to have the cancel other stream from the call executor thread.
The other part of the fix is to have createSubstream to run from the call executor. This is be fixed in a follow up PR.

Tests

global TAP running results currently does not show bad signals. (failing due to already failing issues)

[YifeiZhuang]

@ejona86 did you want to take a look at the new changes? just comments.