[core] Data client operations block forever if server fails.

[ericl]

It seems that data client operations hang in the blocking_send function if the server fails with error. This causes the client to hang.

Got Error from data channel -- shutting down: <_MultiThreadedRendezvous of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "Received http2 header with status: 502"
	debug_error_string = "{"created":"@1610421939.184399675","description":"Received http2 :status header with non-200 OK status","file":"src/core/ext/filters/http/client/http_client_filter.cc","file_line":129,"grpc_message":"Received http2 header with status: 502","grpc_status":14,"value":"502"}"
>
Got Error from logger channel -- shutting down: <_MultiThreadedRendezvous of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "Received http2 header with status: 502"
	debug_error_string = "{"created":"@1610421939.184408826","description":"Received http2 :status header with non-200 OK status","file":"src/core/ext/filters/http/client/http_client_filter.cc","file_line":129,"grpc_message":"Received http2 header with status: 502","grpc_status":14,"value":"502"}"
>
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/home/eric/Desktop/ray/python/ray/util/client/dataclient.py", line 76, in _data_main
    raise e
  File "/home/eric/Desktop/ray/python/ray/util/client/dataclient.py", line 61, in _data_main
    for response in resp_stream:
  File "/home/eric/.local/lib/python3.6/site-packages/grpc/_channel.py", line 416, in __next__
    return self._next()
  File "/home/eric/.local/lib/python3.6/site-packages/grpc/_channel.py", line 706, in _next
    raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "Received http2 header with status: 502"
	debug_error_string = "{"created":"@1610421939.184399675","description":"Received http2 :status header with non-200 OK status","file":"src/core/ext/filters/http/client/http_client_filter.cc","file_line":129,"grpc_message":"Received http2 header with status: 502","grpc_status":14,"value":"502"}"
>

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/home/eric/Desktop/ray/python/ray/util/client/logsclient.py", line 51, in _log_main
    raise e
  File "/home/eric/Desktop/ray/python/ray/util/client/logsclient.py", line 42, in _log_main
    for record in log_stream:
  File "/home/eric/.local/lib/python3.6/site-packages/grpc/_channel.py", line 416, in __next__
    return self._next()
  File "/home/eric/.local/lib/python3.6/site-packages/grpc/_channel.py", line 706, in _next
    raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "Received http2 header with status: 502"
	debug_error_string = "{"created":"@1610421939.184408826","description":"Received http2 :status header with non-200 OK status","file":"src/core/ext/filters/http/client/http_client_filter.cc","file_line":129,"grpc_message":"Received http2 header with status: 502","grpc_status":14,"value":"502"}"
>