[Python] Multi-thread support for python asyncio gRPC clients

[RonanMcCarthy]

This issue was created at the request of @lidizheng on a stackoverflow post.

What version of gRPC and what language are you using?

• Python version: 3.8.6
• grpcio version: 1.35.0

What operating system (Linux, Windows,...) and version?

Issue observed on multiple macOS versions as well as within Ubuntu based docker containers. We have been testing with the python:3.8.6-slim-buster image.

What runtime / compiler are you using (e.g. python version or version of gcc)

Python version: 3.8.6

What did you do?

1. Create a gRPC server and launch it
2. Create multiple (two is enough) workers on separate threads
3. Have each thread make calls to the server simultaneously

What did you expect to see?

No issies.

What did you see instead?

A stream of the following errors:

2021-01-27 09:33:56,937 ERROR [asyncio] [thread_0] Exception in callback PollerCompletionQueue._handle_events()()
handle: )()>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "src/python/grpcio/grpc/_cython/_cygrpc/aio/completion_queue.pyx.pxi", line 147, in grpc._cython.cygrpc.PollerCompletionQueue._handle_events
BlockingIOError: [Errno 11] Resource temporarily unavailable
2021-01-27 09:33:56,937 ERROR [asyncio] [thread_1] Exception in callback PollerCompletionQueue._handle_events()()
handle: )()>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "src/python/grpcio/grpc/_cython/_cygrpc/aio/completion_queue.pyx.pxi", line 147, in grpc._cython.cygrpc.PollerCompletionQueue._handle_events
BlockingIOError: [Errno 11] Resource temporarily unavailable

Anything else we should know about your project / environment?

• The API we are using is stream-stream
• We use interceptors to insert headers into the requests

[lidizheng]

I'm able to reproduce this error. I found the BlockingIOError log appears in Python 3.9 for unary-unary but not 3.7, but both passes. But multi-threading with multiple loops doesn't work stream-stream.

CC @dpcollins-google

[stupidchen]

I'm able to reproduce this error too. Any update?

[wti806]

Seeing this error on the gRPC client when invoking RPCs in multiple threads.