master hanging on "write" sys call

[ninjaahhh]

when running the cluster (testnet201.bootstrap), after mining for a certain period of time, the jsonrpc server became unresponsive for all requests. strace shows following results

$ ps aux | grep py
...
root     15480  0.3  5.0 2049216 1605800 ?     Sl   Nov20   8:17 pypy3 master.py --cluster_config=/code/cluster_config.json
root     15481  0.8  1.5 788468 484300 ?       Sl   Nov20  22:27 pypy3 slave.py --cluster_config=/code/cluster_config.json --node_id=S0
...
$ sudo strace -p 15480 -s 10000
strace: Process 15480 attached
write(2, "I1121 00:26:53.642705 jsonrpc.py:408] {\"jsonrpc\": \"2.0\", \"method\": \"getWork\", \"params\": [\"0x1\"], \"id\": 433}\n", 108
<ctrl+c>
$ sudo strace -p 15481 -s 10000
strace: Process 15481 attached
epoll_wait(3,
<ctrl+c>

so slave server looks fine, but master is hanging on writing the jsonrpc log to strerr (file descriptor 2).

it also points to line 408

pyquarkchain/quarkchain/cluster/jsonrpc.py

Lines 406 to 409 in 5eca86b

	async def __handle(self, request):
	request = await request.text()
	Logger.info(request)

where in turn it calls

pyquarkchain/quarkchain/utils.py

Lines 222 to 225 in 5eca86b

	@classmethod
	def info(cls, msg):
	cls.check_logger_set()
	cls._qkc_logger.info(msg)

need to find out why the write call is blocked.

[ninjaahhh]

PROGRESS:

doing some searches online

1. https://stackoverflow.com/questions/37385755/freeze-when-writing-to-stderr
2. Process in container stuck when writing log to stderr moby/moby#31540 (comment)

and sudo tail -f /proc/15480/fd/2 helps unblock.

[qcdll]

https://github.com/QuarkChain/pyquarkchain/blob/master/quarkchain/cluster/cluster.py#L37
stderr also goes to stdout, and they should all be handled by cluster.py print_output and printed to stdout... as we discussed one suspect may be the docker supervisor failing to tail it