Logger stalls

[mrocklin]

I'm getting odd errors in Python 3.6 when I create and destroy clusters in rapid succession.

  File "/home/mrocklin/workspace/distributed/distributed/worker.py", line 388, in _start
	logger.info('start worker %d', id(self))
  File "/home/mrocklin/Software/anaconda/lib/python3.6/logging/__init__.py", line 1301, in info
	self._log(INFO, msg, args, **kwargs)
  File "/home/mrocklin/Software/anaconda/lib/python3.6/logging/__init__.py", line 1437, in _log
	self.handle(record)
  File "/home/mrocklin/Software/anaconda/lib/python3.6/logging/__init__.py", line 1447, in handle
	self.callHandlers(record)
  File "/home/mrocklin/Software/anaconda/lib/python3.6/logging/__init__.py", line 1509, in callHandlers
	hdlr.handle(record)
  File "/home/mrocklin/Software/anaconda/lib/python3.6/logging/__init__.py", line 856, in handle
	self.acquire()
  File "/home/mrocklin/Software/anaconda/lib/python3.6/logging/__init__.py", line 807, in acquire
	self.lock.acquire()

I've tried calling the reset_logger_locks as in #1491 but then get the following:

  File "/home/mrocklin/workspace/distributed/distributed/nanny.py", line 519, in run
	yield worker._start(*worker_start_args)
  File "/home/mrocklin/workspace/tornado/tornado/gen.py", line 307, in wrapper
	yielded = next(result)
  File "/home/mrocklin/workspace/distributed/distributed/worker.py", line 379, in _start
	logger.info('      Start worker at: %26s', self.address)
  File "/home/mrocklin/Software/anaconda/lib/python3.6/logging/__init__.py", line 1301, in info
	self._log(INFO, msg, args, **kwargs)
  File "/home/mrocklin/Software/anaconda/lib/python3.6/logging/__init__.py", line 1437, in _log
	self.handle(record)
  File "/home/mrocklin/Software/anaconda/lib/python3.6/logging/__init__.py", line 1447, in handle
	self.callHandlers(record)
  File "/home/mrocklin/Software/anaconda/lib/python3.6/logging/__init__.py", line 1509, in callHandlers
	hdlr.handle(record)
  File "/home/mrocklin/Software/anaconda/lib/python3.6/logging/__init__.py", line 858, in handle
	self.emit(record)
  File "/home/mrocklin/Software/anaconda/lib/python3.6/logging/__init__.py", line 990, in emit
	stream.write(self.terminator)

[mrocklin]

from dask.distributed import LocalCluster, Client
import time
for i in range(10000):
    start = time.time()
    with Client(n_workers=2, silence_logs=False) as c:
        pass
    stop = time.time()
    print(i, stop - start)
    print('-' * 120)

[mrocklin]

cc @pitrou this may interest you from a core Python perspective

[pitrou]

Did you truncate the error logs above? I don't see an error message.

[mrocklin]

Ah, sorry, this is a call stack. This thread is stuck there. I have a thread periodically dumping out the call stacks of all of the threads. This thread is consistently at this line without any logs being emitted.