gen_cluster nanny tests intermittently fail on Python 2

[mrocklin]

This fails in Python 2, but not Python 3:

from distributed import Nanny
from distributed.utils_test import gen_cluster
@gen_cluster(client=True, Worker=Nanny)
def test_startup(c, s, a, b):
    pass
for i in range(100):
    test_startup()

[mrocklin]

Git bisect traces this back to 647238e cc @pitrou . I don't yet know how critical this is though. We have lived with this for a long time.

[mrocklin]

I tracked this down to logging code within the worker. Just calling logger.info("Hello") when starting up is enough to sometimes lock down the worker process. Perhaps forking after starting the logger does something unpleasant?

[mrocklin]

From celery/celery#496 which maybe discusses the same problem:

python logging, uses RLock() locks, and those are broken after a fork.
Looking the long history of this python bug, the conclusion that this bug is not going to be solved in python.

[mrocklin]

OK, it looks like they resolved it by recreating the lock at the onset of every new process. This seems to resolve things for me as well.