Add nanny logs

[TomAugspurger]

I had a need to collate logs from the cluster, and would like to include logs from the Nanny. This adds a method to get logs from the nanny as well.

This isn't quite ready.

1. The keys in the returned dict are the address of the workers, not the nannies. Do we have a convention here? I assume we want the address of the nannies.
2. I may be adding the handler to the distribted.nanny.logger multiple times, resulting in duplicate logs. Need to verify what's going on.

[mrocklin]

Do we have a convention here?

Other functions like run and broadcast take a nanny= keyword argument. That might be appropriate here as well.

[mrocklin]

So client.get_worker_logs(nanny=True)

In fact, I suspect that on the scheduler you could just pass through the nanny=nanny keyword to Scheduler.broadcast and everything would work out.

[TomAugspurger]

Thanks @mrocklin that works out quite nicely.

I updated the docstring for Client.run to note that workers should still be the worker addresses when you specify nanny=True, not the nanny address.

[TomAugspurger]

As an aside, having timestamps in logs is quite helpful for collation across the cluster (assuming the clocks synced well enough). What's our backwards compatibility story for configuration things like that?

[TomAugspurger]

Actually, this may not be ready. I thought I had sorted out the duplicate handler, but apparently not.

[mrocklin]

Other than the possible duplicate handler issue the implementation here looks great to me.

[mrocklin]

As an aside, having timestamps in logs is quite helpful for collation across the cluster (assuming the clocks synced well enough). What's our backwards compatibility story for configuration things like that?

I don't have any particular thoughts here.

You may want to take a look here though:

• Cannot maintain effective debug-level on distributed (parent) logger with local scheduler/cluster #2660
• Old/simple/default logging config is opaque and hard to externally-configure #2659

[mrocklin]

Actually, this may not be ready. I thought I had sorted out the duplicate handler, but apparently not.

Can you explain this a bit more? This seems to be the same behavior as before. Is this correct?

[TomAugspurger]

I'm not too sure what's going on.

In [1]: from distributed import Client
   ...: import logging
   ...:
   ...:
   ...: client = Client(n_workers=2, threads_per_worker=1)
   ...:
   ...: logger = logging.getLogger('distributed.nanny')
   ...: logger.handlers

Out[1]:
[<StreamHandler <stderr> (WARNING)>,
 <DequeHandler (NOTSET)>,
 <DequeHandler (NOTSET)>]

So if I get the distributed.nanny logger from my main processes, I see one handler per Nanny.

Oh... but I just realized this is the same behavior as distributed.worker with processes=False.

   ...: client = Client(n_workers=2, threads_per_worker=1, processes=False)
   ...:
   ...: logger = logging.getLogger('distributed.worker')
   ...: logger.handlers
Out[2]: [<DequeHandler (NOTSET)>, <DequeHandler (NOTSET)>]

so perhaps we're OK?

[mrocklin]

Looks good to me. Merging.