Add default logs method to Spec Cluster

[mrocklin]

Builds off of #2875

cc @jacobtomlinson

[mrocklin]

cc @jacobtomlinson you may want to briefly skim through this, just to get a sense of how the combination async/sync API stuff tends to work within dask-distributed today.

[mrocklin]

With #2890 we get this behavior

https://nbviewer.jupyter.org/urls/gist.githubusercontent.com/mrocklin/b350ed9462963f0df784544fd2e07fd9/raw/e8ba500f872b96961cc2f5747218718da5ce4893/cluster-logs.ipynb

[mrocklin]

Also @jcrist are you comfortable with this API for dask-yarn?

[jcrist]

Overall this seems fine. What about getting the scheduler logs, for implementations with a remote scheduler?

[jcrist]

For implementations that grab logs from the backing system (e.g. kubernetes) this option may not make any sense - the only logs will be stdout/stderr of the process tree (IIRC the nanny and the worker share stdout/stderr). I'd be tempted to remove this option entirely.

[mrocklin]

Done

[mrocklin]

Overall this seems fine. What about getting the scheduler logs, for implementations with a remote scheduler?

Currently the scheduler is included in these logs. It's the first entry with the name "Scheduler" (see notebook link above for an example)"

[jcrist]

Perhaps there should be a scheduler=True kwarg (default True) to indicate whether to get the scheduler logs? So calling logs() will get everything, but you can avoid getting the scheduler logs if desired (say you wanted to ask for only a specific worker).

[mrocklin]

Perhaps there should be a scheduler=True kwarg (default True) to indicate whether to get the scheduler logs? So calling logs() will get everything, but you can avoid getting the scheduler logs if desired (say you wanted to ask for only a specific worker).

Sure. Done. The tests that were just added might be a nice way to check this

[mrocklin]

Thanks for the engagement Jim!