Add performance_report context manager for static report generation

[mrocklin]

This generates a static HTML file with many of the same plots as the dashboard.

Example

from dask.distributed import Client, performance_report
client = Client()

import dask.array as da
x = da.random.random((30000, 30000), chunks=(1000, 1000))

with performance_report():
    x = x.persist()
    (x + x.T).sum().compute()

Result

https://matthewrocklin.com/raw-host/dask-report.html

[beckernick]

@mrocklin , when I use this after installing dask.distributed with git+https://github.com/dask/distributed.git (in an environment that already has dask 2.8.1) I get a recursion error from an infinite loop of the following:

  File "/opt/conda/envs/rapids/lib/python3.6/site-packages/dask/sizeof.py", line 36, in sizeof_python_dict
    return getsizeof(d) + sum(map(sizeof, d.keys())) + sum(map(sizeof, d.values()))
  File "/opt/conda/envs/rapids/lib/python3.6/site-packages/dask/utils.py", line 505, in __call__
    return meth(arg, *args, **kwargs)

Larger traceback:

Traceback (most recent call last):
  File "/opt/conda/envs/rapids/lib/python3.6/site-packages/tornado/ioloop.py", line 743, in _run_callback
    ret = callback()
  File "/opt/conda/envs/rapids/lib/python3.6/site-packages/tornado/tcpserver.py", line 327, in <lambda>
    gen.convert_yielded(future), lambda f: f.result()
  File "/opt/conda/envs/rapids/lib/python3.6/site-packages/distributed/comm/tcp.py", line 442, in _handle_stream
    await self.comm_handler(comm)
  File "/opt/conda/envs/rapids/lib/python3.6/site-packages/distributed/core.py", line 415, in handle_comm
    await comm.write(result, serializers=serializers)
  File "/opt/conda/envs/rapids/lib/python3.6/site-packages/distributed/comm/tcp.py", line 226, in write
    context={"sender": self._local_addr, "recipient": self._peer_addr},
  File "/opt/conda/envs/rapids/lib/python3.6/site-packages/distributed/comm/utils.py", line 40, in to_frames
    if FRAME_OFFLOAD_THRESHOLD and sizeof(msg) > FRAME_OFFLOAD_THRESHOLD:
  File "/opt/conda/envs/rapids/lib/python3.6/site-packages/dask/utils.py", line 505, in __call__
    return meth(arg, *args, **kwargs)
  File "/opt/conda/envs/rapids/lib/python3.6/site-packages/dask/sizeof.py", line 36, in sizeof_python_dict
    return getsizeof(d) + sum(map(sizeof, d.keys())) + sum(map(sizeof, d.values()))
  File "/opt/conda/envs/rapids/lib/python3.6/site-packages/dask/utils.py", line 505, in __call__
    return meth(arg, *args, **kwargs)
  File "/opt/conda/envs/rapids/lib/python3.6/site-packages/dask/sizeof.py", line 36, in sizeof_python_dict
    return getsizeof(d) + sum(map(sizeof, d.keys())) + sum(map(sizeof, d.values()))
  File "/opt/conda/envs/rapids/lib/python3.6/site-packages/dask/utils.py", line 505, in __call__
    return meth(arg, *args, **kwargs)

...

RecursionError: maximum recursion depth exceeded
tornado.application - ERROR - Exception in callback functools.partial(<function TCPServer._handle_connection.<locals>.<lambda> at 0x7f8f7b09b8c8>, <Task finished coro=<BaseTCPListener._handle_stream() done, defined at /opt/conda/envs/rapids/lib/python3.6/site-packages/distributed/comm/tcp.py:433> exception=RecursionError('maximum recursion depth exceeded',)>)

Is there perhaps an additional step I may be missing?

[beckernick]

Problem appears to be on my end 👍

[mrocklin]

Oh great. I'm always happy to find that things are fixed once I get to them :)

…

On Mon, Dec 2, 2019 at 2:51 PM Nick Becker ***@***.***> wrote: Problem appears to be on my end 👍 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3282?email_source=notifications&email_token=AACKZTD6NPDKOQVOQN7YDADQWWGQBA5CNFSM4JSW4SP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFWU2UQ#issuecomment-560811346>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKZTFDOE6ELT3YOOHVXNTQWWGQBANCNFSM4JSW4SPQ> .

[KrishanBhasin]

@beckernick Could you share how you fixed this? I've started running into this problem too