Logging an object with incompatible type kills Dask worker

[uellue]

Logging an event with an object type that is not handled by msgpack kills a Dask worker. I am not sure if log_event() should work with such types or if it is the user's responsibility to only use compatible types?

import distributed as dd
import numpy as np

def trigger():
    worker = dd.get_worker()
    worker.log_event("ded", np.int64(23))

if __name__ == '__main__':
    client = dd.Client()

    print(client.scheduler_info())  # has workers

    client.run(trigger)

    print(client.scheduler_info())  # No more workers

Resulting error message on the terminal:

2023-01-12 12:42:28,093 - distributed.batched - ERROR - Error in batched write
Traceback (most recent call last):
  File "c:\users\weber\documents\src\distributed\distributed\batched.py", line 115, in _background_send
    nbytes = yield coro
  File "C:\Users\weber\miniconda3\envs\libertemcleannotebooks\lib\site-packages\tornado\gen.py", line 769, in run
    value = future.result()
  File "c:\users\weber\documents\src\distributed\distributed\comm\tcp.py", line 271, in write
    frames = await to_frames(
  File "c:\users\weber\documents\src\distributed\distributed\comm\utils.py", line 72, in to_frames
    return _to_frames()
  File "c:\users\weber\documents\src\distributed\distributed\comm\utils.py", line 55, in _to_frames
    return list(protocol.dumps(msg, **kwargs))
  File "c:\users\weber\documents\src\distributed\distributed\protocol\core.py", line 109, in dumps
    frames[0] = msgpack.dumps(msg, default=_encode_default, use_bin_type=True)
  File "C:\Users\weber\miniconda3\envs\libertemcleannotebooks\lib\site-packages\msgpack\__init__.py", line 38, in packb
    return Packer(**kwargs).pack(o)
  File "msgpack\_packer.pyx", line 294, in msgpack._cmsgpack.Packer.pack
  File "msgpack\_packer.pyx", line 300, in msgpack._cmsgpack.Packer.pack
  File "msgpack\_packer.pyx", line 297, in msgpack._cmsgpack.Packer.pack
  File "msgpack\_packer.pyx", line 264, in msgpack._cmsgpack.Packer._pack
  File "msgpack\_packer.pyx", line 231, in msgpack._cmsgpack.Packer._pack
  File "msgpack\_packer.pyx", line 291, in msgpack._cmsgpack.Packer._pack
TypeError: can not serialize 'numpy.int64' object

Anything else we need to know?:

Not sure if this is a bug or if the allowed types in log messages should be specified!

Environment:

• Dask version: dask==2022.12.1, distributed current main branch and 2022.12.1
• Python version: 3.9.15
• Operating System: Win 11
• Install method (conda, pip, source): pip, source

[fjetter]

Thanks for reporting this @uellue !

It is a bug that the worker/server is dying. This should clearly not happen. However, that the msg has to be msgpack serializable is document, at least in the the Client.log_event. In #7472 I added doc strings to the respective Worker/Nanny methods and implemented a check that raises a hopefully better error message w/out killing the worker.

[uellue]

Thx for the quick reply and for taking care of it!