Uncaught exception in ZMQStream callback when running your example notebooks using latest or nightly docker image

[deepcoder]

Click to expand!

Issue Type

Bug

Have you reproduced the bug with TF nightly?

Yes

Source

binary

Tensorflow Version

v2.12.0-rc1-12-g0db597d0d75 2.12.0

Custom Code

No

OS Platform and Distribution

Linux gpu02 6.2.11-2-pve #1 SMP PREEMPT_DYNAMIC PVE 6.2.11-2 (2023-05-10T09:13Z) x86_64 x86_64 x86_64 GNU/Linux

Mobile device

No response

Python version

python3.8

Bazel version

No response

GCC/Compiler version

No response

CUDA/cuDNN version

No response

GPU model and memory

No response

Current Behaviour?

Occurs when running any of your example notebooks:

[E 22:36:50.295 NotebookApp] Uncaught exception in ZMQStream callback
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/zmq/eventloop/zmqstream.py", line 584, in _run_callback
        f = callback(*args, **kwargs)
      File "/usr/local/lib/python3.8/dist-packages/zmq/eventloop/zmqstream.py", line 308, in stream_callback
        return callback(self, msg)
      File "/usr/local/lib/python3.8/dist-packages/notebook/services/kernels/handlers.py", line 572, in _on_zmq_reply
        super()._on_zmq_reply(stream, msg)
      File "/usr/local/lib/python3.8/dist-packages/notebook/base/zmqhandlers.py", line 256, in _on_zmq_reply
        self.write_message(msg, binary=isinstance(msg, bytes))
      File "/usr/local/lib/python3.8/dist-packages/tornado/websocket.py", line 339, in write_message
        return self.ws_connection.write_message(message, binary=binary)
      File "/usr/local/lib/python3.8/dist-packages/tornado/websocket.py", line 1086, in write_message
        fut = self._write_frame(True, opcode, message, flags=flags)
      File "/usr/local/lib/python3.8/dist-packages/tornado/websocket.py", line 1061, in _write_frame
        return self.stream.write(frame)
      File "/usr/local/lib/python3.8/dist-packages/tornado/iostream.py", line 546, in write
        self._handle_write()
      File "/usr/local/lib/python3.8/dist-packages/tornado/iostream.py", line 976, in _handle_write
        self._write_buffer.advance(num_bytes)
      File "/usr/local/lib/python3.8/dist-packages/tornado/iostream.py", line 182, in advance
        assert 0 < size <= self._size
    AssertionError
[E 22:36:50.297 NotebookApp] Uncaught exception in zmqstream callback
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/zmq/eventloop/zmqstream.py", line 634, in _handle_events
        self._handle_recv()
      File "/usr/local/lib/python3.8/dist-packages/zmq/eventloop/zmqstream.py", line 663, in _handle_recv
        self._run_callback(callback, msg)
      File "/usr/local/lib/python3.8/dist-packages/zmq/eventloop/zmqstream.py", line 584, in _run_callback
        f = callback(*args, **kwargs)
      File "/usr/local/lib/python3.8/dist-packages/zmq/eventloop/zmqstream.py", line 308, in stream_callback
        return callback(self, msg)
      File "/usr/local/lib/python3.8/dist-packages/notebook/services/kernels/handlers.py", line 572, in _on_zmq_reply
        super()._on_zmq_reply(stream, msg)
      File "/usr/local/lib/python3.8/dist-packages/notebook/base/zmqhandlers.py", line 256, in _on_zmq_reply
        self.write_message(msg, binary=isinstance(msg, bytes))
      File "/usr/local/lib/python3.8/dist-packages/tornado/websocket.py", line 339, in write_message
        return self.ws_connection.write_message(message, binary=binary)
      File "/usr/local/lib/python3.8/dist-packages/tornado/websocket.py", line 1086, in write_message
        fut = self._write_frame(True, opcode, message, flags=flags)
      File "/usr/local/lib/python3.8/dist-packages/tornado/websocket.py", line 1061, in _write_frame
        return self.stream.write(frame)
      File "/usr/local/lib/python3.8/dist-packages/tornado/iostream.py", line 546, in write
        self._handle_write()
      File "/usr/local/lib/python3.8/dist-packages/tornado/iostream.py", line 976, in _handle_write
        self._write_buffer.advance(num_bytes)
      File "/usr/local/lib/python3.8/dist-packages/tornado/iostream.py", line 182, in advance
        assert 0 < size <= self._size
    AssertionError
Exception in callback BaseAsyncIOLoop._handle_events(33, 1)
handle: <Handle BaseAsyncIOLoop._handle_events(33, 1)>
Traceback (most recent call last):
  File "/usr/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/local/lib/python3.8/dist-packages/tornado/platform/asyncio.py", line 206, in _handle_events
    handler_func(fileobj, events)
  File "/usr/local/lib/python3.8/dist-packages/zmq/eventloop/zmqstream.py", line 634, in _handle_events
    self._handle_recv()
  File "/usr/local/lib/python3.8/dist-packages/zmq/eventloop/zmqstream.py", line 663, in _handle_recv
    self._run_callback(callback, msg)
  File "/usr/local/lib/python3.8/dist-packages/zmq/eventloop/zmqstream.py", line 584, in _run_callback
    f = callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/zmq/eventloop/zmqstream.py", line 308, in stream_callback
    return callback(self, msg)
  File "/usr/local/lib/python3.8/dist-packages/notebook/services/kernels/handlers.py", line 572, in _on_zmq_reply
    super()._on_zmq_reply(stream, msg)
  File "/usr/local/lib/python3.8/dist-packages/notebook/base/zmqhandlers.py", line 256, in _on_zmq_reply
    self.write_message(msg, binary=isinstance(msg, bytes))
  File "/usr/local/lib/python3.8/dist-packages/tornado/websocket.py", line 339, in write_message
    return self.ws_connection.write_message(message, binary=binary)
  File "/usr/local/lib/python3.8/dist-packages/tornado/websocket.py", line 1086, in write_message
    fut = self._write_frame(True, opcode, message, flags=flags)
  File "/usr/local/lib/python3.8/dist-packages/tornado/websocket.py", line 1061, in _write_frame
    return self.stream.write(frame)
  File "/usr/local/lib/python3.8/dist-packages/tornado/iostream.py", line 546, in write
    self._handle_write()
  File "/usr/local/lib/python3.8/dist-packages/tornado/iostream.py", line 976, in _handle_write
    self._write_buffer.advance(num_bytes)
  File "/usr/local/lib/python3.8/dist-packages/tornado/iostream.py", line 182, in advance
    assert 0 < size <= self._size
AssertionError

Standalone code to reproduce the issue

Run any of your Jupyter example in your docker image.

Relevant log output

No response

[sushreebarsa]

@deepcoder
In order to expedite the trouble-shooting process, please provide the example code snippet to reproduce the issue reported here. Thank you!

[deepcoder]

As I stated in my opening post, running any of the example programs that you provide in the Tensorflow CUDA docker image caused the failures referencing the ZMQStream message. In terms of helping you further with this, unfortunately for that, I have moved on to another docker image I found on Docker Hub that provides recent Tensorflow 2 version, cuda support and so far is working fine for me.

[sachinprasadhs]

Hi,

Thanks for reporting the issue.

Feel free to close the issue if the issue is resolved in the latest docker version.

For our latest docker images, please follow https://hub.docker.com/r/tensorflow/tensorflow/tags

[github-actions]

This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you.

[github-actions]

This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further.

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No