Assertion failed: _input_stopped

[jean-airoldie]

Issue description

I get an assertion failure caused by: https://github.com/zeromq/libzmq/blob/latest_release/src/stream_engine.cpp#L467 while using CLIENT and SERVER socket. Specifically the assertion occurs when the Server socket is streaming to a slow CLIENT that also has a very low recv high water mark and heartbeating turned on.

Environment

• libzmq version 4.3.2
• OS: centos7

Minimal test code / Steps to reproduce the issue

• Create a CLIENT and SERVER socket and connect them to each other via TCP.
• Set the CLIENT's heartbeat interval to 1 second and its recv high water mark to 1.
• Send an empty message from the CLIENT to the SERVER and retrieve the routing
  id of the CLIENT.
• Move the SERVER to a new thread and send empty messages to the CLIENT using the routing id in a loop.
• Now create a loop for the CLIENT where it receives a message then waits 3 seconds after receiving each message to simulate a slow client.

What's the actual result? (include assertion message & call stack if applicable)

Note that I vendor the libzmq source in a rust package, which explains the naming.

Assertion

Assertion failed: _input_stopped (/home/maxence/.cargo/registry/src/github.com-1ecc6299db9ec823/zeromq-src-0.1.7+4.3.2/vendor/src/stream_engine.cpp:467)

Backtrace

#0  zmq::stream_engine_t::restart_input (this=0x7fffe8000e30) at /home/maxence/.cargo/registry/src/github.com-1ecc6299db9ec823/zeromq-src-0.1.7+4.3.2/vendor/src/stream_engine.cpp:467
#1  0x000055555579d009 in zmq::session_base_t::write_activated (this=0x555555b07000, pipe_=0x555555b0f660)
    at /home/maxence/.cargo/registry/src/github.com-1ecc6299db9ec823/zeromq-src-0.1.7+4.3.2/vendor/src/session_base.cpp:313
#2  0x00005555557686d2 in zmq::pipe_t::process_activate_write (this=0x555555b0f660, msgs_read_=5) at /home/maxence/.cargo/registry/src/github.com-1ecc6299db9ec823/zeromq-src-0.1.7+4.3.2/vendor/src/pipe.cpp:287
#3  0x0000555555760233 in zmq::object_t::process_command (this=0x555555b0f660, cmd_=...) at /home/maxence/.cargo/registry/src/github.com-1ecc6299db9ec823/zeromq-src-0.1.7+4.3.2/vendor/src/object.cpp:79
#4  0x000055555575c81c in zmq::io_thread_t::in_event (this=0x555555b03350) at /home/maxence/.cargo/registry/src/github.com-1ecc6299db9ec823/zeromq-src-0.1.7+4.3.2/vendor/src/io_thread.cpp:91
#5  0x000055555575b6d5 in zmq::epoll_t::loop (this=0x555555b03890) at /home/maxence/.cargo/registry/src/github.com-1ecc6299db9ec823/zeromq-src-0.1.7+4.3.2/vendor/src/epoll.cpp:206
#6  0x000055555576ac49 in zmq::worker_poller_base_t::worker_routine (arg_=0x555555b03890) at /home/maxence/.cargo/registry/src/github.com-1ecc6299db9ec823/zeromq-src-0.1.7+4.3.2/vendor/src/poller_base.cpp:139
#7  0x000055555577c1f9 in thread_routine (arg_=0x555555b038e8) at /home/maxence/.cargo/registry/src/github.com-1ecc6299db9ec823/zeromq-src-0.1.7+4.3.2/vendor/src/thread.cpp:225
#8  0x00007ffff74b3dd5 in start_thread (arg=0x7ffff63c5700) at pthread_create.c:307
#9  0x00007ffff6fc702d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

What's the expected result?

The code should run forever without an assertion error.

[brettviren]

I tried but failed to reproduce this problem with libzmq master.

https://gist.github.com/brettviren/dde42c6f1c8078c07e0ce595dd619919

@jean-airoldie can you please check this code to see if it correctly reflects your steps to reproduce?

[jean-airoldie]

I'm no longer using ZeroMq and I got rid of the reproduction code I'm afraid. I'm fine with closing this issue.

[brettviren]

My test has been happily printing "client" every three seconds for almost a day now so it seems there's no problem.

@bluca can we close this issue?

[lukaszsamson]

I'm hitting the same assertion in on DEALER connected to ROUTER over tcp (v4.3.2, macos 10.15.4)

* thread #34, name = 'ZMQbg/IO/0', stop reason = signal SIGABRT
  * frame #0: 0x00007fff7331f33a libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff733dbe60 libsystem_pthread.dylib`pthread_kill + 430
    frame #2: 0x00007fff732a6808 libsystem_c.dylib`abort + 120
    frame #3: 0x000000001d8106e9 libzmq.5.dylib`zmq::zmq_abort(char const*) + 9
    frame #4: 0x000000001d840f0b libzmq.5.dylib`zmq::stream_engine_t::restart_input() + 651
    frame #5: 0x000000001d82f6b8 libzmq.5.dylib`non-virtual thunk to zmq::session_base_t::write_activated(zmq::pipe_t*) + 136
    frame #6: 0x000000001d811afc libzmq.5.dylib`zmq::io_thread_t::in_event() + 60
    frame #7: 0x000000001d815658 libzmq.5.dylib`zmq::kqueue_t::loop() + 408
    frame #8: 0x000000001d845d9d libzmq.5.dylib`thread_routine(void*) + 61
    frame #9: 0x00007fff733dc109 libsystem_pthread.dylib`_pthread_start + 148
    frame #10: 0x00007fff733d7b8b libsystem_pthread.dylib`thread_start + 15