Assertion failed: (src/mailbox.cpp:99)

[paololucente]

Issue description

I'm from a project called pmacct ( https://github.com/pmacct/pmacct ) which can optionally make use of ZeroMQ for passing messages internally. I got three reports in the last three weeks of users compiling pmacct against ZeroMQ 4.2.5 and getting a failed assertion message back:

shell> nfacctd -f ./nfacctd.detailed.conf
Assertion failed: ok (src/mailbox.cpp:99)
Aborted (core dumped)

Only an hint: one of the users made some testing and apparently all works well with up to gcc6. But compiling with gcc7 or gcc8 shows the above issue.

Environment

• libzmq version: 4.2.5
• OS: Debian 10, Ubuntu 18.04

Minimal test code / Steps to reproduce the issue

1. apt-get install libpcap-dev pkg-config libtool autoconf automake bash
2. compile and install ZeroMQ 4.2.5
3. git clone https://github.com/pmacct/pmacct.git
4. cd pmacct ; ./autogen ; ./configure --enable-zmq; make ; make install
5. Edit a nfacctd.conf file, put inside the following line:
   plugin_pipe_zmq: true
6. Start pmacct: nfacctd -f /path/to/nfacctd.conf
7. It should pretty much immediately show the failed assertion message

What's the actual result? (include assertion message & call stack if applicable)

gdb /usr/local/sbin/nfacctd core
GNU gdb (Ubuntu 8.1-0ubuntu3) 8.1.0.20180409-git
Copyright (C) 2018 Free Software Foundation, Inc.
< .. cut .. >
Reading symbols from /usr/local/sbin/nfacctd...done.
[New LWP 6120]
[New LWP 6118]
[New LWP 6117]
[New LWP 6119]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `nfacctd: Core Process [detailed] '.
Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51    ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
[Current thread is 1 (Thread 0x7f209000d700 (LWP 6120))]
(gdb) where
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007f2092961801 in __GI_abort () at abort.c:79
#2  0x00007f20936b1529 in ?? () from /usr/lib/x86_64-linux-gnu/libzmq.so.5
#3  0x00007f20936b60b4 in ?? () from /usr/lib/x86_64-linux-gnu/libzmq.so.5
4  0x00007f20936da472 in ?? () from /usr/lib/x86_64-linux-gnu/libzmq.so.5
#5  0x00007f20936daf36 in ?? () from /usr/lib/x86_64-linux-gnu/libzmq.so.5
#6  0x00007f20936fb5e9 in ?? () from /usr/lib/x86_64-linux-gnu/libzmq.so.5
#7  0x00007f20936fb669 in zmq_recv () from /usr/lib/x86_64-linux-gnu/libzmq.so.5
#8  0x000055ec462a574c in p_zmq_recv_str(sock=sock@entry=0x55ec4657fec8 <channels_list+7432>) at zmq_common.c:460
#9  0x000055ec462a58bb in p_zmq_zap_handler (zh=0x55ec4657fec0 <channels_list+7424>) at zmq_common.c:518
#10 0x00007f20936ef554 in ?? () from /usr/lib/x86_64-linux-gnu/libzmq.so.5
#11 0x00007f2092d196db in start_thread (arg=0x7f209000d700) at pthread_create.c:463
#12 0x00007f2092a4288f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb)

[bluca]

are you using a socket from multiple threads?

[paololucente]

Yes and no. zmq_socket() is called in one thread, followed-up by a call zmq_threadstart() passing the socket as argument; then only that specific thread uses the created socket. Effectively the issue happens on trying to read for the first time from the socket in the thread. I will correct the issue, test all and report back. Thanks, Paolo

[bluca]

ok, I'll close this in the meanwhile, reopen it if when the socket is opened and used from a single thread only it still happens

[paololucente]

Hi @bluca ,

Unfortunately the change did not help. Just FYI this is the patch i made ( pmacct/pmacct@b1b61ce ) but it still bails out on the same call as before, that is, the first use of the socket inside p_zmq_zap_handler().

Is it possible it does not like that the socket is part of a structure passed as argument of zmq_threadstart() - even if never used/initialised/touched outside the thread itself?

Paolo

[pramodk]

We were having this issue on multiple clusters and debugging this for quite some time. As @paololucente mentioned:

Only an hint: one of the users made some testing and apparently all works well with up to gcc6. But compiling with gcc7 or gcc8 shows the above issue.

Everything works fine with GCC v6.1.0 but not with newer GCC 7.3.0 or GCC v8.3.0. So certainly there is an issue.

Thank you very much @paololucente for the hint!

cc: @jorblancoa @matz-e

[pramodk]

Just to add : we tested different compilers:

• Intel 17
• Intel 18
• Intel 19
• GCC 6
• GCC 7
• GCC 8

From above list, GCC 7 and GCC 8 doesn't work!

By the way, we don't have any our software component. We are just using ipython/ipyparallel.

[pramodk]

@bluca : I think it would be better to reopen this issue? Also, let me know if/how can help here, for example running tests?

[aolney]

I'm seeing Assertion failed: ok (src/mailbox.cpp:99) in a Jupyterlab notebook using the IRKernel, running irt.fa with the psych package:

• Ubuntu 20.04
• gcc version 9.3.0 (Ubuntu 9.3.0-10ubuntu2)

I compiled libzmq 4.3.2 from source using gcc 9, hoping this might resolve the problem, but it did not.

[pramodk]

@aolney : just fyi : as mentioned in the previous comments, only working configuration we have found is to use old compiler (or using Intel compiler but this is more from our scientific computing use case!)

[aolney]

@pramodk Many thanks for the helpful comment. I was hoping using gcc 9 would solve. I just recompiled libzmq 4.3.2 using icc 19.1.2.254, and the error remains.

ldconfig lists my compiled library before the distribution's version, so I'm not sure why using icc 19 worked for you and not for me. Any suggestions appreciated.

[pramodk]

ldconfig lists my compiled library before the distribution's version, so I'm not sure why using icc 19 worked for you and not for me.

Could you elaborate this bit? On our system we have following (you can see libzmq.so.5 is compiled with Intel 19.0.4):

$ ldd /my-gpfs-path/libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/py-zmq-19.0.0-4ltw4hvhfh/lib/python3.7/site-packages/zmq/backend/cython/message.cpython-37m-x86_64-linux-gnu.so
    linux-vdso.so.1 =>  (0x00007fffedb02000)
    libzmq.so.5 => /my-gpfs-path/external-libraries/2020-02-01/linux-rhel7-x86_64/intel-19.0.4/zeromq-4.3.2-msncwceskl/lib/libzmq.so.5 (0x00007fffed3f0000)
    librt.so.1 => /lib64/librt.so.1 (0x00007fffed1e8000)
    libpython3.7m.so.1.0 => /my-gpfs-path/external-libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/python-3.7.4-tfxecymrkn/lib/libpython3.7m.so.1.0 (0x00007fffeccb0000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fffeca94000)
    libc.so.6 => /lib64/libc.so.6 (0x00007fffec6c7000)
    libsodium.so.23 => /my-gpfs-path/external-libraries/2020-02-01/linux-rhel7-x86_64/intel-19.0.4/libsodium-1.0.17-dlfjex7os4/lib/libsodium.so.23 (0x00007fffec40d000)
    libimf.so => /my-gpfs-path/compilers/2020-02-01/linux-rhel7-x86_64/gcc-4.8.5/intel-19.0.4-3ahwr77o2x/compilers_and_libraries_2019.4.243/linux/compiler/lib/intel64/libimf.so (0x00007fffebe6d000)
    libirng.so => /my-gpfs-path/compilers/2020-02-01/linux-rhel7-x86_64/gcc-4.8.5/intel-19.0.4-3ahwr77o2x/compilers_and_libraries_2019.4.243/linux/compiler/lib/intel64/libirng.so (0x00007fffebafb000)
    libm.so.6 => /lib64/libm.so.6 (0x00007fffeb7f9000)
    libcilkrts.so.5 => /my-gpfs-path/compilers/2020-02-01/linux-rhel7-x86_64/gcc-4.8.5/intel-19.0.4-3ahwr77o2x/compilers_and_libraries_2019.4.243/linux/compiler/lib/intel64/libcilkrts.so.5 (0x00007fffeb5bc000)
    libstdc++.so.6 => /my-gpfs-path/compilers/2020-02-01/linux-rhel7-x86_64/gcc-4.8.5/gcc-8.3.0-r24zbv3waz/lib64/libstdc++.so.6 (0x00007fffeb238000)
    libintlc.so.5 => /my-gpfs-path/compilers/2020-02-01/linux-rhel7-x86_64/gcc-4.8.5/intel-19.0.4-3ahwr77o2x/compilers_and_libraries_2019.4.243/linux/compiler/lib/intel64/libintlc.so.5 (0x00007fffeafc6000)
    libsvml.so => /my-gpfs-path/compilers/2020-02-01/linux-rhel7-x86_64/gcc-4.8.5/intel-19.0.4-3ahwr77o2x/compilers_and_libraries_2019.4.243/linux/compiler/lib/intel64/libsvml.so (0x00007fffe9622000)
    libgcc_s.so.1 => /my-gpfs-path/compilers/2020-02-01/linux-rhel7-x86_64/gcc-4.8.5/gcc-8.3.0-r24zbv3waz/lib64/libgcc_s.so.1 (0x00007fffe940a000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007fffe9206000)
    libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007fffe8fcf000)
    libintl.so.8 => /my-gpfs-path/tools/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/gettext-0.19.8.1-ngmcb6sv3g/lib/libintl.so.8 (0x00007fffe8dc4000)
    libutil.so.1 => /lib64/libutil.so.1 (0x00007fffe8bc1000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fffed8e3000)
    libfreebl3.so => /lib64/libfreebl3.so (0x00007fffe89be000)
    libiconv.so.2 => /my-gpfs-path/tools/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/libiconv-1.16-idppdaoskq/lib/libiconv.so.2 (0x00007fffe86c2000)

[aolney]

Sure, here's my ldd output:

ldd /usr/lib/python3/dist-packages/zmq/backend/cython/message.cpython-38-x86_64-linux-gnu.so
linux-vdso.so.1 (0x00007fffb99f7000)
libzmq.so.5 => /usr/local/lib/libzmq.so.5 (0x00007f98353d0000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f98353ad000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f98351bb000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f98351b0000)
libimf.so => not found
libirng.so => not found
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f983505f000)
libcilkrts.so.5 => /usr/lib/x86_64-linux-gnu/libcilkrts.so.5 (0x00007f9835040000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f9834e5f000)
libintlc.so.5 => not found
libsvml.so => not found
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f9834e44000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f9834e3c000)
/lib64/ld-linux-x86-64.so.2 (0x00007f983550e000)

/usr/local/lib/libzmq.so.5 is the one I compiled with icc.

Your question got me thinking though about what's going on in virtual environments, where I'm doing my testing. Here's the output in one of those, which is clearly not using the icc version:

ldd /vm/tf-py3/lib/python3.6/site-packages/zmq/backend/cython/message.cpython-36m-x86_64-linux-gnu.so 
linux-vdso.so.1 (0x00007ffcfa1ec000)
libzmq-1358af2c.so.5.2.2 => /vm/tf-py3/lib/python3.6/site-packages/zmq/backend/cython/../../.libs/libzmq-1358af2c.so.5.2.2 (0x00007f44e632c000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f44e62f6000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f44e62d3000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f44e60e1000)
libsodium-bcf9f097.so.23.3.0 => /vm/tf-py3/lib/python3.6/site-packages/zmq/backend/cython/../../.libs/./libsodium-bcf9f097.so.23.3.0 (0x00007f44e5dbc000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f44e5bd9000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f44e5a8a000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f44e5a6f000)
/lib64/ld-linux-x86-64.so.2 (0x00007f44e6822000)

As before, any suggestions are appreciated.

[martininsulander]

Hi,
I may also have this issue.

Running uwsgi and a python3 app with zmq connecting to a backend.

Requests to backend mostly works the first time and then the app crashes at next.

It works when I run uwsgi from command line, but when running in "emperor mode" it hangs.
I tried to set processes and threads to 1 in uwsgi but it doesn't help.

Using zmq.REQ - zmq.REP on a file socket.
sock.send works, but when calling sock.recv it hangs.
Messages are send to server, but then the app hangs with only this error message.
Assertion failed: ok (src/mailbox.cpp:99)

On Debian testing:
libzmq5 4.3.2-2 4.3.2-2
python3-zmq 19.0.2-1
python3.8 3.8.5-2
uwsgi 2.19.1

[aolney]

@martininsulander for me it was so specific to a particular kernel and a particular library call with that kernel that I just did a work around and left it unsolved.

[caballist-accsys]

For anyone still watching this ticket: zeromq's configure script is broken for use with gcc v7+ due to gcc introducing a new warning (builtin-declaration-mismatch) which is turned on by default; because zeromq's configure script turns all warnings into errors for it's own feature existence tests, the feature tests for fork() and memset() fail.
The memset() feature failure is probably just an efficiency issue, but the fork() failure means any program which forks while using zeromq, is liable to fail in various asserts.
You can fix the issue by adding "-Wno-builtin-declaration-mismatch" to the configure script lines where "-Werror" is added to the "libzmq_cv_cxx_werror_flag" and doing a distclean.

[pramodk]

thank you very much @caballist-accsys for sharing this information! In coming weeks we will give a try with our toolchain and see if this fix the issue!

[jimklimov]

FWIW, I think I wrote at least part of the iterations about these warnings' support in configure and/or ci_build.sh scripts. The code and ideas evolved further in NUT configure.ac script and m4 files it includes, as it also has to deal with as much portability as it can grab. So feel free to cross-pollinate back :) I think the code responsible was in zproject however. Memory is faded, maybe libzmq or czmq did deviate from templates far enough to require backports rather than regeneration ...

…

On Mon, Mar 1, 2021, 15:18 Pramod Kumbhar ***@***.***> wrote: thank you very much @caballist-accsys <https://github.com/caballist-accsys> for sharing this information! In coming weeks we will give a try with our toolchain and see if this fix the issue! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3313 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMPTFHEIJYNTXMBY2KNEMTTBOO2RANCNFSM4GFVF52Q> .

[aolney]

@caballist-accsys I haven't been able to replicate your solution on Ubuntu 20.04. Steps to repro:

sudo apt-get install libtool pkg-config build-essential autoconf automake libsodium-dev uuid-dev checkinstall

# latest libzmq version as of 3/21 is 4.3.4
wget https://github.com/zeromq/libzmq/releases/download/v4.3.4/zeromq-4.3.4.tar.gz
tar -xvf zeromq-4.3.4.tar.gz
cd zeromq-4.3.4
# ADD "-Wno-builtin-declaration-mismatch" to the configure script lines where "-Werror" is added to the "libzmq_cv_cxx_werror_flag"
# If previous builds, do make uninstall and make distclean
./configure
make
sudo checkinstall #makes and installs a .deb; useful for replacing any install package version
sudo ldconfig

# Check installed
ldconfig -p | grep zmq
dpkg -L packagename libzmq5

Where configure has the following on lines 24532 and 24541:

libzmq_cv_cxx_werror_flag="-Werror -Wno-builtin-declaration-mismatch"

ldconfig shows the source-built package has priority over the distro package:

ldconfig -p | grep zmq
	libzmq.so.5 (libc6,x86-64) => /usr/local/lib/libzmq.so.5
	libzmq.so.5 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libzmq.so.5
	libzmq.so (libc6,x86-64) => /usr/local/lib/libzmq.so

Yet still getting the assertion failed:

Assertion failed: ok (src/mailbox.cpp:99)
#0  0xfe7ae63f9228 in ? (?+0x7f3d71c16828)
#1  0xfe7ae6400598 in ? (?+0x7f3d71c16828)
#2  0xfe7ae63fc628 in ? (?+0x7f3d71c16828)
#3  0xfe7ae63f8478 in ? (?+0x7f3d71c16828)
#4  0xfe7ae643a088 in ? (?+0x7f3d71c16828)
#5  0x7f3d777e8609 in /lib/x86_64-linux-gnu/libpthread.so.0 (start_thread+0xd9)
#6  0x7f3d78a1a2d0 in /lib/x86_64-linux-gnu/libc.so.6 (clone+0x43)

Did I implement your solve incorrectly?

[caballist-accsys]

@aolney, I cannot say whether the assert you are getting is a result of the fork configuration issue specifically - there may be another dozen reasons for it. To check whether you have correctly implemented the fork configuration issue I suggest you start from a fresh zmq distribution directory with no changes, run the configuration again and check that you see something like the following in the config.log file:

configure:24809: checking for fork
configure:24809: /opt/rh/devtoolset-8/root/usr/bin/g++ -o conftest -pedantic -Werror -Wall -Wno-long-long -g -O2 -D_GNU_SOURCE -D_REENTRANT -D_THREAD_SAFE conftest.cpp -lrt -lpthread >&5
conftest.cpp:95:6: error: declaration of 'char fork()' conflicts with built-in declaration 'int fork()' [-Werror=builtin-declaration-mismatch]
char fork ();
^~~~

This will confirm whether you have the fork configuration issue in the first place. If you have, re-implement my suggested fix (it looks to me like you got it right) and then re-run configure and again check the config.log to make sure the error above is no longer present.

Good luck

[MenosGrandes]

Hi All,
I have same error. And.. it showed up recently when I updated the Ubuntu from 18 to 20
'm using the PupilLab software, and after rebase I saw that they changed the version of pyZMQ
Is it possible that the version of pyzmq is.. messed up?
Yep. Version of pyZMQ is.. broken?

Or the version of libzmq have changed? As I saw that there are two versions:
libzmq5 and libzmq3.

I have made same changes as the @aolney suggested, but no luck.
So now I'm trying to figure it out on VBox of fresh installed Ubuntu 18

I have changed the pyZMQ from newest (22.0.3) to 20.0.0 and.. there is no error any more. As from now one... I haven't tested it thoroughly... but it's promising.
Edit 2) It doesn't throw the error but it hangs... so no luck
Edit 3) Fresh installation of Ubuntu 18 on VBox works like a charm. So this issue is somewhat about different in gcc/libs/I don't know between Ubuntu 18 and Ubuntu 20( In MY case of course).
Edit 4) Fresh installed Ubuntu 18 on real HW ( Laptop) showed that zeroMQ does not work properly. All PupilLab hangs

[andre-merzky]

It seems to me that the binary wheel distributed by pypi is broken this way. When I force a recompile (pip install --no-use-wheel pyzmq), things work again. This is really bad though, as many of our end users will face this error, and it is not easy to track down and resolve for inexperienced users. I would appreciate it if this ticket could be re-opened and the issue addressed.

[aolney]

@andre-merzky I tried pip install --no-binary :all: --force-reinstall pyzmq and the error still exists with pyzmq-22.0.3. My previous version was pyzmq-20.0.0. This is all in the context of building libzmq from source with the fix discussed above.

[andre-merzky]

@aolney: yes, I am probably lucky that my local compiler works - just wanted to point out that the distributed wheel is broken...

[MenosGrandes]

So.. I still have a problem with it. Any idea how to override it?
Is it connected somehow with specific CPU instructions, as async operations has to be "hacked" somehow to be operable?

I have tried with different types of builds for pyZMQ but with no luck...
This is really hilarious.. All I have to do Is restart computer.. and it works for couple of restarts of software.

[aolney]

@MenosGrandes for me it fails only on specific library calls in R. My workaround is the following

• From Jupyter, write an R script to disk, where the last step in the script serializes the R workspace/variable of interest to disk
• From Jupyter, execute that R script
• From Jupyter, read the serialized file from disk into the R kernel

This avoids ZMQ for the problematic call and isn't too ugly from my point of view

[MenosGrandes]

@aolney thanks for respond.

But pupilaLabs use the ZMQ for communications between processes.. I cannot have workaround for it..

[ariSunz]

I hit this bug while working on code that abuses relies heavily on fork().

Using ZeroMQ 4.3.4 on Debian testing.

Any updates?

[itaych]

@caballist-accsys wrote:

zeromq's configure script is broken for use with gcc v7+

Thanks, your workaround does indeed work, but it appears that this issue has been closed without being fixed. Is there a chance the detection of fork() will be fixed for an upcoming release?

[bluca]

@caballist-accsys wrote:

zeromq's configure script is broken for use with gcc v7+

Thanks, your workaround does indeed work, but it appears that this issue has been closed without being fixed. Is there a chance the detection of fork() will be fixed for an upcoming release?

If someone sends a PR to fix it, yes

[dgloe-hpe]

I got fed up with this issue after running into it a couple of times, so I submitted a PR to fix it. Hopefully that does the trick!