Segfault dumping flows to (batched) MySQL

[cardigliano]

A user which is collecting traffic with a single zmq endpoint from multiple (cento) producers reported a segfault on the flow dump to (batched) MySQL:

Core was generated by `/usr/local/bin/ntopng /run/ntopng.conf'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x000000000052c786 in BatchedMySQLDBEntry::checkLen (this=0x0) at /home/deri/ntopng/pro/BatchedMySQLDBEntry.h:55
55 /home/deri/ntopng/pro/BatchedMySQLDBEntry.h: No such file or directory.
[Current thread is 1 (Thread 0x7f23b5976700 (LWP 19242))]
Missing separate debuginfos, use: yum debuginfo-install brotli-1.0.6-1.el8.x86_64 bzip2-libs-1.0.6-26.el8.x86_64 cairo-1.15.12-3.el8.x86_64 cyrus-sasl-lib-2.1.27-1.el8.x86_64 expat-2.2.5-3.el8.x86_64 f ontconfig-2.13.1-3.el8.x86_64 freetype-2.9.1-4.el8.x86_64 fribidi-1.0.4-7.el8_1.x86_64 glib2-2.56.4-7.el8.x86_64 glibc-2.28-72.el8.x86_64 gmp-6.1.2-10.el8.x86_64 gnutls-3.6.8-8.el8.x86_64 harfbuzz-1.7. 5-3.el8.x86_64 hiredis-0.13.3-11.el8.x86_64 json-c-0.13.1-0.2.el8.x86_64 keyutils-libs-1.5.10-6.el8.x86_64 krb5-libs-1.17-9.el8.x86_64 libX11-1.6.7-1.el8.x86_64 libXau-1.0.8-13.el8.x86_64 libXext-1.3.3 -9.el8.x86_64 libXrender-0.9.10-7.el8.x86_64 libblkid-2.32.1-17.el8.x86_64 libcap-2.26-1.el8.x86_64 libcom_err-1.44.6-3.el8.x86_64 libcurl-7.61.1-11.el8.x86_64 libdatrie-0.2.9-7.el8.x86_64 libffi-3.1-2 1.el8.x86_64 libgcc-8.3.1-4.5.el8.x86_64 libidn2-2.2.0-1.el8.x86_64 libmaxminddb-1.2.0-6.el8.x86_64 libnghttp2-1.33.0-1.el8_0.1.x86_64 libpng-1.6.34-5.el8.x86_64 libpsl-0.20.2-5.el8.x86_64 libselinux-2 .9-2.1.el8.x86_64 libsodium-1.0.18-2.el8.x86_64 libssh-0.9.0-4.el8.x86_64 libstdc++-8.3.1-4.5.el8.x86_64 libtasn1-4.13-3.el8.x86_64 libthai-0.1.27-2.el8.x86_64 libunistring-0.9.9-3.el8.x86_64 libunwind -1.3.1-3.el8.x86_64 libxcb-1.13-5.el8.x86_64 libxcrypt-4.1.1-4.el8.x86_64 libxml2-2.9.7-5.el8.x86_64 libzstd-1.4.4-1.el8.x86_64 mysql-libs-8.0.17-3.module_el8.0.0+181+899d6349.x86_64 nettle-3.4.1-1.el8 .x86_64 openldap-2.4.46-10.el8.x86_64 openpgm-5.2.122-21.el8.x86_64 openssl-libs-1.1.1c-2.el8.x86_64 p11-kit-0.23.14-5.el8_0.x86_64 pango-1.42.4-6.el8.x86_64 pcre-8.42-4.el8.x86_64 pcre2-10.32-1.el8.x8 6_64 pixman-0.36.0-1.el8.x86_64 radcli-1.2.11-3.el8.x86_64 rrdtool-1.7.0-16.el8.x86_64 sqlite-libs-3.26.0-3.el8.x86_64 sssd-client-2.2.0-19.el8.x86_64 systemd-libs-239-18.el8_1.1.x86_64 xz-libs-5.2.4-3 .el8.x86_64 zeromq-4.3.2-1.el8.x86_64 zlib-1.2.11-10.el8.x86_64
(gdb) bt
#0 0x000000000052c786 in BatchedMySQLDBEntry::checkLen (this=0x0) at /home/deri/ntopng/pro/BatchedMySQLDBEntry.h:55
#1 0x000000000052ba76 in BatchedMySQLDB::checkBufferQueue (this=0x5750710, q=0x57816e0, entry=0x0) at pro/BatchedMySQLDB.cpp:322
#2 0x000000000052bd2d in BatchedMySQLDB::dumpFlow (this=0x5750710, when=1580857886, f=0x7f2332d3f280,
json=0x7f22310c1f20 "{ "8": "1.2.3.4", "12": "8.8.4.4", "7": 38337, "11": 53, "4": 17, "57590": 126, "57591": "Google", "2": 1, "1": 68, "24": 1, "23": 162, "22" : 1580857886, "21": 1580857886, "SRC_IP_COUNTRY": "US""...) at pro/BatchedMySQLDB.cpp:460
#3 0x00000000004efc0a in NetworkInterface::dumpFlow (this=0x1d0e940, when=1580857886, f=0x7f2332d3f280) at src/NetworkInterface.cpp:631
#4 0x000000000044cac2 in Flow::dumpFlow (this=0x7f2332d3f280, tv=0x7f23b5975360, dumper=0x1d0e940) at src/Flow.cpp:937
#5 0x000000000044f48b in Flow::periodic_dump_check (this=0x7f2332d3f280, tv=0x7f23b5975360) at src/Flow.cpp:1497
#6 0x0000000000450a3d in Flow::periodic_hash_entry_state_update (this=0x7f2332d3f280, user_data=0x7f23b5975330) at src/Flow.cpp:1947
#7 0x00000000004f6732 in NetworkInterface::generic_periodic_hash_entry_state_update (node=0x7f2332d3f280, user_data=0x7f23b5975330) at src/NetworkInterface.cpp:2682
#8 0x00000000004e5e6e in GenericHash::walkAllStates (this=0x950b240, walker=0x4f66f0 <NetworkInterface::generic_periodic_hash_entry_state_update(GenericHashEntry*, void*)>, user_data=0x7f23b5975330)
at src/GenericHash.cpp:139
#9 0x00000000004e48df in FlowHash::walkAllStates (this=0x950b240, walker=0x4f66f0 <NetworkInterface::generic_periodic_hash_entry_state_update(GenericHashEntry*, void*)>, user_data=0x7f23b5975330)
at src/FlowHash.cpp:114
#10 0x00000000004f690c in NetworkInterface::periodicHTStateUpdate (this=0x1d0e940, deadline=1580858085, vm=0x7f23a4000e38, skip_user_scripts=false) at src/NetworkInterface.cpp:2730
#11 0x00000000004ab154 in ntop_periodic_ht_state_update (vm=0x7f23a4000e38) at src/LuaEngine.cpp:6184
#12 0x00000000005af237 in luaD_precall.part.3 ()
#13 0x00000000005ba0a5 in luaV_execute ()
#14 0x00000000005af488 in luaD_call ()
#15 0x00000000005af4b1 in luaD_callnoyield ()
#16 0x00000000005ae90c in luaD_rawrunprotected ()
#17 0x00000000005af7bb in luaD_pcall ()
#18 0x00000000005ace14 in lua_pcallk ()
#19 0x00000000004b9d03 in LuaEngine::run_loaded_script (this=0x7f23a4000e10) at src/LuaEngine.cpp:11990
#20 0x0000000000415f16 in ThreadedActivity::runScript (this=0x1024a320, script_path=0x7f2384001030 "/usr/share/ntopng/scripts/callbacks/interface/ht_state_update.lua", iface=0x1d0e940,
deadline=1580858085) at src/ThreadedActivity.cpp:238
#21 0x00000000004e1712 in ThreadPool::run (this=0x1024b640) at src/ThreadPool.cpp:102
#22 0x00000000004e13ad in doRun (ptr=0x1024b640) at src/ThreadPool.cpp:31
#23 0x00007f23e57ce2de in start_thread () from /lib64/libpthread.so.0
#24 0x00007f23e34f04b3 in clone () from /lib64/libc.so.6
(gdb)

(gdb) info threads
Id Target Id Frame

• 1 Thread 0x7f23b5976700 (LWP 19242) 0x000000000052c786 in BatchedMySQLDBEntry::checkLen (this=0x0) at /home/deri/ntopng/pro/BatchedMySQLDBEntry.h:55
  2 Thread 0x7f23cfb06700 (LWP 19192) 0x00007f23e34e5591 in poll () from /lib64/libc.so.6
  3 Thread 0x7f23cf305700 (LWP 19193) 0x00007f23e57d44dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  4 Thread 0x7f23cdb02700 (LWP 19196) 0x00007f23e57d44dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  5 Thread 0x7f23ce303700 (LWP 19195) 0x00007f23e57d44dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  6 Thread 0x7f23b5175700 (LWP 19243) 0x00007f23e3475e90 in malloc_consolidate () from /lib64/libc.so.6
  7 Thread 0x7f23b785d700 (LWP 19216) 0x00007f23e57d7e16 in recv () from /lib64/libpthread.so.0
  8 Thread 0x7f23aa7fc700 (LWP 19248) 0x00007f23e34bc848 in nanosleep () from /lib64/libc.so.6
  9 Thread 0x7f23b4974700 (LWP 19244) 0x00007f23e57d8386 in open64 () from /lib64/libpthread.so.0
  10 Thread 0x7f23da9de700 (LWP 18936) 0x00007f23e34f07e7 in epoll_wait () from /lib64/libc.so.6
  11 Thread 0x7f23da1dd700 (LWP 18937) 0x00007f23e34f07e7 in epoll_wait () from /lib64/libc.so.6
  12 Thread 0x7f23a3fff700 (LWP 19252) 0x00007f23e34bc848 in nanosleep () from /lib64/libc.so.6
  13 Thread 0x7f23a8ff9700 (LWP 19251) 0x00007f23e34bc848 in nanosleep () from /lib64/libc.so.6
  14 Thread 0x7f23cd301700 (LWP 19197) 0x00007f23e57d44dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  15 Thread 0x7f23a9ffb700 (LWP 19249) 0x00007f23e34bc848 in nanosleep () from /lib64/libc.so.6
  16 Thread 0x7f23a1ffb700 (LWP 19256) 0x00007f23e346dfc7 in _IO_strn_overflow () from /lib64/libc.so.6
  17 Thread 0x7f23abfff700 (LWP 19245) 0x00007f23e34eaee7 in mmap64 () from /lib64/libc.so.6
  18 Thread 0x7f23a37fe700 (LWP 19253) 0x00007f23e34bc848 in nanosleep () from /lib64/libc.so.6
  19 Thread 0x7f23a27fc700 (LWP 19255) 0x00007f23e34bc848 in nanosleep () from /lib64/libc.so.6
  20 Thread 0x7f23a2ffd700 (LWP 19254) 0x00007f23e34bc848 in nanosleep () from /lib64/libc.so.6
  21 Thread 0x7f23ab7fe700 (LWP 19246) 0x00007f23e34bc848 in nanosleep () from /lib64/libc.so.6
  22 Thread 0x7f23aaffd700 (LWP 19247) 0x00007f23e34bc848 in nanosleep () from /lib64/libc.so.6
  23 Thread 0x7f23e7290ec0 (LWP 18885) 0x00007f23e34e7b1f in select () from /lib64/libc.so.6
  24 Thread 0x7f23a97fa700 (LWP 19250) 0x00007f23e34bc848 in nanosleep () from /lib64/libc.so.6
  25 Thread 0x7f23ceb04700 (LWP 19194) 0x00007f23e57d44dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

[cardigliano]

Note: this could be related to the same issue in #3353 (comment)